XR experiences are becoming increasingly common in the culture and communication sectors. The technology is used in movies and plays to fully immerse the audience in the experience, while companies use AR to showcase prototypes. But even though XR has become part of everyday life, the technology has to meet increasingly higher demands as to what it should be able to do.
Joseph keeps rowing in his memories. First through the streets of Copenhagen, which is marked by the devastation of World War II, then on the open sea, where the waves threaten to overturn the small rowing boat. Eventually, he arrives in Sweden, where he is confronted with everything he was forced to leave behind in Denmark.
End of Night is the story of Josef, a Jewish man who fled from Denmark to Sweden in 1943, told through a film created in Virtual Reality by the production company Makropol.
“The main character sits right there in front of you, looks you in the eye, and tells you a story. And because he is sitting right there, a different kind of presence is created. You feel the protagonist’s fear when you’re on open water. You do not experience it second hand, but on your own body,” says Mads Damsbo, founder and partner in Makropol.
The production company started working in earnest with VR in 2014 when Makropol was tasked with telling a story from five different angles. The project was called Skammekrogen and the focal point was a dinner with five family members.
“I had heard about VR but did not think it was relevant for us. Still, we bought a pair of VR glasses on Kickstarter, and when we sat down to play with it, we realized that you could show a movie on the glasses. All of a sudden you could experience life through the eyes of another person,” Damsbo says.
Macropol has received great recognition for their work which lies in the intersection between film and VR. End of Night won the award for Best Immersive Story at the Venice Film Festival last year. Despite the success, however, Makropol continues to challenge the VR medium and the technology behind it.
“Right now it is not possible to smell, taste, or feel in VR – but it is probably only a question of time before that is possible. The question then becomes how do we use it in an ethical and exciting way. I hope that more resources are spent on researching how humans can exist and experience virtual worlds. It is still foreign country,” Damsbo says.
VR brings the audience back to 1146
In Viborg, Jakob Tekla Jørgensen is waiting for more technological power. He is the artistic director of White Hole Theater, which in the summer of 2024 will be premiering the XR performance 3 Kings.
With a little help from VR, the performance will take the audience back to Stænderpladsen in Viborg in 1146, where Svend, Knud, and Valdemar fought for the Danish throne. Some audience members will watch the performances through VR glasses, while others will watch the physical theater performance while being able to see the virtual world through their phones.
“Some of the things we need to be able to do in 2024 are not possible now. So we depend on the road maps that both the tech giants and our smaller partners have to pan out. We are a theater, so we have to be able to have a lot of people watch the play live in XR. And much of the technology we use is not geared for that yet,” Jørgensen says.
White Hole Theater is, therefore, constantly testing the limits of the technology. In the summer of 2021, the audience could watch the mini-performance Arnold’s Vision, which featured a single actor in a motion capture suit. In the summer of 2022, White Hole Theater takes the next step by having four actors and five extras re-enact a battle that took place in Viborg.
“I hope the audience feels that the earth is a little bit different when the play is over. That they feel transported back to a time when men fought to become king,” Jørgensen says.
The audience also plays a direct role in the battle, as White Hole Theater has incorporated a gaming element into the performance. Four selected audience members play alongside the actors and influence how the battle ends.
“They feel like they are part of something big, while also having their own perspective on the battle. In the long run, we hope that VR theater can be a very special immersive experience for more than four people,” Jørgensen says.
Rokoko creates presence in XR
Rokoko is one of the companies that helps VR plays and movies become a reality. The tech company sells motion capture equipment, which enables game companies and filmmakers to record human movements, which can subsequently be linked to animated characters. Thus, a human can become a troll, alien, or something completely different.
“You can interact with the virtual world through our technology. Either by recording a series of movements and creating an animation or by putting on the motion capture equipment and walking around the metaverse in real time. In other words, you become a human joystick,” says Jakob Balslev, co-founder and CEO of Rokoko.
If an authentic presence is to be created in XR, however, it is crucial for Balslev that the individual’s facial expressions and body language are also transferred to the digital world.
“Until you have your unique body language, you are not really present. Your family must be able to recognize you, solely on how you move – even if your character is a frog,” Balslev says.
So far, the XR industry is not Rokoko’s biggest customer. But it is probably only a matter of time, Balslev says. Therefore, Rokoko has also started optimizing their equipment for the XR industry. Specifically, it is a problem that the sensors on Rokoko’s suits and gloves, which detect movements, move over time.
“You can’t play the piano in VR for an hour and still hit the right keys. At the same time, there is a delay in your movements. And we could optimize these elements, but there hasn’t yet been great incentive to do so,” Balslev says.
That will change in the autumn, however, when Rokoko launches a new product, which is basically tailored to XR. Balslev cannot yet reveal what the new product is, but he promises that it will be a slam dunk for the market.
For many, Augmented Reality is still foreign land. But Hololink wants to change that. The company makes it possible to create interactive AR experiences without coding, which is opened directly in the phone’s browser.
“Our main goal is to enable all companies and organizations to start experimenting with AR as a medium before it completely overruns them in a few years,” said Lucas Nygaard, co-founder and CEO of Hololink.
He compares the XR wave to the Internet in the 1990s and social media in the 2010s. If you did not adopt the trend of building a website or having a presence on social media early on, you were quickly overtaken by all the companies that had already started experimenting.
“If smaller companies and organizations adopt XR, it’s not only up to the big Silicon Valley companies to decide what the metaverse should be. We want to take it to a simpler level, where more people can participate,” Nygaard says.
Hololink has about 1,000 users worldwide, which includes 3rd graders who make AR Christmas cards, to large international companies whose employees can quickly create a prototype and display it in AR.
“With our tool, any employee can test an idea and take it all the way up through the organization. They can quickly and without having to code make a prototype that actually works,” Nygaard says.
Therefore, he hopes that companies and educational institutions will begin experimenting with the technology.
“We have been talking about AR for many years, we have read all the big headlines and been to all the exhibitions. But now we have to take the next step, where we find out what it looks like in everyday life. And we will not get there if we all sit around, waiting for AR glasses to arrive. If you’re waiting for, say, Apple to launch AR glasses before you create an AR strategy you’re already behind,” Nygaard says.
A look at the climate crisis through VR
In 2017, the organization Connect 4 Climate organized an international pitch competition: How can you utilize VR to focus on a climate problem?
The Danish production house Mannd, which specializes in XR storytelling, won the competition and thus the film X-Ray Fashion was created. The VR documentary focuses on the fashion industry and the environmental and climate challenges that come with producing huge quantities of clothes.
“Whether you’re interested in fashion or not, everyone has a relationship with the clothes they put on every morning. So everyone can relate to the topic. Therefore, we chose to focus on the fashion industry, which is a slightly different take on the climate crisis than the one we usually see,” says Maria Engermann, owner of Mannd.
In the documentary, the audience is brought into a sweatshop in India, where men and women sew clothes under miserable conditions. For Engermann, however, it was not enough for the audience to see how their clothes were made. They also had to feel the heat in the enclosed sweatshop, feel the water around their feet when the chemicals from the garment production are flushed out into the local river, and smell the stench from the landfill where much of the clothing ends up.
“We wanted to go a step further and our goal was to make people really feel like they were present in the production. Therefore, we decided early on that we also had to use physical effects, because the tactile element plays a huge role in terms of whether one feels present or not. If we create a strong VR experience that makes a great impression on the audience, then we have a better chance of changing their consumption habits,” Engermann says.
And it is exactly here VR falls short. Because even though VR, according to Engermann, is a strong medium, it only caters to two senses: sight and sound. At least at the moment.
“We have to figure out how to involve tactile elements. Maybe it’s a suit you’re connected to, even though it’s a little hard to imagine right now,” Engermann says with a smile.