Behind the Scenes of Eight VR

Eight VR is a mixed reality project by Dutch composer and director Michel van der Aa, a trailblazing fusion of musical theatre and VR. Eight tells the story of a woman’s life in reverse chronological order. Visitors, equipped with VR goggles and headphones, move through a physical installation. They can manipulate physical and virtual objects and meet the woman at various crucial moments of her life. The woman’s memories are spread out across the flexible walls in the form of interactive hieroglyphs, ready to be activated. Physical hallways merge with virtual ones in an incredible fashion, creating an infinite space.

To achieve this, composer and director Michel Van der Aa collaborated with The Virtual Dutch Men, a pioneering 3D and VR Dutch company. They joined forces to create a unique and groundbreaking fusion of music, theatre and VR, which currently makes the buzz in the Netherlands: EIGHT just got shortlisted with the best 10 projects for the “Gouden Kalf” – Best Interactive award in the Netherlands Film Festival.

TimeAware, which handled the mocap had a chance to speak to The Virtual Dutch Men in a brief web session.

TimeAware: 

Congratulations for your “Best Interactive award” nomination at the “Gouden Kalf”! What was the main challenge with “EIGHT”?

Roelof Terpstra, Senior Producer and CEO of “The Virtual Dutch Men”:

The challenges we faced with the project were of 3 types : 1-technological complexity , 2- time and 3- budget. The biggest one was the complexity of the technology. Neither Michel (the Director) nor we had done such a demanding and innovative project in VR so far. We have a lot of experience with VR, but never with extremely realistic human beings. We knew we had to learn a lot and use some tricks to get there.

TimeAware: 

What do you mean by “technological complexity”?

Richard de Paauw, Lead Developer:

VR is very different from traditional video and film productions, where the DoP controls composition, camera angle and distance. In VR the spectator can walk right next to the 3D characters and look at them into the eyes from, say, 10cm away. This presents a couple of challenges:

– First one was how to get the user to follow the story instead of just looking around. We needed very compelling and lively characters for this.

– Second challenge was how to prevent users from being scared of the imperfections of the relatively low poly 3D model from such a close point of view. This phenomenon is better known as the “Uncanny Valley” effect and occurs when the 3D characters are so realistic that you start noticing every flaw – so instead of compelling they become repelling.

– Last challenge was how to deal with almost 30 minutes of facial animation under tight budget and incredible turnaround times.

Roelof Terpstra:

Given the task at hand we decided to go with a hybrid technique: record videos of the faces of all three actresses during the mocap sessions and later project these videos on the faces of the 3D models in VR. Unfortunately video projection alone is not enough. We needed the facial mesh to deform correctly when the characters were singing, especially when the mouth was wide open. Luckily the guys from Timeaware, who were doing the mocap, had access to Grabber software from Dynamixyz and the beta version of Dynamixyz generic facial tracker. The effect was stunning: in just few days videos were tracked with good results. Videos were recorded with a GoPro, which had no stable white balance and lighting conditions in the studio were constantly changing.

Ahnill Christiaans, Assistant Developer:

We also highly appreciated the great technical support provided by Dynamixyz team to get us up and running in the game engine. We practically did not have a learning curve. Their developers were writing code for us a couple of times per day sometimes. You don’t get such support even by top industry giants now-a-days.

TimeAware: 

Thank you for sharing your thoughts with us, we’re looking forward to receiving more news on the project.

Roelof Terpstra:

Eight was just presented during the “Festival Aix-en-Provence” in South of France. We introduced minor updates to the experience before it leaves for Beijing for the “Beijing Music Festival” in October.

Additional information:

Eight VR: www.eightvr.net

EightVR tour dates:  www.eightvr.net/dates

The Virtual Dutch Men: www.thevirtualdutchmen.com

Michel Van Der Aa: www.vanderaa.net

 

 

 

0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *