Virtual Sets with Adrian Gonzalez

As we’ve been seeing lots of changes and advances in production and workflows lately, we wanted to take some time to catch up with Adrian Gonzalez, Mistika Boutique Product Manager, to discuss some of the ins and outs of using and working with Visual Sets. Here’s some helpful information about where things currently are with this new technology, the role Mistika Boutique plays in the complete creation of this content and also what capabilities are on the horizon.

Los Angeles Post Production Group: Can you tell us about yourself and your background. How did you start working with SGO and on Virtual Sets?

Adrian Gonzalez: I joined SGO eight years ago, and since then I have been working as a demo artist, training mentor and in the past two years as a Mistika Boutique Product Manager. My profile is halfway between the creative and the technical, so I regularly collaborate with the development team at SGO defining new Mistika Technology features and products, and Graphic User Interface design updates.

Our technology is one of the pioneering solutions in the immersive post-production technology sector and Mistika Boutique (and Ultima) is the only finishing system that enables the complete creation of immersive content from initial optical flow stitching, color grading, VFX and all the way to the final deliverables, so being able to apply it to Virtual Sets is a natural progression.

LAPPG: Can you explain to us the benefits of using Virtual Sets?

AG: Beside the fact that Virtual Sets tear down the limits of what can be seen through the camera on a live-action set and what needs to be imagined to be added digitally many months later, virtual production has been making the content creation process more sustainable and economical, eliminating the need for location shoots.

In addition, it removes many of the issues related with the green screens like the green hue contamination in reflective surfaces and of course the time needed to make a good selection in a complex green screen. By using Virtual Sets, the integration between the virtual footage and the real part of the shot is much better and organic.

LAPPG: What type of productions have you seen currently that are making the most use of this technology?

AG: Virtual Production is more or less just at the beginning of its journey and because of the high investment required, only bigger studios and production companies are able to afford it at the moment. Basically, all types of production are suitable for Virtual Production, however the most impactful for on-set development is definitely VFX-heavy content. One of the most well-known projects in the Virtual Set is Disney’s The Mandalorian, of which over half was filmed indoors on a virtual set.

LAPPG: Are there any situations where a typical green screen would be a better choice than Virtual Sets or should we be trying to use this newer technology as much as possible?

AG: LED backgrounds are able to provide more realistic environments for the actors and other functions that green and blue screens may lack. Shooting in Virtual Sets is smoother and provides better lighting and colors and even reflections on metallic surfaces, for example.

Green screen can be a better choice in shots where the backgrounds are extremely complex, for example CG content with camera movement, physics simulation, etc. In those shots, the background will be recreated in the post production phase rather than in the shooting phase, because it requires time and the implication of VFX and 3D packages in a complex pipeline. So, in those scenarios recreating that kind of background before shooting and before knowing exactly how that shot would look can be problematic.

LAPPG: What are the biggest challenges that occur when you are planning to use virtual sets?

AG: As mentioned before, this is just the beginning of Virtual Production and therefore there is a lack of knowledge and experience among industry professionals. Another challenge is a technical one. In order to get the best possible quality, it is pretty typical to work during most parts of the pipeline with big resolution files in formats like EXR or high quality ProRes. So, you need a powerful technical ecosystem that is capable of managing these kinds of files. As a final challenge, it requires much better preparation, because those backgrounds need to be done in advance, before the final shooting.

LAPPG: How does Mistika Technology help create solutions and solve problems?

AG: Mistika Technology enables the complete creation of Virtual Backgrounds – from the initial optical flow stitching to color grading, VFX, and final deliverables. At this moment, Mistika is the industry standard solution to work with 180º/360º shots, thanks to the unrivaled VR capabilities. At the same time, we have optimized the system to work with huge resolution formats and heavy media, so the performance is a key factor here as well. So, if you combine the fact that you can create the virtual backgrounds from scratch, in any 360º format and with the best performance, what you have at the end is the best tool for this new technique.

LAPPG: Are there any best practices or things we should be aware of when we are setting out to build virtual sets?

AG: Resolution and color workflow, probably. Resolution is key to know exactly what kind of images we need to deliver to the virtual set, and which kind of camera or 360 rig we need to use.

At the same time, knowing the display color space is important in order to build a proper color pipeline. Those images are going to be adjusted and graded like any other image, so we need to design a good workflow to manage those backgrounds from the format and the color science point of view.

LAPPG: It seems that Mistika Boutique has all the functions of Mistika VR plus more tools for compositing and grading along with an amazing timeline that allows users to edit with any kind of clip including 360. So, in what cases would we use Mistika VR instead of Mistika Boutique or is it best to use them together?

AG: Exactly – Mistika VR is actually just a small piece of Mistika Boutique. Mistika VR handles just the first part of the immersive content creation – optical flow stitching and stabilisation. Once this is done, you could simply take the project data to Mistika Boutique without the need to render in order to complete the Color and VFX. However, you could also just do the entire project (including Optical Flow stitching) in Mistika Boutique.

Mistika VR is especially useful when a big team is involved in creating those virtual backgrounds. Most of the work will be the stitching, and for that it is more efficient to have several Mistika VR licenses (which are cheaper obviously) and leave Mistika Boutique only for the finishing part. Communication between Mistika products is perfect, all you need is to save the project in Mistika VR and open it in Mistika Boutique, and you’ll have access to all the adjustments done by the VR user. From there, remember without rendering, you can grade your shot, make any composition and finally deliver it. This allows parallelization of the production of these backgrounds and saves a huge amount of time.

LAPPG: It appears Mistika VR & Mistika Boutique form a system for covering all the needs for building virtual sets. Are there any capabilities that you are looking forward to for these programs being able to do in the future that are not currently in place yet?

AG: We want to improve the management of VR images with complex aspect ratios. Normally VR shots work in a 2:1 aspect ratio, as this is the aspect ratio of an unwrapped sphere, but with Virtual Sets sometimes you need different settings that require at this moment an extra step to deliver, such as cropping an interesting area. In the future we will remove that extra step to manage those scenarios in a quicker and a smoother way. And of course, because this is a new technique that is growing every day, we will face new challenges, so we will improve our tools following those challenges and the feedback of our users, as we always do.