June 8, 2018
Head of Virtual Production, MergeXR
Enhancing Virtual Production
Design and Development Challenges
I embarked on my virtual production journey in 2021 with a passion for blending technology and creativity. Since then, I've been on an incredible path of discovery and innovation. Throughout my time at MergeXR, I've gathered invaluable knowledge from various resources and experiences, shaping my unique approach to creating a future of storytelling tools.
So what is Virtual Production?
It is the future of storyteling!
Virtual production lets you shoot the impossible, edit the incredible, and create the unthinkable—all in real-time with the help of Unreal Engine!
Think of it as directing a blockbuster from your laptop, with virtual sets, real-time effects, and no travel headaches. It’s the ultimate creative playground where imagination meets technology!
The Process
3a.Design first approach
If our domain falls under Enterprise VR and Consumer VR, We usually like to follow Double Diamond Design methods first.
3a
1.Discover
Your Idea
2.Define
Define your scope and what you want to do
3a.Design & Refine
What you want to create
Rapid Prototyping Loop
Rapid prototyping helps us to be extremely explorative and test our concepts and Ideas.
RP
4.Validate
Perform usability test
5.Implement
Begin Development
RP Loop
3b.Prototype first approach
If our domain falls under Entertainment, Game, or testing new Interactions, We usually like to develop a prototype first.
3b
3b.Prototype
Always develop your VR concepts quickly
It's different!
The process of designing for VR is very different from the traditional approach used for UI/UX Design (for 2D interfaces). The key difference would be iterative design and prototyping. You have to test your concepts as soon as you can.
Most of your users would likely be "VRgins" so remember that not everyone is comfortable with new technology. Keep new users always in mind. "keep it simple, and test often" is our success mantra!
7 Things to consider while designing
Play
Area
Input
methods
Interface
Design
Interaction
Design
Audio
Design
3D
World
Performance &
Comfort
1. Define your Play Area
Just like Artboard is to 2D, the Play Area is to VR
Seated
For comfortable media and information consumption in VR
Standing
For Unidirectional experience with limited physical locomotion in VR
Room Scale
For an omnidirectional and completely immersive experience with room/world scale locomotion
Why is it important?
Consider the "Play Area" to be your canvas. Just like while designing 2D interfaces we define our artboard size. Play Area plays similar importance in VR. Defining your play area helps you to structure your UI design and define/ideate on what interactions would be more suitable for the user and resonates with the experience you want to create.
Suggestion chart
Seated
Standing
Room Scale
Session Time/ Play Area
>15 Mins
10 - 15 Mins
<10 Mins
Category/ Play Area
Entertainment
Aircraft Sim
Nature Experience
Infotainment
Social Spaces
Education
Music Viz
Military Training
Tourism / Heritage Viz
Field Training
Military SiM
Healthcare Sim
Museums
Architecture Viz
Retail
Product / Industrial Viz
Games
Behavioral Changes
Movement Restriction
Head Movement
Body Movement
Hand Movement
Safety
Moderate
High
Moderate
Low Risk
Moderate
Moderate
Low
Moderate Risk
Low
Low
Low
High Risk
2. Define your Input Methods and Modalities
Always have a clear idea of how you want your users to interact with your application
Gaze Based Input
Using eye-tracking or head rotation as a input methods to interact with UI/World elements
Mobile Input
Using Augmented Virtuality / Digital Twin to use mobile phones as input methods to interact
Motion Controllers
Using dedicated motion controllers paired with your headset as an input method
Voice Input
Using voice commands as an input method in VR
Hand & Body Tracking
Using Hand and body tracking with gesture recognition to interact with elements
Custom Controllers
Using programable boards to design a custom controller to be used as an input device for VR
There are a plethora of input options available for VR which can be used to interact with VR content. It could be very tempting to go multimodal while designing your application. But having a clear direction of which methods to use can also help you design your content better
There are many other methods that can be used for your purpose. The listed methods are the one which we have tested/ build in Unreal Engine. You can also design custom controllers with programable boards and interface it with the application you want to build. Imagination is the limit!
Watch them in action / Suggested Scenarios
Gaze based / Eye-tracking
Can be preferred for Infotainment and Entertainment Apps and Applications that require lazy interactions.
Hand / Body Tracking
Can be preferred for applications that require direct manipulation and natural interactions
Motion Controllers
Can be preferred for Intense Games and Applications that require precise tracking and numerous controls
Mobile Input
Can be preferred for applications that require extensive typing
Interface Design
UI Distance
People love reaching out to grab/interact with UI. We try to keep our interactable UI within reach of human hands, allowing the user to touch it.
UI Size and Zone
It is recommended to position your UI in a comfortable head rotation zone. However, you would have to compromise on UI size and visibility.
UI Color
Try not to use contrasting colors, Blasting White color in eyes of the user can be discomforting.
Instead prefer darker colors, more relaxing colors for your UI.
Canvas Shape
Consider curved canvas shape if your UI is large or break it into sections consisting of 2D planes placed in a circular formation
Visibility of System Status
If something is loading clearly state that to your user. Always create a UI motion flow so that the user is notified of his action and can easily see it happening.
Reactive UI
Be reactive to the user's action and movement. Let the user know if he is doing something wrong or right with visual and audio cues
Particles and World
Consider curved canvas shape if your UI is large or break it into sections consisting of 2D planes placed in a circular formation
Typography
Explore and be experimental. Play with weights, strokes, and texture of font. Avoid using light typefaces
3D UI
Play with the new unlocked dimension. Try adding real-world tangible interfaces and be playful
Types of UI in VR
Bars and Waypoint Element seems to be in 3D space but they are depth independent. Meaning, the size of bar and waypoint will not change even if the objects they are connected to move closer or far from user
Direction Element is present in 3D space but does not exist in world.
Outline Element seems to be in 3D space but overlayed as stylized 2D on user's HUD. Overlay size is dependent on the distance of outlined object
Keypad, Watch, Bolt and Knob Elements are part of the world
Status Elements are part of player's HUD and does not exist in world
Elements present on VR spectator screen are Meta elements for VR user