As the world moves ever closer to an integration between the physical and digital, Microsoft’s HoloLens is fast becoming the leader in mixed reality technology. At FX Digital we’ve been fortunate enough to have our R&D team working on creating and adapting content for the HoloLens for some time, and it’s staggering to know just what it’s capable of. But what goes into creating an application?
Gaze input tracks the direction a user is looking in based on spatial mapping of the environment, then determines what objects the user is looking at. We can make use of a gaze cursor to interact in a similar manner to VR applications.
Gestures are hand signals used to control the system. There are two core gestures that can be used, Air Tap and Bloom. Bloom is reserved for starting the menu in HoloLens. The system can also recognise combined gestures such as taps and hold.
Air Tap is possibly the most common mode of gesture interaction within HoloLens applications:
Somewhat impressively, we can also use voice controls to interact with HoloLens scenes, the most frequently used being:
Take a picture
Developing an application that uses voice control is fraught with challenges. People in a single region may speak in many different ways, and when taking a global population into account the difficulties are compounded. There are, however, a number of general practices that developers can undertake to make their voice controlled applications as optimal as possible:
Use concise commands
Use simple vocabulary
Use commands that are not destructive so users can go back
Avoid similar sounding commands
Unregister the commands when not in use
Test with accents
Avoid using system commands
Visual Studio with Windows 10 SDK
HoloLens emulator (Windows 10 Home cannot run Hyper-V, which is required)
Unity and Vuforia, which can create holographic apps that recognise elements of an environment
To augment any given environment with digital objects we need to first understand the environment we’re viewing via spatial mapping. This process maps physical surfaces into the virtual world, giving us a way to make convincing interactions between holograms and the real world. The HoloLens can scan its surroundings and refine based on new data scanned. Some surfaces are invisible to the HoloLens, including transparent, black, and reflective surfaces.
There are different ways to collect spatial data. We can ask users to use bloom gestures, or provide custom scanning for different applications. For a better experience we should also consider the way the application presents the scanning process to the user. We can also pre-collect the environment data to give a more accurate representation.