As the world moves ever closer to an integration between the physical and digital, Microsoft’s HoloLens is fast becoming the leader in mixed reality technology. At FX Digital we’ve been fortunate enough to have our R&D team working on creating and adapting content for the HoloLens for some time, and it’s staggering to know just what it’s capable of. But what goes into creating an application?
Getting started with HoloLens application development
Microsoft HoloLens enables us to create mixed reality experiences with inputs from humans, the computer, and crucially the environment. The internet is filled with tools, examples, and tutorials detailing the platform itself, and how to design and build an application for it. And Microsoft’s own developer network has a wealth of information. But before you sit down at your keyboard you need to decide what your app is for, why you’re building it, and how it will positively impact the life of its users. This isn’t so much of a “how to” guide, rather it’s a whistle stop tour of what goes into the process.
Designing a HoloLens application
HoloLens scenes place digital content on top on the real world, resulting in a mixed reality environment. The scene renders holograms in additive displays, which add digital light to that of the real world. However, enhancements such as dynamic lighting are resource intensive so should be avoided (if possible) because of the performance issues they cause.
Positioning Objects
Objects in a mixed reality space can be body locked (tag-along) and world locked (billboarding). Objects with billboarding always facing the user, while tag-along objects always follow the direction of vision. Depending on the purpose of any given object or the user interface itself, we can consider different methods of displaying content when designing the scene.
User Comfort
Microsoft provides some guidelines on the HoloLens user experience. When placing holograms into the environment the optimal zone for display is 1.25m to 5m. For the angle of view, Microsoft recommends an optimal gaze angle from 0o to 35o below the horizon to avoid user discomfort.
Design Process
To build a HoloLens application we need to have a clear understanding of its purpose and how the HoloLens enhances the user experience. Because the HoloLens is a relatively new platform it’s helpful to determine best practices for 3D its UI. This comes with a process of user testing, prototyping, and trial and error. AfterNow and Microsoft both recommend a design process based on the principles of product design, which includes steps of envisioning, prototyping, and building. We can adjust the process based on the needs of different applications.
HoloLens Inputs
Gaze
Gaze input tracks the direction a user is looking in based on spatial mapping of the environment, then determines what objects the user is looking at. We can make use of a gaze cursor to interact in a similar manner to VR applications.
Gestures
Gestures are hand signals used to control the system. There are two core gestures that can be used, Air Tap and Bloom. Bloom is reserved for starting the menu in HoloLens. The system can also recognise combined gestures such as taps and hold.
Air Tap is possibly the most common mode of gesture interaction within HoloLens applications:
Voices
Somewhat impressively, we can also use voice controls to interact with HoloLens scenes, the most frequently used being:
Go Home
Take a picture
Start/Stop recording
Increase/Decrease Volume
Developing an application that uses voice control is fraught with challenges. People in a single region may speak in many different ways, and when taking a global population into account the difficulties are compounded. There are, however, a number of general practices that developers can undertake to make their voice controlled applications as optimal as possible:
Use concise commands
Use simple vocabulary
Use commands that are not destructive so users can go back
Avoid similar sounding commands
Unregister the commands when not in use
Test with accents
Avoid using system commands
Performance and Limitations
The minimum target performance of the HoloLens is 60fps and the memory should be less than 900MB for the application to run properly. If an application exceeds that limit, the application will be terminated. In visually demanding scenes it may even be necessary to lower the resolution for displaying content and reducing the complexity of shaders as required. The system performance page can be used to track real time performance of the application.
Building a HoloLens application
HoloLens development environments will, of course, vary from person to person but there are a number of common tools that the process requires. It’s important to consider the power of the machine you’re building the experience on compared to that of the machines that are likely to run the end product.
Development Tools
Visual Studio with Windows 10 SDK
HoloLens emulator (Windows 10 Home cannot run Hyper-V, which is required)
Unity and Vuforia, which can create holographic apps that recognise elements of an environment
Spatial Mapping
To augment any given environment with digital objects we need to first understand the environment we’re viewing via spatial mapping. This process maps physical surfaces into the virtual world, giving us a way to make convincing interactions between holograms and the real world. The HoloLens can scan its surroundings and refine based on new data scanned. Some surfaces are invisible to the HoloLens, including transparent, black, and reflective surfaces.
There are different ways to collect spatial data. We can ask users to use bloom gestures, or provide custom scanning for different applications. For a better experience we should also consider the way the application presents the scanning process to the user. We can also pre-collect the environment data to give a more accurate representation.
Use Cases
There are many potential applications to build with HoloLens. It can be used in entertainment, education and workplace, or helping companies or brands to communicate to each other and customers.
For us this means an opportunity to offer our clients a broader, next level service to reach their customers. Engaging with audiences around the world through mixed reality environments brings a new range of possibilities along with it, including the ability to share experiences and spaces for a more fluid collaborative process.
Combining with other technologies like AI and voice recognition, there are many opportunities for HoloLens applications to create different value adds for business, and deeper engagement for users.
