Building HoloLens Applications with Unity

HoloLens with Unity

Unity is the recommended game engine used to develop HoloLens applications. It uses C# as its programming language and includes tools like an API and SDK to help with the development of HoloLens experiences.

It’s actually possible to start creating a HoloLens application without the HMD by installing a HoloLens emulator onto your PC and using it to test the holographic display you’re building. The official Microsoft documentation provides a series of tutorials on how you can make use of these tools to build your own application. Here, we’ll outline the basic settings and functions for getting started with the Microsoft HoloLens.

Settings in Unity

To enable Unity’s ability to build for HoloLens application we need to first setup a few options to ensure the smooth running of the application on the device. These include the build setting, project setting and the camera setting.

The Project Setting in Unity

Set quality to “Very Low” for faster performance and to maintain a good frame rate. Then target the application to Windows Holographic SDK.

In the XR setting select “Windows Mixed Reality” as supported Virtual Reality SDKs. Then set the scripting backend to “.NET”.

The Camera Setting in Unity

Cameras in a desktop app represent the screen you’re looking at. In HoloLens applications they represent the user’s points of view. Setup the camera as position 0, then set the “Clear Flags from Skybox” to “Solid Color” and set the background as RBGA(0, 0, 0, 0). This will represent the real world behind any virtual objects.

The near clip plane of the camera can be set from 0.3 to 0.85 metres.

Deploying to Windows Store

Launch Visual Studio and click “start without debugging” then choose the target as “Emulation” (or choose “HoloLens” if you have the device connected). Now you have your first HoloLens application running.

Adding Holograms

When you add 3D objects to your 3D scene you’re adding holograms to your environment. Place the Z position an the object to allow the viewer to see it. Measurements in the Unity scene are mapped onto the real world. When you place an object two metres away it will appear two meters away in the real world. You can scale the holograms as you can scale 3D objects in any scene.

Using Different Interactions

Microsoft HoloLens supports gaze, gestures and voice controls for user interactions. In Unity we can use different functions and the API to customise these interactions for our applications.

Gaze Targeting

We can define how the gaze control works in Unity, including the target placement and targeting behaviors. We can also define the feedback after the user gazes at a 3D object.

Gestures

Built in gestures include Tap, Double Tap, Hold, Manipulation and Navigation, each of which can be used in Unity via the API.

Voice

Vocal input is provided by the Windows 10 API. There are three styles of input: the keyword recogniser, grammar recogniser and the dictation recogniser. Keyword recognition listens for specific phrases spoken by the user. The grammar recogniser is used when complex phrases need to be recognised and the app needs to use more than a few simple keywords. The dictation recogniser can convert speech to text, which becomes useful if feedback is given afterwards.

Other Settings for Better HoloLens Applications

There are some other settings that can help improve stability and interaction with the real world to make a more convincing holograms and a better HoloLens application overall.

Focus Point

HoloLens performs automatic holographic stabilisation based on motion and changes in points of view. By defining a stabilisation plane that plane would have the maximised hardware stabilisation. We can use the FocusPoint API to define a plane to determine where the user is focusing. This can improve the stability of the holograms that we placed in the scene.

World Anchor

We can use a world anchor to lock game objects in a physical world, overriding the position of the object. For example, when we want to place an object on a table we can use the world anchor component to achieve this effect.

Spatial Mapping

Spatial mapping is to create a 3D map of the environment and makes it possible to put holograms onto real surfaces.

The HoloLens constantly scans its surroundings and updates the sensor data. We can access the spatial mapping data through SurfaceObserver in Unity to give us the data of any surfaces added, changed and removed. Surfaces are numbers of regularly-sized geometries mapping the world and the data is stored on the device.

There are two spatial mapping components: the Spatial Mapping Renderer and the Spatial Mapping Collider. Spatial Mapping Renderer components give a visual representation of spatial mapping surfaces, which allows us to debug surfaces and add visual effects to the environment. Every time there are changes in the physical world they will be baked into the game object. The Spatial Mapping Collider enables the interaction of holographic content and physical surfaces, and allows for raycasting to the surfaces.

Next Steps

Developing HoloLens applications as a core process is relatively straightforward, particularly with the tools and libraries available. Designers and developers can then focus on thinking of new ideas for any given application, and combining other technologies to create new user experience. This is where the art comes in.

By using the tools available to us, and adding additional ones as needed, we can begin to see how HoloLens applications can be constructed from the ground up. Each application needs to be created with its target audience in mind, giving them the deeper experience that only mixed reality can provide.