Developing for Hololens

//Developing for Hololens

Developing for Hololens

Hi there reader,

My name is Paul Thornton and I am the CTO of SoftApps based in Norwich at St George’s Works, Colegate. I am a qualified games developer and in recent years I have turned my skills to commercial purpose developing for Virtual Reality and Augmented Reality for businesses in Norwich, Ipswich, Cambridge in the U.K  creating content for oculus rift, HTC-VIVE, GearVR and now Microsoft’s Hololens. 

Over the last few months I have been lucky enough to be able to work with the Hololens, Microsoft’s Augmented Reality hardware offering, and it’s awesome.. its awesome and it’s heavy: Like super heavy. Wearing it for hours a day really starts to hurt, Take the nose bridge adaptor thing out. And spend the time adjusting it. Your face will thank you.

Its got a terrible Field Of View (FOV):   Horizontally it’s “ok”, but vertically it’s terrible. It’s not the worst AR headset I’ve used by any means (it’s actually the best by quite a large margin). But still needs work. Looking at physically large things up close will always be an issue with the tops and bottom being cut off. Other people have covered this in depth. It is an issue. It doesn’t mean the hololens is a bad experience however. Far from it. It could just be better.

The GPU is flat out terrible.  Whilst I understand power size heat battery life are all things needing considering. It would totally benefit from a better GPU.

To give an idea. I opened unity, created a new scene., added a single terrain. (default settings no changes). Built & deployed and I was already running with some quite serious stutter.

 

Animations:the suggested approach is to use 1 bone for blending animations. I just found this looked terrible. At least 2 bones. Ideally 4 if you have the performance to spare.


Avoid Shadows / Lights, Ideally you want to be using the unlit mobile shaders. The default standard shader  will hurt performance badly.  


Spatial Mapping is Awesome, However avoid Spatial Understanding unless you actually need it. Its yet another performance drain. And unless you need to explicitly place items in the world and need that data.  Do not bother.  Spatial mapping and the Collision data it generates can work fine.

You can likely get what you need out of just a couple of raycasts on the Spatial Physics mesh rather than using full spatial understanding.

Example: Placing a UI on a wall can just be a raycast to position and and rotate the ui to face along the returned normal.

Particle Track Behavior at a Wall Boundary



Unity’s new Runtime NavMesh stuff (https://github.com/Unity-Technologies/NavMeshComponents) is really awesome combined with spatial mapping. Being able to just slap down a nav surface and have it bake out navigation based on the real world is awesome. (Dont let it on every update to the spatial data. Just run a bake every few seconds. It’s more than responsive enough),

UI buttons should also respond to voice controls. It can be as simple as raycasting along the users gaze direction and invoking what every button you hit when the user utters “Select” for example. Voice commands should ideally have matching GUI buttons. Voice detection is good.

But certain accents will throw it off badly. Dont leave the user stranded shouting at the headset trying to progress.

It seems it has some inbuilt learning / improvement stuff going on with cortana. It certainly has gotten way better at detecting my voice than it was to start with.

 

The inbuilt KeywordRecognizer is useful, but won’t let you add or remove voice commands at runtime.

Build a list of all keywords your app could ever use and hand that over to the Keyword Recognizer at start up.  Keep another Dictionary<string, Action> and add remove your event handlers there. Can just invoke the function if it exists at the time the keyword is detected

 

 public class DemoSpeechHandler : MonoBehaviour { 


       public Dictionary<string, System.Action> keywords = new Dictionary<string, System.Action>();
       private KeywordRecognizer keywordRecognizer;

       void Start() { 
        TextAsset TextFile = Resources.Load<TextAsset>("Data/Phrases.txt"); // One keyword per line in a text file. 
        if (TextFile != null)
        {
            using (StreamReader sr = new StreamReader(new MemoryStream(TextFile.bytes)))
            {
                while (!sr.EndOfStream)
                {
                    var keyword = sr.ReadLine();
                    if (keywords.ContainsKey(keyword.ToLower()))
                    {
                        continue;
                    }
                    else
                    {
                        RegisterKeywordHandler(keyword, () =>
                        {
                            Debug.Log("Keyword Detected: \"" + keyword + "\" But no handler has ever been registered");
                        });
                    }
                }
            }
        }

        keywordRecognizer = new KeywordRecognizer(keywords.Keys.ToArray());
        keywordRecognizer.OnPhraseRecognized += KeywordRecognizer_OnPhraseRecognized;
        keywordRecognizer.Start();
    } 
 
    //Reset a keyword to have a default handler.
    public void ResetKeywordHandler(string keyword)
    {
        if(keywords.ContainsKey(keyword.ToLower()))
        {
            keywords[keyword.ToLower()] = () =>
            {
                Debug.Log("Keyword Detected: \"" + keyword.ToLower() + "\" But no handler has ever been registered");
            };
        }
    }
 
    // Register a keyword handler
    public void RegisterKeywordHandler(string keyword, Action handler, bool Override = true)
    {
        if(!Override 
            && keywords.ContainsKey(keyword.ToLower()))
        {
            Debug.Log("Wanted to register a keyword Handler: " + keyword.ToLower() + " but slot was taken");
        }
 
        keywords[keyword.ToLower()] = handler;
    }
 
    // Invoke keyword handler if exists.
    private void KeywordRecognizer_OnPhraseRecognized(PhraseRecognizedEventArgs args)
    {
        System.Action keywordAction;
        if (keywords.TryGetValue(args.text.ToLower(), out keywordAction))
        {
            keywordAction.Invoke();
        }
    }
}

 

Tooling still needs work. Having to build a visual studio project as the unity build step. Then open and build that project to deploy.  It just makes the feedback time a little to long for my liking.

All things considered the hololens is an awesome little device to work with., but you do need to take far more care about performance than other platforms.. Overall it’s a great Minimum Viable Product and I cannot wait to see what they do with Version 2 due 2019/2020.

 

By | 2017-07-25T22:44:08+00:00 July 7th, 2017|Augmented Reality|Comments Off on Developing for Hololens

About the Author: