Logo

dev-resources.site

for different kinds of informations.

SteamVR Overlay with Unity: Follow Device

Published at
6/16/2024
Categories
unity3d
steamvr
openvr
vr
Author
kurohuku
Main Article
https://dev.to/kurohuku/part-6-follow-device-33i
Categories
4 categories in total
unity3d
open
steamvr
open
openvr
open
vr
open
Author
8 person written this
kurohuku
open
SteamVR Overlay with Unity: Follow Device

Follow the HMD

Image description

Overlay following the HMD

Remove the position code

First, remove the absolute position code we added in the previous part.

private void Start()
{
    InitOpenVR();
    overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);

    SetOverlaySize(overlayHandle, 0.5f);

-   var position = new Vector3(0, 2, 3);
-   var rotation = Quaternion.Euler(0, 0, 45);
-   SetOverlayTransformAbsolute(overlayHandle, position, rotation);

    ShowOverlay(overlayHandle);

    ...
Enter fullscreen mode Exit fullscreen mode

Device Index

In SteamVR, connected devices are identified with Device Index that automatically allocates from the system. (read the wiki for details)
For HMD, it is defined as OpenVR.k_unTrackedDeviceIndex_Hmd and is always 0.

Prepare position and rotation

Let’s display the overlay at 2 m ahead (Z-axis) of the HMD. As in the previous part, let position and rotation variables.

private void Start()
{
    InitOpenVR();
    overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

+   var position = new Vector3(0, 0, 2);
+   var rotation = Quaternion.Euler(0, 0, 0);

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);

    SetOverlaySize(overlayHandle, 0.5f);
    ShowOverlay(overlayHandle);
}
Enter fullscreen mode Exit fullscreen mode

Set relative position based on the HMD

Use SetOverlayTransformTrackedDeviceRelative() to set the relative position based on the HMD. (read the wiki for details)

private void Start()
{
    InitOpenVR();
    overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

    var position = new Vector3(0, 0, 2);
    var rotation = Quaternion.Euler(0, 0, 0);
    var rigidTransform = new SteamVR_Utils.RigidTransform(position, rotation);
    var matrix = rigidTransform.ToHmdMatrix34();
+   var error = OpenVR.Overlay.SetOverlayTransformTrackedDeviceRelative(overlayHandle, OpenVR.k_unTrackedDeviceIndex_Hmd, ref matrix);
+   if (error != EVROverlayError.None)
+   {
+       throw new Exception("Failed to set overlay position: " + error);
+   }

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);

    SetOverlaySize(overlayHandle, 0.5f);
    ShowOverlay(overlayHandle);
}
Enter fullscreen mode Exit fullscreen mode

Pass the HMD device index (OpenVR.k_unTrackedDeviceIndex_Hmd) and the transformation matrix.
Run the program, and check the overlay is shown 2 m ahead of the HMD.

Image description

Organize code

Move the relative position code into SetOverlayTransformRelative().

private void Start()
{
    InitOpenVR();
    overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

    var position = new Vector3(0, 0, 2);
    var rotation = Quaternion.Euler(0, 0, 0);
-   var rigidTransform = new SteamVR_Utils.RigidTransform(position, rotation);
-   var matrix = rigidTransform.ToHmdMatrix34();
-   var error = OpenVR.Overlay.SetOverlayTransformTrackedDeviceRelative(overlayHandle, OpenVR.k_unTrackedDeviceIndex_Hmd, ref matrix);
-   if (error != EVROverlayError.None)
-   {
-       throw new Exception("Failed to set overlay position: " + error);
-   }
+   SetOverlayTransformRelative(overlayHandle, OpenVR.k_unTrackedDeviceIndex_Hmd, position, rotation);

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);

    SetOverlaySize(overlayHandle, 0.5f);
    ShowOverlay(overlayHandle);
} 

...

+ // Pass deviceIndex as argument.
+ private void SetOverlayTransformRelative(ulong handle, uint deviceIndex, Vector3 position, Quaternion rotation)
+ {
+     var rigidTransform = new SteamVR_Utils.RigidTransform(position, rotation);
+     var matrix = rigidTransform.ToHmdMatrix34();
+     var error = OpenVR.Overlay.SetOverlayTransformTrackedDeviceRelative(handle, deviceIndex, ref matrix);
+     if (error != EVROverlayError.None)
+     {
+         throw new Exception("Failed to set overlay position: " + error);
+     }
+ }
Enter fullscreen mode Exit fullscreen mode

Follow the controller

Image description

Overlay following the controller

Use controller device index instead of HMD to make the overlay follow a controller.

Get the controller device index

Get the left controller’s device index with GetTrackedDeviceIndexForControllerRole() of OpenVR.System.

private void Start()
{
    InitOpenVR();
    overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

    var position = new Vector3(0, 0, 2);
    var rotation = Quaternion.Euler(0, 0, 0);
+   var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
    SetOverlayTransformRelative(overlayHandle, OpenVR.k_unTrackedDeviceIndex_Hmd, position, rotation);

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);

    SetOverlaySize(overlayHandle, 0.5f);
    ShowOverlay(overlayHandle);
}
Enter fullscreen mode Exit fullscreen mode

The argument is:
Left controller: EtrackedControllerRole.LeftHand
Right controller: EtrackedControllerRole.RightHand

If it fails to get the device index like the controller is disconnected, GetTrackedDeviceIndexForControllerRole() returns k_unTrackedDeviceIndexInvalid.

Follow the controller

We have got the left controller index, then make the overlay follow the controller. Pass the controller index to SetOverlayTransformRelative() that we previously created for the HMD.

private void Start()
{
    InitOpenVR();
    overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

    var position = new Vector3(0, 0, 2);
    var rotation = Quaternion.Euler(0, 0, 0);
    var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
-   SetOverlayTransformRelative(overlayHandle, OpenVR.k_unTrackedDeviceIndex_Hmd, position, rotation);
+   if (leftControllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
+   {
+       SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);
+   }

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);

    SetOverlaySize(overlayHandle, 0.5f);

    ShowOverlay(overlayHandle);
}
Enter fullscreen mode Exit fullscreen mode

Make sure the left controller is connected to the SteamVR window, then run the program.

The overlay should follow the controller instead of the HMD.

Image description

Adjust overlay position

To make a watch application, we will adjust the overlay position on the left wrist. Make position parameters editable at runtime on the Unity editor.

Add member variables

Add size, position, and rotation variables as class members. Use Range() attribute to show sliders on the inspector.

public class WatchOverlay : MonoBehaviour
{
    private ulong overlayHandle = OpenVR.k_ulOverlayHandleInvalid;

+   [Range(0, 0.5f)] public float size = 0.5f;
+   [Range(-0.2f, 0.2f)] public float x;
+   [Range(-0.2f, 0.2f)] public float y;
+   [Range(-0.2f, 0.2f)] public float z;
+   [Range(0, 360)] public int rotationX;
+   [Range(0, 360)] public int rotationY;
+   [Range(0, 360)] public int rotationZ;

    ...
Enter fullscreen mode Exit fullscreen mode

Sliders are shown on the inspector.

Image description

Replace variables in the code

Replace the existing size and position variables in the code with the added member variables.

private void Start()
{
    InitOpenVR();
    overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

-   var position = new Vector3(0, 0, 2);
-   var rotation = Quaternion.Euler(0, 0, 0);
+   var position = new Vector3(x, y, z);
+   var rotation = Quaternion.Euler(rotationX, rotationY, rotationZ); 
    var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
    if (leftControllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
    {
        SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);
    }

    SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);

-   SetOverlaySize(overlayHandle, 0.5f);
+   SetOverlaySize(overlayHandle, size);
    ShowOverlay(overlayHandle);
}
Enter fullscreen mode Exit fullscreen mode

Update size and position in Update()

Make the size and position editable at runtime by adding code to the Update(). Note that this code will be deleted later. It is temporary for determining new positions and rotation.

private void Start()
{
    InitOpenVR();
    overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

    var position = new Vector3(x, y, z);
    var rotation = Quaternion.Euler(rotationX, rotationY, rotationZ);
    var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
    if (leftControllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
    {
        SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);
    }

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);

    SetOverlaySize(overlayHandle, size);
    ShowOverlay(overlayHandle);
}

+ private void Update()
+ {
+     SetOverlaySize(overlayHandle, size);
+ 
+     var position = new Vector3(x, y, z);
+     var rotation = Quaternion.Euler(rotationX, rotationY, rotationZ); 
+     var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
+     if (leftControllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
+     {
+         SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);
+     }
+ }

...
Enter fullscreen mode Exit fullscreen mode

Run the program. Ensure the inspector slider changes the overlay size and position at runtime.

Image description

Adjust overlay position

Move sliders to adjust the overlay position to the left wrist. I recommend changing sliders from the desktop window that is opened in the SteamVR dashboard.

Image description

Image description
Control the Unity editor from the SteamVR dashboard.

Another way is looking at the HMD view on the desktop and adjusting parameters.

Image description

Image description
It’s helpful if you don’t want to put on the HMD.

Here are sample parameters.

size = 0.08
x = -0.044
y = 0.015
z = -0.131
rotationX = 154
rotationY = 262
rotationZ = 0
Enter fullscreen mode Exit fullscreen mode

When adjusting is done, right click on the WatchOverlay component name, and select Copy Component.

Image description

Stop the program, right click the WatchOverlay component again, and paste the copied values with Paste Component Value.

Image description

Run the program. Check the overlay is on the left wrist.

Image description

Remove temporary code

Remove the code from the Update().

private void Update()
{
-   SetOverlaySize(overlayHandle, size);
-   
-   var position = new Vector3(x, y, z);
-   var rotation = Quaternion.Euler(rotationX, rotationY, rotationZ); 
-   var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
-   if (leftControllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
-   {
-       SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);
-   }
}
Enter fullscreen mode Exit fullscreen mode

When the controller is not connected

Currently, the controller must be connected at launch because we get the controller device index at the Start().
Move the getting device index code from Start() to Update() to support the cases where the controller connects or disconnects in the middle.

private void Start()
{
    InitOpenVR();
    OverlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

-   var position = new Vector3(x, y, z);
-   var rotation = Quaternion.Euler(rotationX, rotationY, rotationZ);
-   var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
-   if (leftControllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
-   {
-       SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);
-   }

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);

    SetOverlaySize(overlayHandle, size);

    ShowOverlay(overlayHandle);
}

+ private void Update()
+ {
+     var position = new Vector3(x, y, z);
+     var rotation = Quaternion.Euler(rotationX, rotationY, rotationZ);
+     var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
+     if (leftControllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
+     {
+         SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);
+     }
+ }
Enter fullscreen mode Exit fullscreen mode

Now, it works even if the controller is connected in the middle.

Final code 

using UnityEngine;
using Valve.VR;
using System;

public class WatchOverlay : MonoBehaviour
{
    private ulong overlayHandle = OpenVR.k_ulOverlayHandleInvalid;

    [Range(0, 0.5f)] public float size;
    [Range(-0.2f, 0.2f)] public float x;
    [Range(-0.2f, 0.2f)] public float y;
    [Range(-0.2f, 0.2f)] public float z;
    [Range(0, 360)] public int rotationX;
    [Range(0, 360)] public int rotationY;
    [Range(0, 360)] public int rotationZ;

    private void Start()
    {
        InitOpenVR();
        overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

        var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
        SetOverlayFromFile(overlayHandle, filePath);

        SetOverlaySize(overlayHandle, size);
        ShowOverlay(overlayHandle);
    }

    private void Update()
    {
        var position = new Vector3(x, y, z);
        var rotation = Quaternion.Euler(rotationX, rotationY, rotationZ);
        var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
        if (leftControllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
        {
            SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);
        }
    }

    private void OnApplicationQuit()
    {
        DestroyOverlay(overlayHandle);
    }

    private void OnDestroy()
    {
        ShutdownOpenVR();
    }

    private void InitOpenVR()
    {
        if (OpenVR.System != null) return;

        var error = EVRInitError.None;
        OpenVR.Init(ref error, EVRApplicationType.VRApplication_Overlay);
        if (error != EVRInitError.None)
        {
            throw new Exception("Failed to initialize OpenVR: " + error);
        }
    }

    private void ShutdownOpenVR()
    {
        if (OpenVR.System != null)
        {
            OpenVR.Shutdown();
        }
    }

    private ulong CreateOverlay(string key, string name)
    {
        var handle = OpenVR.k_ulOverlayHandleInvalid;
        var error = OpenVR.Overlay.CreateOverlay(key, name, ref handle);
        if (error != EVROverlayError.None)
        {
            throw new Exception("Failed to create overlay: " + error);
        }

        return handle;
    }

    private void DestroyOverlay(ulong handle)
    {
        if (handle != OpenVR.k_ulOverlayHandleInvalid)
        {
            var error = OpenVR.Overlay.DestroyOverlay(handle);
            if (error != EVROverlayError.None)
            {
                throw new Exception("Failed to dispose overlay: " + error);
            }
        }
    }

    private void SetOverlayFromFile(ulong handle, string path)
    {
        var error = OpenVR.Overlay.SetOverlayFromFile(handle, path);
        if (error != EVROverlayError.None)
        {
            throw new Exception("Failed to draw image file: " + error);
        }
    }

    private void ShowOverlay(ulong handle)
    {
        var error = OpenVR.Overlay.ShowOverlay(handle);
        if (error != EVROverlayError.None)
        {
            throw new Exception("Failed to show overlay: " + error);
        }
    }

    private void SetOverlaySize(ulong handle, float size)
    {
        var error = OpenVR.Overlay.SetOverlayWidthInMeters(handle, size);
        if (error != EVROverlayError.None)
        {
            throw new Exception("Failed to set overlay size: " + error);
        }
    }

    private void SetOverlayTransformAbsolute(ulong handle, Vector3 position, Quaternion rotation)
    {
        var rigidTransform = new SteamVR_Utils.RigidTransform(position, rotation);
        var matrix = rigidTransform.ToHmdMatrix34();
        var error = OpenVR.Overlay.SetOverlayTransformAbsolute(handle, ETrackingUniverseOrigin.TrackingUniverseStanding, ref matrix);
        if (error != EVROverlayError.None)
        {
            throw new Exception("Failed to set overlay position: " + error);
        }
    }

    private void SetOverlayTransformRelative(ulong handle, uint deviceIndex, Vector3 position, Quaternion rotation)
    {
        var rigidTransform = new SteamVR_Utils.RigidTransform(position, rotation);
        var matrix = rigidTransform.ToHmdMatrix34();
        var error = OpenVR.Overlay.SetOverlayTransformTrackedDeviceRelative(handle, deviceIndex, ref matrix);
        if (error != EVROverlayError.None)
        {
            throw new Exception("Failed to set overlay position: " + error);
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Here, We attach the overlay on the left wrist. Let’s draw Unity camera output to the overlay instead of the static image file in the next part.

vr Article's
30 articles in total
Favicon
Explore the World of AR/VR: 5 Essential Skills for 2025
Favicon
Exploring the Synergy Between Virtual Reality and Preconstruction Planning
Favicon
The Transformative Landscape of VR Game Development in India
Favicon
Introduction to Android XR
Favicon
VR and Architecture: Visualizing Real-World Designs Through Gaming Technology
Favicon
From Bricks& Beams to Bits&Bytes
Favicon
Why Are We Still Learning in 2D When the World is 3D?
Favicon
AI-Driven Background Removal: Streamlining Photography Workflows
Favicon
The Future of VR Gaming: Immersive Worlds Beyond Reality
Favicon
The Role of VR Technology in Education, Medical Sector, and Industry: A Path to Profit for Newcomers
Favicon
Role of AI In Enhancing VR Experiences
Favicon
Building IP in Gaming: From Concept to Brand
Favicon
What is Cloud Gaming for VR?
Favicon
Cross-Platform Game Development: Building Games for All Devices
Favicon
The Evolution of Game Design: A Journey Through Time
Favicon
The Evolution of Gaming: From Pixels to Virtual Realities and Beyond
Favicon
AR VR Company in Australia: Next XR Group
Favicon
Step into the Metaverse with Alibaba Cloud and Goes Beyond Reality
Favicon
Virtual Reality Rehabilitation Vs Conventional Physical Therapy for Stroke Rehabilitation
Favicon
Virtual Reality (VR) in Virtual Reality (VR) in Healthcare - Applications & Benefits
Favicon
The Future of Construction Collaboration Virtual and Augmented Reality Tools
Favicon
LED companies use VR technology to layout the smart education market
Favicon
CSS for VR and AR: Styling for Virtual Worlds
Favicon
SteamVR Overlay with Unity: Controller Input
Favicon
Dev: AR/VR
Favicon
SteamVR Overlay with Unity: Appendix
Favicon
SteamVR Overlay with Unity: Overlay Events
Favicon
SteamVR Overlay with Unity: Dashboard Overlay
Favicon
SteamVR Overlay with Unity: Follow Device
Favicon
SteamVR Overlay with Unity: Draw Image

Featured ones:

Favicon
Open
abubakersiddique761
Favicon
Open
0x3d_site
Favicon
Open
0x2e_tech
Favicon
Open
0x4c-quest
Favicon
Open
gittech