Yolo Detection Holo Lens Unity by doughtmw - 4

FrameworksCameraVR & AR

Object detection on the HoloLens through streaming PV camera frames to a companion PC, processing those using a YOLO framework, and returning the bounding box data which is rendered using Unity.

Unknown VersionUnknown LicenseUpdated 129 days agoCreated on November 14th, 2019
Go to source


Object detection sample using the Yolo framework, HoloLens photo/video sensor stream, and HoloLens device for visualization of bounding boxes. C++ project to receive PV sensor frames, and send bounding boxes is made available for use in Unity through IL2CPP Windows Runtime support. More details are presented in my blog post.


Desktop (server) HoloLens (client)

Difference in above bounding box size is caused by capturing frames from different time points and the viewing perspective through the HoloLens.


Yolo Detect Sample

Build HoloLensForCV project and configure YoloDetectionHoloLensUnity Unity project

  1. Open HoloLensForCV sample in VS2017 and install included OpenCV nuget package to HoloLensForCV project
  2. Build the HoloLensForCV project (x86, Debug or Release)
  3. Copy all output files from HoloLensForCV path (dlls and HoloLensForCV.winmd) to the Assets->Plugins->x86 folder of the YoloDetectionHoloLensUnity project
  4. Open YoloDetectionHoloLensUnity Unity project, enter the IP address of your desktop PC into the relevant field on the script holder game object
  5. Under Unity build settings, switch the platform to Universal Windows Platform and adjust relevant settings. Build project using IL2CPP
  6. Open VS solution from Unity build, build then deploy to device

Build and run ComputeOnDesktopYolo project

  1. Open the HoloLensForCV sample and build the ComputeOnDesktopYolo project (x64, Debug or Release)
  2. Deploy project to Local Machine

Combining the samples to process HoloLens PV camera frames

  1. Ensure the HoloLens and PC are on the same network and the HoloLens is currently running the YoloDetectionHoloLensUnity sample
  2. When the deployed ComputeOnDesktopYolo desktop app opens, enter the IP of the HoloLens to connect and click connect
  3. Sensor frames from the HoloLens should begin to stream to the desktop and appear in the app window
  4. When prompted on the HoloLens, perform the double tap gesture to connect to the host socket and begin receiving information about detected objects in streamed frames (bounding boxes and updated on screen text)
  • Be sure to check the Package.appxmanifest files for both the C++ ComputeOnDesktopYolo project and the C# YoloDetectionHoloLensUnity project to ensure permissions are satsified
  • Required capabilities are as follows:


  • Restricted capabilities package:
  <Package xmlns="http://schemas.microsoft.com/appx/manifest/foundation/windows10" 
    IgnorableNamespaces="uap mp">
  • Modified capabilities with with new package:
    <Capability Name="internetClient" />
    <Capability Name="internetClientServer" />
    <Capability Name="privateNetworkClientServer" />


  • These capabilites should be updated automatically through the Unity project build settings

  • Ensure that permissions to internet client, internet client server, private network client server, and webcam are all satisfied

  • Restricted capabilities package:

  IgnorableNamespaces="uap uap2 uap3 uap4 mp mobile iot" 
  • Modified capabilities with with new package:
    <Capability Name="internetClient" />
    <Capability Name="internetClientServer" />
    <Capability Name="privateNetworkClientServer" />
    <uap2:Capability Name="spatialPerception" />
    <DeviceCapability Name="webcam" />
Show all projects by doughtmw