KR 10 Leap Motion VR Control by YoushaaMurhij - 7

NetworkInputsVR & AR

Online/Offline Control of KUKA-KR10 using Leapmotion sensor and VR controllers inside Unity Environment

Unknown VersionApache License 2.0Updated 263 days agoCreated on January 31st, 2020
Go to source


Online/Offline Control of KUKA-KR10 using Leapmotion sensor and VR controllers inside Unity Environment.

In this repo:

A development and implementation of a VR application controlling an industrial manipulator was done to improve the usability of this manipulator and enable the interaction between the user and the robot and its workspace features

  • Hardware: Our Hardware part included an industrial manipulator connected to a PC using TCP/IP connection. In addition to AR/VR supported headset and controllers with an IR stereo camera (Leapmotion) as an option.
  • Software: A Unity3D VR application containing the 3D model of the manipulator connected with the real robot and VR instruments.

Data input methods:

  1. IR stereo camera sensor (Leapmotion) : captures both hands movements in real time.
  2. AR/VR sensors ( HTC VIVE pro instruments) : captures head and controllers movements in real time.

System Procedure:

  • Leapmotion: hand movements read as position coordinates
  • VR : the head and controllers movements read as 6DOF data.
  • Data Filtering is sometimes required in this stage to remove noise and outliers. (Complementary filter was used).
  • 6DOF data of the controllers is mapped to the position and orientation of the industrial robot’s end effector in the application.
  • Inverse Kinematics is used to calculate the generalized parameters of the virtual robot.
  • Generalized parameters are sent to the joints of the real robot.


If you find our work useful in your research, please consider citing:

        title = "An application to simulate and control industrial robot in virtual reality environment integrated with IR stereo camera sensor",
        journal = "IFAC-PapersOnLine",
        year = "2019",
        note = "19th IFAC Conference on Technology, Culture and International Stability TECIS 2019",
        issn = "2405-8963",
        doi = "",
        author = "Youshaa Murhij and Vladimir Serebrenny",


  • Install Unity Software, SteamVR, HTC VIVE
  • Clone this repository :
$git clone  
  • ~Add dependencies: VIVE Input, SteamVR to Unity.~ [Done]
  • Plug in HTC Vive VR instruments and LeapMotion sensor [Optional].
  • Create your own scene and Start.
  • Connect the real KR10 robot [Optional].
  • Enjoy 😃


You can check my project on My ResearchGate Profile

Show all projects by YoushaaMurhij