Create voice commands for VR experiences with Watson services
Read this in other languages: 中国.
Watson Conversation is now Watson Assistant. Although some images in this code pattern may show the service as Watson Conversation, the steps and processes will still work.
In Virtual Reality, where you truly “inhabit” the space, speech can feel like a more natural interface than other methods. Providing speech controls allows developers to create more immersive experiences. The Oculus Rift is the 4th most popular head-mounted VR devices (not including Google Cardboard) and an ideal candidate for Speech interaction, selling roughly 355 thousand units in 2016.
When the reader has completed this Code Pattern, they will understand how to:
- Add IBM Watson Speech-to-Text and Assistant to a Virtual Reality environment build in Unity.
- User interacts in virtual reality and gives voice commands such as “Create a large black box”.
- The Oculus Rift Headset microphone picks up the voice command and the running application sends it to Watson Speech-to-Text.
- Watson Speech-to-Text converts the audio to text and returns it to the running Application that powers the Oculus Rift.
- The application sends the text to Watson Assistant. Watson conversation returns the recognized intent “Create” and the entities “large”, “black”, and “box”. The virtual reality application then displays the large black box (which falls from the sky).
- IBM Watson Assistant: Create a chatbot with a program that conducts a conversation via auditory or textual methods.
- IBM Watson Speech-to-Text: Converts audio voice into written text.
- Unity: A cross-platform game engine used to develop video games for PC, consoles, mobile devices and websites.
Watch the Video
1. Before You Begin
2. Create IBM Cloud services
On your local machine:
git clone https://github.com/IBM/vr-speech-sandbox-vive.git
In IBM Cloud:
- Create a Speech-To-Text service instance.
- Create a Assistant service instance.
- Once you see the services in the Dashboard, select the Assistant service you created and click the button.
- After logging into the Assistant Tool, click the button.
- Import the Assistant
workspace.jsonfile located in your clone of this repository.
3. Building and Running
If you followed the previous steps you should already be inside your local clone and ready to get started running the app from Unity.
git clone https://github.com/watson-developer-cloud/unity-sdk.git
Open Unity and inside the project launcher select the button.
Navigate to where you cloned this repository and open the “Creation Sandbox” directory.
If prompted to upgrade the project to a newer Unity version, do so.
Follow these instructions to add the Watson Unity SDK downloaded in step 1 to the project.
Open the script
Fill in the credentials for Speech to Text and Assistant, and the Assistant workspace id:
private string stt_username = "blahblah"; private string stt_password = "abc123youandme"; private string stt_url = "https://stream.watsonplatform.net/speech-to-text/api"; private string convo_username = "dlkfjasdkfdkfjdkjf"; private string convo_password = "kdfjlsdjf"; private string convo_url = "https://gateway.watsonplatform.net/conversation/api"; .....<snip>..... private string convo_workspaceId = "kdjfls2323";
- Install Blender
- In the
Unity editor projecttab, select
MainMenuand double click to load the scene.
- Artificial Intelligence Code Patterns: Enjoyed this Code Pattern? Check out our other AI Code Patterns.
- AI and Data Code Pattern Playlist: Bookmark our playlist with all of our Code Pattern videos
- With Watson: Want to take your Watson app to the next level? Looking to utilize Watson Brand assets? Join the With Watson program to leverage exclusive brand, marketing, and tech resources to amplify and accelerate your Watson embedded commercial solution.