A tool for gesture-based interactions with smartphone sensors.
The idea of gestures here means patterns of bi-dimensional coordinates over time. To use gesture recognition, some template gestures are stored in a training set that is later used to compare and classify gestures, giving each try a match with a score. As long as you can map two float values to the gesture recorder and recognizer, you can define gestures from all sorts of input sources.
- If coordinates are attached to an accelerometer sensor, gestures will involve the player moving the phone on a specific way.
- If coordinates are attached to touch position on the phone screen, then gestures will be more similar to drawings on that surface.
- If you use magnetic fields as coordinates, then it will detect specific movements in relation to Earth’s magnetic field or a close-by magnet.
This tool was developed and used in the Red Dirt game project. You can find a more detailed discussion of this tool’s design and process over here.
- Integration with the $1 Unistroke Gesture Recognizer.
- Two workflows: an OSC-based one for prototyping and an native sensor one for builds (Why two workflows?).
- Example interface for recording gestures for later use (on the
The tool is saved here both as a project and as a unitypackage file. AFter downloading the project, you can copy the contents of the
SensingGestures folder to your own project or you can double-click the unitypackage and import the files from there.
- Add the
Gesturesprefab to a scene. It contains a few things to make easier to work with gestures, such as a basic debug user interface to visualize values and gestures, as well as a standard “wiring” of events.
- Connect a sensor to the
UpdateInputValuefunction of the
GestureRecognizercomponent. You need to define which sensor you will use in the Inspector window for both the OSC Sensor and Native Sensor game objects. Make sure that both game objects are getting information from the same sensors. To connect, you can use the UnityEvent interface to access the
GestureRecognizer.UpdateInputValuemethod (check highlights below).
- OSC Sensor: Check the prototyping workflow instructions here. Use the same port and OSC address value as in the Sensors2OSC app or other OSC-server you are using.
- Native Sensor: Select the sensor and make sure that the
Sensor Readerand the
Native Sensor Dispatchercomponents reference the same values (
Gravityin the example).
- Now you can run the scene in the editor to test gesture recording and recognizing using the
Rkeyboard keys. On the training set file in your Project window, you can change the name of your gestures to something more usable. You can also duplicate it and switch between different sets, if you want.
- In a script from your game, you can declare a method with the header
public void YourMethodName (Result result)and connect it to the
OnRecognizedGestureevent on the
GestureRecognizercomponent. Your method will then be called with information about the result that you can use in your game logic.
If you have ideas for features or bugs you encountered while using the shellphone project, please open an issue. If you’d like to contribute with code or features yourself, please fork the repository and use a feature branch. Pull requests are warmly welcome.
Ideas for contributions
- Add other gesture recognizer algorithms, such as the PDollar recognizer, which supports multistroke gestures (which is great for touchscreen-drawing gestures).
- Prepare an example using touch-position coordinates as inputs.
- Documentation is available both at the wiki and on the
- Here you can find more Information about the $1 unistroke gesture recognizer and the Unity implementation which I’m using.
- Open Sound Control (OSC) is a flexible way of communicating between devices and applications. Sensing Gestures uses the OscJack library for Unity. I recommend using the Sensors2OSC app for accessing smartphone sensors and sending their values over a wifi network.
- To access Android phone sensors beyond accelerometers and touchscreen, I have used the Android Sensors for Unity library. It is used when building the app for the phone.