Sdsandbox by tawnkramer - 1


This provides a sandbox simulator for training a self-driving car. This uses Unity for simulation and Python with Keras and Tensorflow for training. Recently updated to work on Python 3.4+ and Keras 2+

Unknown VersionBSD 3-Clause "New" or "Revised" LicenseUpdated 28 days agoCreated on January 14th, 2017
Go to source


Self Driving Car Sandbox



Use Unity 3d game engine to simulate car physics in a 3d world. Generate image steering pairs to train a neural network. Uses NVidia PilotNet NN topology. Then validate the steering control by sending images to your neural network and feed steering back into the simulator to drive.

Some videos to help you get started

Training your first network


World complexity


Creating a robust training set



You need to have Unity installed, and all python modules listed in the Requirements section below.

Linix Unity install here. Check last post in this thread.

You need python 3.4 or higher, 64 bit. You can create a virtual env if you like:

virtualenv -p python3 env
source env/bin/activate

And then you can install the dependancies. This installs a specific version of keras only because it will allow you to load the pre-trained model with fewer problems. If not an issue for you, you can install the latest keras.

pip install -r requirements.txt

This will install Donkey Gym and Donkey Car packages from source.

Note: Tensorflow >= 1.10.1 is required

If you have an cuda supported GPU - probably NVidia

pip install tensorflow-gpu

Or without a supported gpu

pip install tensorflow


  1. Load the Unity project sdsandbox/sdsim in Unity. Double click on Assets/Scenes/road_generator to open that scene.

  2. Hit the start button to launch. Then the “Use NN Steering”. When you hit this button, the car will disappear. This is normal. You will see one car per client that connects.

  3. Start the prediction server with the pre-trained model.

cd sdsandbox/src
python --model=../outputs/highway.h5

If you get a crash loading this model, you will not be able to run the demo. But you can still generate your own model. This is a problem between tensorflow/keras versions.

Note* You can start multiple clients at the same time and you will see them spawn as they connect.

#To create your own data and train

Generate training data

  1. Load the Unity project sdsandbox/sdsim in Unity.

  2. Create a dir sdsandbox/sdsim/log.

  3. Hit the start arrow in Unity to launch project.

  4. Hit button “Generate Training Data” to generate image and steering training data. See sdsim/log for output files.

  5. Stop Unity sim by clicking run arrow again.

  6. Run this python script to prepare raw data for training:

cd sdsandbox/src
  1. Repeat 4, 5, 6 until you have lots of training data.

Train Neural network

python --model=../outputs/mymodel.h5

Let this run. It may take a few hours if running on CPU. Usually far less on a GPU.

Run car with NN

  1. Start Unity project sdsim

  2. Push button “Use NN Steering”

  3. Start the prediction client. This listens for images and returns a steering result.

python --model=../outputs/mymodel.h5


**Note: pygame only needed if using which gives a live camera feed during inferencing.

Show all projects by tawnkramer