by tawnkramer

This provides a sandbox simulator for training a self-driving car. This uses Unity for simulation and Python with Keras and Tensorflow for training. Recently updated to work on Python 3.4+ and Keras 2+

( Crawled 24 minutes ago )


Self Driving Car Sandbox



Use Unity 3d game engine to simulate car physics in a 3d world. Generate image steering pairs to train a neural network. Uses comma ai training code with NVidia NN topology. Then validate the steering control by sending images to your neural network and feed steering back into the simulator to drive.

Some videos to help you get started

Training your first network


World complexity


Creating a robust training set



You need to have Unity installed, and all python modules listed in the Requirements section below.


1) Start the prediction server with the pre-trained model.

cd sdsandbox/src
python highway

2) Load the Unity project sdsandbox/sdsim in Unity. Double click on Assets/Scenes/main to open that scene.

3) Hit the start button to launch. Then the "Use NN Steering".

To create your own data and train

Generate training data

1) Load the Unity project sdsandbox/sdsim in Unity.

2) Create a dir sdsandbox/sdsim/log.

3) Hit the start arrow in Unity to launch project.

4) Hit button "Generate Training Data" to generate image and steering training data. See sdsim/log for output files.

5) Stop Unity sim by clicking run arrow again.

6) Run this python script to prepare raw data for training:

cd sdsandbox/src
python --clean

7) Repeat 4, 5, 6 until you have lots of training data. 30gb+ is good. On your last run, prepare a validation set:

python --validation --clean

Train Neural network

python mymodel

Let this run. It may take 12+ hours if running on CPU.

Run car with NN

1) Start the prediction server. This listens for images and returns a steering result.

python mymodel

2) Start Unity project sdsim

3) Push button "Use NN Steering"


Note: also works with Python 3.5+. But you will need to train your own models. The stock models will not load. *Note: pygame only needed if using which gives a live camera feed during inferencing.

You can install requirements with pip

pip install -r requirements

Only tensorflow should be done manually. try:

pip install tensorflow

or if you have the gpu card and libraries installed:

pip install tensorflow-gpu


Tawn Kramer, Riccardo Biasini, George Hotz, Sam Khalandovsky, Eder Santana, and Niel van der Westhuizen