Self Driving Car Sandbox
Use Unity 3d game engine to simulate car physics in a 3d world. Generate image steering pairs to train a neural network. Uses NVidia PilotNet NN topology. Then validate the steering control by sending images to your neural network and feed steering back into the simulator to drive.
Some videos to help you get started
Training your first network
Creating a robust training set
You need to have Unity installed, and all python modules listed in the Requirements section below.
Linix Unity install here. Check last post in this thread.
You need python 3.4 or higher, 64 bit. You can create a virtual env if you like:
virtualenv -p python3 env source env/bin/activate
And then you can install the requirements.
pip install -r requirements.txt
If you have an cuda supported GPU - probably NVidia
pip install tensorflow-gpu
Or without a supported gpu
pip install tensorflow
- Start the prediction server with the pre-trained model.
cd sdsandbox python src/predict_server.py outputs/highway.h5
If you get a crash loading this model, you will not be able to run the demo. But you can still generate your own model. This is a problem between tensorflow/keras versions.
Load the Unity project sdsandbox/sdsim in Unity. Double click on Assets/Scenes/main to open that scene.
Hit the start button to launch. Then the “Use NN Steering”.
#To create your own data and train
Generate training data
Load the Unity project sdsandbox/sdsim in Unity.
Create a dir sdsandbox/sdsim/log.
Hit the start arrow in Unity to launch project.
Hit button “Generate Training Data” to generate image and steering training data. See sdsim/log for output files.
Stop Unity sim by clicking run arrow again.
Run this python script to prepare raw data for training:
cd sdsandbox python src/prepare_data.py --src=sdsim/log/*.* --dest=dataset
- Repeat 4, 5, 6 until you have lots of training data. Over 50K images for best results.
Train Neural network
python src/train.py outputs/mymodel.h5
Let this run. It may take many minutes if running on CPU. A GPU is faster, but requires you setup tensorflow-gpu.
Run car with NN
- Start the prediction server. This listens for images and returns a steering result.
python src/predict_server.py --model=outputs/mymodel.h5
Start Unity project sdsim
Push button “Use NN Steering”