Ngx by keijiro - 138


Ngx - Neural network based visual generator and mixer

Unity 2018.3.0b5Unknown LicenseUpdated 3 years agoCreated on October 2nd, 2018
Go to source

Ngx - Neural network based visual generator and mixer

Ngx is an attempt at utilizing a neural network for VJing. It implements pix2pix (image-to-image translation with cGAN) as an ad-hoc next-frame prediction model that is trained with pairs of consecutive frames extracted from a video clip, so that it can generate an image sequence for an infinite duration just by repeatedly feeding frames back. It also has functionality for mixing (crossfading) two pix2pix models that gives unexpected variation and transition to generated video.

gif gif

(The gif on the right is from a tweet by @chaosgroove.)

Vimeo - Ngx demonstration

Pre-built executables

You can download a pre-built executable from Releases page. It contains pre-trained pix2pix models with Beeple’s VJ clips.

Model Name Original clip Vimeo link
Beeple 2 redgate.v1
Beeple 3 p-crawl
Beeple 4 exhaust
Beeple 5 quicksilver
Beeple 7 TENDRIL
Beeple 8 shifting pains

The original video clips are licensed under a Creative Commons Attribution license (CC-BY 3.0). These pre-trained models are also attributed to the original author.

How to control

Ngx can be controlled with the on-screen controller or a MIDI controller.

Name Description MIDI CC
Mix Crossfading between model 1 and 2 77
Noise 1 Noise injection (low density) 78
Noise 2 Noise injection (medium density) 79
Noise 3 Noise injection (high density) 80
Name Description MIDI CC
Feedback Feedback rate 81, 82
FColor False color effect 83
Hue Shift Amount of hue shift 55, 56
Invert Color inversion 84
Name MIDI notes
Model Select 1 41, 42, 43, 44, 57, 58, 59, 60
Model Select 2 73, 74, 75, 76, 89, 90, 91, 92

The default MIDI mapping is optimized for Novation Launch Control XL: You can change model with the track buttons and control parameters with the track faders. The 7th and 8th panning knobs are used to tweak the hue shift values.

How to open the project in Unity

This repository uses Git submodules to manage dependent packages. To check these submodules out, execute git submodule init && git submodule update from the command line, or specify --recursive option when initially cloning.

After cloning the repository, .pict (pix2pix weight data) files should be manuyally downloaded and copied into the Streaming Assets directory because these files are excluded from the repository to save bandwidth and storage usage. Please download the pict file package and extract it into Assets/StreamingAssets.

How to train pix2pix models

Prerequisites: Basic experience on a machine learning framework. It’s recommended to have a look into the pix2pix tutorial from Machine Learning for Artists if you haven’t tried pix2pix or any similar GAN model.

You can train your own pix2pix model using pix2pix-tensorflow. I personally recommend using Google Colaboratory for training. It’s easy to set up, handy to use and cost effective (it’s free!).

The followings are Colaboratory notebooks I used to train the default models. These notebooks are written to use Google Drive as a dataset storage.

Frequently asked questions

What are the hardware recommendations?

I used a Windows system with GeForce GTX 1070 for a live performance. It runs at around 21 fps; this might be the minimum practical spec for the system.

Show all projects by keijiro