Hello, World!

A Deep Neural Network (DNN) opening its eyes for the first time, and trying to understand what it sees as it trains in realtime on a live camera feed.

“Hello, World!” is a custom software that performs realtime DNN (CNN-VAE) training on a live video feed, while allowing a user to manipulate a number of hyperparameters in realtime (e.g. using faders on a midi controller). These hyperparameters include learning rate, momentum, gradient clipping thresholds, optimizer function, loss function and many more. Observing the results of these hyperparameter manipulations in realtime, help build a qualitative understanding of how they impact the training process.

Furthermore, it turns out realtime continuous manipulation of hyperparameters while training has great performative potential, turning the training DNN into a Deep Visual Instrument. With realtime continuous feedback, it’s relatively trivial (and fun) to find hyperparameter configurations which provide “stable oscillation points”, or “controlled explosions”.

Finally as the network “learns” more about its environment, it starts projecting its expectations the incoming signal.

For a much deeper conceptual and technical analysis, please see Chapter 4 of my PhD Thesis "Deep Visual Instruments: Realtime Continuous, Meaningful Human Control over Deep Neural Networks for creative expression" (Memo Akten, 2021) available online :)

This neural network has not been trained on anything. It starts off completely blank (i.e. randomly initialized). It is opening its eyes for the first time and trying to ‘understand’ what it sees. In this context, ‘understanding’ means trying to find patterns, trying to find regularities in what it’s seeing, so that it can efficiently compress and organise incoming information in context of its past experience and make accurate and efficient predictions of the future.

But the network is training in realtime, it’s constantly learning, and updating its ‘filters’ and ‘weights’, to try and improve its compressor, to find more optimal and compact internal representations, to build a more ‘universal world-view’ upon which it can hope to reconstruct future experiences. Unfortunately, the network also ‘forgets’. When too much new information comes in, and it doesn’t re-encounter past experiences, it slowly loses those filters and representations required to reconstruct those past experiences.

These ideas are not behaviours which I have explicitly programmed into the system. They are characteristic properties of deep neural networks which I’m exploiting and exploring.

Originally created in 2017, minted in 2022. Memo Akten, memo.tv