New Snow
New Snow is an interactive drawing table that investigates human interaction with a deep generative model based on Edvard Munch’s sketching practice. Through drawings with pen and paper, the user can interact with the model which will return synthetic sketches based on the input drawings in real time. The model is a reflection of the training data, and it is thus constrained to representing images within the latent space of Edvard Munch’s sketching practice.
The system works by tracking the lines the user makes on a piece of paper in real-time. A cleaned-up version of this image is sent to the machine-learning system that interprets the input-line as an Edvard Munch-like sketch. This drawing is then projected back onto the paper and creates an ongoing drawing dialogue with the user. Since the model is built solely on Munch’s sketches the system will draw the user to draw in a similar way. This educates the attention of the users to the salient visual aesthetics of Munch’s practice while they play with the system.
The project was shown at the Interactivity track of CHI ’23 in Hamburg, Germany and you can read more about it in the extended abstract Exploring a Digital Art Collection through Drawing Interactions with a Deep Generative Model.
This project was created with the support of René Haas (Machine learning pipeline design), Halfdan Hauch Jensen (Physical prototype), Anders Sundnes Løvlie (PhD supervisor and co-author), MUNCH (Funding, conceptual support and data curation), research division of Random International (Tracking system concept)
Video teaser
A user drawing with the system.
These four panels illustrate the image generation process. Top left is the raw input image. Top right is the cleaned input image used for prompting the DGM. Bottom left is the image returned from the DGM. Bottom right is the composite image that is projected back on the paper.