How Google Uses Machine Learning To Turn Paint Into Sound

March 9, 2021

2 min read

A machine learning system lets visitors to a new art exhibition experience the sounds of colours through the artist Kandinsky’s eyes.
The abstract art pioneer had synaesthesia, a condition that can make people hear sounds when looking at colours and shapes.

Have you ever heard colours?

As part of a new exhibition, the worlds of culture and technology collide, bringing sound to the colours of abstract art pioneer Wassily Kandinsky.

Kandinsky had synaesthesia, where looking at colours and shapes causes some with the condition to hear associated sounds. With the help of machine learning, virtual visitors to the Sounds Like Kandinsky exhibition, a partnership project by Centre Pompidou in Paris and Google Arts & Culture, can have an aural experience of his art.

An eye for music

Kandinsky’s synaesthesia is thought to have heavily influenced his painting. Seeing yellow summoned up trumpets, evoking emotions like cheekiness; reds produced violins portraying restlessness; while organs representing heavenliness he associated with blues, according to the exhibition notes.

Virtual visitors are invited to take part in an experiment called Play a Kandinsky, which allows them to see and hear the world through the artist’s eyes.

a picture of the 1925 painting 'Yellow, Red, Blue' by Kandinsky — Kandinsky’s synaesthesia is thought to have heavily influenced his 1925 painting Yellow, Red, Blue.

Image: Guillaume Piolle/Wikimedia Commons

In 1925, the artist’s masterpiece, “Yellow, Red, Blue”, broke new ground in the world of abstract art, guiding the viewer from left to right with shifting shapes and shades. Almost a century after it was painted, Google’s interactive tool lets visitors click different parts of the artwork to journey through the artist’s description of the colours, associated sounds and moods that inspired the work.

But Google’s new toy is not the only tool developed to enhance the artistic experience.

Artist Neil Harbisson has developed an artificial way to emulate Kandinsky by turning colours into sounds. He has a rare form of colour blindness and sees the world in greyscale. But a smart antenna attached to his head translates dominant colours into musical notes, creating a real-world soundtrack of what’s in front of him. The invention could open up a new world for people who are colour blind.

This article is republished from WeForum.