Latest

Google I/O 2015 Audio Experiment

My name is Thomas Reynolds. Five years ago, I stumbled across the Google I/O 2011 Experiment, which was an incredibly fun HTML5 Canvas countdown by some little agency in Portland, Oregon. I reached out to Instrument and told them that things like that are exactly what I want to be building every single day. Luckily for me, I got the job and I've been able to spend the last 4 years working on I/O Experiments, including 2012, 2013 and 2014.

This year, we took the color, interactions and motion of Google's material design and envisioned how they would sound.

Every year we want to out-do ourselves. This year, we took the color, interactions and motion of Google's material design and envisioned how they would sound. We built a WebGL and Web Audio experiment that overlays the conference website (designed by us, but built by Google's team), converting all the normal page elements into musical instruments and visualizers.

The Google I/O Developer audience wants to be blown away by new technology on the web, but they also want to dig in and see how everything works. This means the I/O experiments are a little different. For one, every year we Open Source the experiment to allow developers to explore and even modify it. In the past, developers have taken it upon themselves to build massive functional Rube Goldberg machines that look like pixel art logos. They've discovered "commented out code" which unlocks new functionality. And, they've tracked down Chrome Developer Console jokes in code, rather than actually wait for them to appear during the experience.

This year is no different; so, as we were building the experiment, we had to remember that obfuscation is not always the best approach. We built a single, serializable data structure that contains as much visual and interactive information as we could fit in. Then, we made sure developers could save their state to a URL and reload it on another device. Finally, we posted the encoding document on GitHub to allow developers to see the data structure, tweak it, and hopefully share new variations. 6 drums not enough for you? Feel free to add 100. Want to make a picture instead of playing the guitar? You can set up a massive pixel grid. Want to learn way too much about Hexagon math? Well, get ready. Want to activate Cat Mode? Well, you'll just have to figure that out on your own.

6 drums not enough for you? Feel free to add 100. Want to make a picture instead of play the guitar? You can setup a massive pixel grid.

The data serializer stores nested data in a query-string-optimized format. Some characters are converted or escaped when placed in a query string and there is a limit to the lengths browsers will allow, so we wanted to optimize for shortened strings. The safe characters ended up being: (, ), _. We reduced our data into arrays of numbers, then converted square brackets into parenthesis and commas into underscores. Finally, all numbers were converted in hexadecimal strings. The resulting string for the default state looks like:

The experiment can overlay any site on the web. Once we Open Source it, we're secretly hoping someone will create a Chrome Extension to turn other material designed websites into musical experiments.

#io15

Written by Thomas Reynolds, Technical Director

Related