Reminder: My repository will all of my p5 assignments can be found here.
The end of the semester in Creative Coding meant we all were tasked with creating a final project using at least one p5.js library. After considering several ideas, I decided to make a project that played with images and sound. I learned that the p5.sound library is a great library for inputing and outputting sound files in a way that is integrated with the whole p5.js sketch.
My idea evolved to become an interface for exploring color through sound. An image, in this case my own photograph of two parrots, can be clicked on at any point on the canvas. A note will play depending on the color of the pixel that is under the mouse position when the user clicks on the canvas.
The mp3 files that I used were derived from an open database of musical instrument sounds, recorded and maintained by the University of Iowa. I chose notes played on the guitar for this project.
The final result displays my sample photograph, plays the correct note on-click and updates the color of the sample square in the upper left corner to match the last selected pixel. However, it wasn’t a complete success because of the challenges I faced when I wanted to ‘play’ the whole photo. I attempted to utilize code that iterated through all the pixels that were loaded when the photograph opens, similar to this processing example, and also related to referencing the pixels array like in the website.
There were two problems with this part of the project.
1. When I attempted to iterate through all of the pixels, using two nested loops that counted up to the number of pixels in the width and height respectively, my browser would crash. This happened even if I only printed a statement at each iteration of the loop, without playing any sound.
2. When I modified it to loop through only the first 100 pixels (50 across and 50 tall), the browser no longer crashes, but as you can see in the example below, the p5 script attempts to play all of the sounds at once instead of one at a time.
To fix the second problem, I believe I could implement a counter using millis, but I am looking for a solution within the p5.sound library. There are options for stringing sounds together into phrases and compositions which might solve this problem.
Overall, I learned a lot about not only the p5.sound library, but also a little about how sound is processed and played with scripts on all sorts of different websites. Additionally, I am inspired to expand on this project and fix the problem mentioned above, eventually also adding the ability for the user to upload their own photograph to listen to. After a full semester of Creative Coding, I feel ready to implement p5.js in my personal and work projects.
Try it here! Click anywhere to hear some colors!