Homework 2: (Due 2015.10.27 by noon)

In this homework, you will:

• Implement elementary image and sound processing
• Read in your own picture and sound files
• Display images using so called "textures" and play back portions of your sound files using Gamma
• Treat each pixel of your image as an independent particle and rearrange these particles according to various color spaces
• Ask the question "What is the equivalent of a pixel in the sound domain?"
• Rearrange the "pixels" of the sound according to various audio metrics (energy, spectral brightness, etc.)

To start your homework, please:

• Gather some images and sounds that you like.
• Make a subdirectory of your personal repo (e.g. frank.zappa) named media (i.e. frank.zappa/media).
• Copy at least one image file to this directory. Please convert your image(s) into a format that AlloSystem supports. The documentation for the AlloCore class al::Image lists the types of images AlloSystem can read. Also, make sure each file is small: less than 2MB and under 1000x1000 pixels.
• Copy at least one sound file to this directory. Convert your sound file(s) into a format that AlloSystem supports. The documentation for the Gamma class gam::SoundFile lists the types of soundfiles Gamma can read. Also, make sure each file is small: less than 2MB and under 10 seconds.
• Commit and push these files with the familiar sequence: git add ..., git commit -m ..., git push, git fat push.
• Modify the homework 2 starter codes to use your own image(s) and sound(s).

Turn in the following:

1. hw2/p1.cpp: Sorting pixels into color spaces.

   • The problem 1 starter code treats each pixel as a point that can be placed arbitrarily in 3D space. Each pixel's default location is determined by its position in the original image file.

   • Consider an RGB color cube positioned in front of the camera. What size should it be and where should it be centered? Find each pixel's position in this color space and store it in a Vec3f.

   • Animate the pixels between these two spaces. Pressing the 1 key should move the pixels over the course of one second to their original positions, and pressing 2 should move them over the course of one second to their positions as sorted in the RGB color space.

   • Do the same as above but with 3 sorting pixels into HSV space. Note that HSV is a cylinder with the hue represented as an angle - you will need a tiny bit of trigonometry to convert from HSV coordinates to 3D position.

   • Pick a 2D color space and make the 4 key sort the pixels into that space. It can be a trivial projection of RGB or HSV down to 2D, or any other space of your choosing. Display this 2D space on the same plane as for 1. Here's an example of a 2D color space.

2. hw2/p2.cpp: Sorting sound grains according to their properties

   • The problem 2 starter code reads in a sound file, divides it into grains using rectangular windows, finds the RMS of each grain, sorts the grains in increasing order of RMS, then synthesizes a new sound by reassembling the grains from quiet to loud. Modify it to load in your own sound file and experiment with different window sizes.

   • Add a proper window function to each grain. The window function functions as an amplitude envelope that smoothly fades each grain in and out. Make a plain old C float array whose length is (at least) the size of your grains, e.g., float win[1000]. Use one of the functions in the gam::tbl namespace that fills an array with a window, such as hamming. Apply this window to each grain during granular synthesis. We recommend using very long grains (e.g., 1-second) when testing, so you can easily see and hear whether each grain is being windowed properly.

   • Overlap the grains. The window function avoids the clicking caused by abruptly starting and stopping each grain, but now it causes the overall synthesized sound to fade in and out over the course of each grain. The solution is to overlap the grains, so that at any given time the previous grain is fading out while the next grain is fading in. In DSP lingo we say the hop size is half of the window size, or (equivalently) that the windows overlap by a factor of two. Make the hop size a variable (so you can easily change it by modifying just a single number in your program) and make the granular synth have an adjustable hop size. If the hop size is half the window size and you're using hamm or hanning windows then the resulting sound should be steady, without the warbling. If the hop size is greater than the window size then you should hear each grain individually, with silent gaps between them. If the hop size is much less than half the window size then you should hear many overlapping grains at any time.

   • Analyze the number of zero crossings in each grain. The current code analyzes the RMS of each grain, so you can follow the same basic structure. You may assume that no audio sample's value is ever exactly zero (though that's not strictly true); otherwise you need to think carefully about zero-valued samples as a special case. Sort the grains into increasing or decreasing order of zero crossings.

3. hw2/p3.cpp: Do something cool. Here are some possibilities, or you can use your own idea.

   • Make the motion of the pixel cubes more physical, such as by simulating their motion with a damped spring "pulling" them to their new position, and/or by detecting and somehow handling collisions as they move.

   • Make another spatial layout for the pixel cubes. For example, X and Y could come from the original image with Z some function of pixel brightness.

   • Cause the 3D representations of the pixel cubes to rotate automatically, to make the 3D more apparent. The easiest way will be to rotate the camera in a circle around the pixels, always pointing at the center.

   • Divide the input sound into grains at zero crossings (or some other detectable event in the audio) rather than making all grains have the same size.

   • Somehow tie the visual aspects of problem 1 together with the audio aspects of problem 2.