Learn about Linux
Download Linux
Get Linux helpSpecial Offers
Feature
Using sound to display trends on a Web site
By on
July 23, 2003 (8:00:00 AM)
Share
Print
Comments
- by Peter Wayner -
When it comes to grokking information, we learn with our eyes but we feel with our ears. There is something in the way we're wired that lets music and sound tap straight into our emotional core. Hollywood understands this very well; the amount of energy devoted to engineering and composing the sounds we hear is a significant part of a film's budget. If you have any doubt whether this pays off, watch "Jaws" again and notice the effect of the strong base line in driving the final scene's climax.
When it comes to grokking information, we learn with our eyes but we feel with our ears. There is something in the way we're wired that lets music and sound tap straight into our emotional core. Hollywood understands this very well; the amount of energy devoted to engineering and composing the sounds we hear is a significant part of a film's budget. If you have any doubt whether this pays off, watch "Jaws" again and notice the effect of the strong base line in driving the final scene's climax.
Web designers know this, but limit the use
of sound on Web pages for other reasons. Sound consumes more bandwidth than
image files. One minute of an MP3 sound file takes about a
megabyte of disk space. That's less than a movie, but still very expensive compared
to a nice, short GIF. Sounds can also be annoying in quiet places
like offices and libraries. If the sound isn't adding more than
glitz to a page, there's little reason to keep it.
This is too bad because we're avoiding an important channel to our brain. If there were some way to put sound to work without wasting bandwidth and annoying people, then it might be worth adding to the pantheon of Web services offered through sites.
There is plenty of academic literature on this topic filed away under buzzwords like "Auditory Display" or "Sonification." These concepts are even starting to make their way into commercial products.
Accentus is just one company trying to commercialize the space. If you have information to convey, you can send it by changing the pitch, tempo, timbre, attack, volume, or any one of a surprisingly large number of parameters in the world of music. The challenge to doing this well is as much artistic as technical, just as there's a big difference between a brilliant musical composition and a flat one.
As an experiment, I created code that plays a short composition based upon the movement of the U.S. stock market. This was inspired by the colored orb that changes from green to red reflecting the change in stock prices. I tried to recreate the way that the orb would convey useful information in the background. The chimes in my example play every 15 minutes or so, letting people getting a feeling for the market without paying too much attention to it.
The challenge is doing this with as much efficiency as possible. I chose a Java applet embedded in a Web page because the Java Sound API is quite rich and embedded in all versions after Java 1.3. The applets can play MIDI music if they run in an up-to-date environment.
Another pretty good solution is Macromedia's Flash because SWF files can also contain embedded MIDI information. This may be the best solution for some environments where Flash is used extensively, but I'm a programmer, not a graphic designer. QuickTime also includes a MIDI synthesizer, so it might make sense for some applications.
The Java world, though, struck me as the best because it comes with a solid, full-featured programming language, an efficient distribution mechanism, a wide range of open source tools, and a well-engineered music API. It also seems likely that Java will get wider distribution now that Sun has won many of its legal battles with Microsoft. It may not be perfect, but nothing is.
Architecture
There are two parts to my system: the server application and the applet at the client. Information about the state of the market travels between them formatted as an HTML file. The server application puts together a summary every few minutes and writes it to the file system. The client requests the file via HTTP, and Apache's static system handles the request. The current version has no servlet. The dynamic work is done by a crontab daemon running a Java job every n minutes.
The size of the HTML file is minimal. The current version is several hundred bytes, but it could be made much smaller by stripping away some of the information that makes it readable. The current version includes HTML to display the data in a browser, a technique that makes it a bit easier to debut. It also includes longer descriptions of the data. These could be embedded in the client to save space in a production implementation.
Here's a sample copy of the file:
If you want to roll your own applet, the best place to begin is with Sun's basic demonstration. While the API has many features, the steps are pretty basic:
The music
Composing music is an art, and I don't pretend that this version is anything great. I simply reworked a piano exercise with chords to represent the flow of the stock market. If the market is going up, the notes get higher. If the market is going down, the notes get lower.
I thought of using a scale, but that seemed a bit too simple. This version uses notes from the C major chord and arranges them in a progression of chords. An uptick plays the three notes going up: C then E then G. A downtick plays them going down: G then E then C. To add some variety, I invert the chords after each tick. Two upticks in a row would be played CEG pause EGC. The second C is one octave higher. Down downticks in a row would be played GEC pause ECG. The second G would be one octave lower.
This technique produces a composition that reflects the tone of the market. If all of the prices are rising, then the progression of chords will get higher and higher. A mixed market will stay in the center of the range. A falling one will generate a song that plunges into bass notes.
This approach only scratches the surface of what can be done. It might be neat to increase the tempo as well as the pitch if the prices are climbing. Or maybe the tempo could indicate volume and the pitch could indicate price. Many other details could be encoded in the other parameters, although it's not clear how well the human brain could process them. Perhaps some composers will become interested in the area and try to create algorithms that generate beautiful and instructive music.
Future work
There are a number of different ways this concept can be extended and improved. Giving the applet the ability to change the tempo of the music and the instrument would be a simple way to provide more control. A better solution might let the user pick and choose which stock prices to include in the song.
The server could also avoid some of the problems with compatability by rendering the song into different files. Unfortunately the Java Sound API does not make it easy to convert a MIDI file into a WAV or AU file despite the fact that the API includes the ability to play and record MIDI, WAV, and AU files. Some of the pipes just don't connect for some reason. One Sun employee suggested with some chagrin that you can connect line-out to line-in with a wire.
The server could also spit out pure MIDI files in the right format for QuickTime and other players. If your compostions aren't too involved or challenging, the MIDI files may even be smaller than the text files used in this example. The user, though, will have little control over the music except perhaps to set the tempo. The applet can change the composition and customize it for the user.
Peter Wayner 's latest books include Translucent Databases and Disappearing Cryptography.
This is too bad because we're avoiding an important channel to our brain. If there were some way to put sound to work without wasting bandwidth and annoying people, then it might be worth adding to the pantheon of Web services offered through sites.
There is plenty of academic literature on this topic filed away under buzzwords like "Auditory Display" or "Sonification." These concepts are even starting to make their way into commercial products.
Accentus is just one company trying to commercialize the space. If you have information to convey, you can send it by changing the pitch, tempo, timbre, attack, volume, or any one of a surprisingly large number of parameters in the world of music. The challenge to doing this well is as much artistic as technical, just as there's a big difference between a brilliant musical composition and a flat one.
As an experiment, I created code that plays a short composition based upon the movement of the U.S. stock market. This was inspired by the colored orb that changes from green to red reflecting the change in stock prices. I tried to recreate the way that the orb would convey useful information in the background. The chimes in my example play every 15 minutes or so, letting people getting a feeling for the market without paying too much attention to it.
The challenge is doing this with as much efficiency as possible. I chose a Java applet embedded in a Web page because the Java Sound API is quite rich and embedded in all versions after Java 1.3. The applets can play MIDI music if they run in an up-to-date environment.
Another pretty good solution is Macromedia's Flash because SWF files can also contain embedded MIDI information. This may be the best solution for some environments where Flash is used extensively, but I'm a programmer, not a graphic designer. QuickTime also includes a MIDI synthesizer, so it might make sense for some applications.
The Java world, though, struck me as the best because it comes with a solid, full-featured programming language, an efficient distribution mechanism, a wide range of open source tools, and a well-engineered music API. It also seems likely that Java will get wider distribution now that Sun has won many of its legal battles with Microsoft. It may not be perfect, but nothing is.
Architecture
There are two parts to my system: the server application and the applet at the client. Information about the state of the market travels between them formatted as an HTML file. The server application puts together a summary every few minutes and writes it to the file system. The client requests the file via HTTP, and Apache's static system handles the request. The current version has no servlet. The dynamic work is done by a crontab daemon running a Java job every n minutes.
The size of the HTML file is minimal. The current version is several hundred bytes, but it could be made much smaller by stripping away some of the information that makes it readable. The current version includes HTML to display the data in a browser, a technique that makes it a bit easier to debut. It also includes longer descriptions of the data. These could be embedded in the client to save space in a production implementation.
Here's a sample copy of the file:
The applet is also nothing fancy. A production version would let the user customize the tempo, timbre, instrument, and information included in the sound. This version just displays the raw information and plays the sound. No neat controls are available.
<html><head></head><body>
Current as of Mon Jun 23 10:02:27 EDT 2003
DJ Aerospace=-1.11%
DJ Automobile=-1.35%
DJ Biotech=-1.16%
DJ Chemicals=-1.44%
DJ Construction=-0.40%
DJ Financial=-0.70%
DJ Healthcare=-1.04%
DJ Energy=-0.46%
DJ Titan Retail=-1.03%
</body></html>
If you want to roll your own applet, the best place to begin is with Sun's basic demonstration. While the API has many features, the steps are pretty basic:
- You begin by creating a Sequence
object.
- Each Sequence contains a number of Tracks. I use only one, but you may want to have multiple instruments in your composition.
- Notes are created by adding two MidiEvent objects to a Track. The first one starts a note and the second one ends it.
- The Sequence can be played by handing it to a Sequencer. The standard one for a system can be found by calling MidiSystem.getSequencer().
- The Synthesizer object actually plays the notes but you can't just give the Sequence to it. Well, you could, but there's a special object known as a Transmitter designed to make your life a bit easier. You give the Sequence to the Transmitter, which parcels it out to the Synthesizer. Most of the work is actually hooking up the APIs so the data flows in the right direction. You can see the code in Applet to understand how to do it.
The music
Composing music is an art, and I don't pretend that this version is anything great. I simply reworked a piano exercise with chords to represent the flow of the stock market. If the market is going up, the notes get higher. If the market is going down, the notes get lower.
I thought of using a scale, but that seemed a bit too simple. This version uses notes from the C major chord and arranges them in a progression of chords. An uptick plays the three notes going up: C then E then G. A downtick plays them going down: G then E then C. To add some variety, I invert the chords after each tick. Two upticks in a row would be played CEG pause EGC. The second C is one octave higher. Down downticks in a row would be played GEC pause ECG. The second G would be one octave lower.
This technique produces a composition that reflects the tone of the market. If all of the prices are rising, then the progression of chords will get higher and higher. A mixed market will stay in the center of the range. A falling one will generate a song that plunges into bass notes.
This approach only scratches the surface of what can be done. It might be neat to increase the tempo as well as the pitch if the prices are climbing. Or maybe the tempo could indicate volume and the pitch could indicate price. Many other details could be encoded in the other parameters, although it's not clear how well the human brain could process them. Perhaps some composers will become interested in the area and try to create algorithms that generate beautiful and instructive music.
Future work
There are a number of different ways this concept can be extended and improved. Giving the applet the ability to change the tempo of the music and the instrument would be a simple way to provide more control. A better solution might let the user pick and choose which stock prices to include in the song.
The server could also avoid some of the problems with compatability by rendering the song into different files. Unfortunately the Java Sound API does not make it easy to convert a MIDI file into a WAV or AU file despite the fact that the API includes the ability to play and record MIDI, WAV, and AU files. Some of the pipes just don't connect for some reason. One Sun employee suggested with some chagrin that you can connect line-out to line-in with a wire.
The server could also spit out pure MIDI files in the right format for QuickTime and other players. If your compostions aren't too involved or challenging, the MIDI files may even be smaller than the text files used in this example. The user, though, will have little control over the music except perhaps to set the tempo. The applet can change the composition and customize it for the user.
Peter Wayner 's latest books include Translucent Databases and Disappearing Cryptography.
Comments
on Using sound to display trends on a Web siteThere are no comments attached to this item.
