When it comes to grokking information, we learn with our eyes but we
feel with our ears. There is something in the way we’re wired that lets
music and sound tap straight into our emotional core. Hollywood
understands this very well; the amount of energy devoted to
engineering and composing the sounds we hear is a significant part of
a film’s budget. If you have any doubt whether this pays off, watch
“Jaws” again and notice the effect of the strong base line in driving
the final scene’s climax.Web designers know this, but limit the use
of sound on Web pages for other reasons. Sound consumes more bandwidth than
image files. One minute of an MP3 sound file takes about a
megabyte of disk space. That’s less than a movie, but still very expensive compared
to a nice, short GIF. Sounds can also be annoying in quiet places
like offices and libraries. If the sound isn’t adding more than
glitz to a page, there’s little reason to keep it.
This is too bad because we’re avoiding an important channel to our
brain. If there were some way to put sound to work without wasting bandwidth
and annoying people, then it might be worth adding to the pantheon of
Web services offered through sites.
There is plenty of academic literature on this topic filed away under
buzzwords like “Auditory Display” or
“Sonification.” These concepts are even starting to make their way into commercial products.
Accentus is just one company
trying to commercialize the space. If you have information to convey,
you can send it by changing the pitch, tempo, timbre, attack,
volume, or any one of a surprisingly large number of parameters in the
world of music. The challenge to doing this well is as much artistic as
technical, just as there’s a big difference between a brilliant musical
composition and a flat one.
As an experiment, I created code that plays a short composition
based upon the movement of the U.S. stock market. This was inspired by
the colored
orb that changes from green to red reflecting the change in stock
prices. I tried to recreate the way that the orb would convey useful
information in the background. The chimes in my example play
every 15
minutes or so, letting people getting a feeling for the market without
paying too much attention to it.
The challenge is doing this with as much efficiency as possible. I
chose a Java applet embedded in a Web page because the Java Sound API
is quite rich and embedded in all versions after Java 1.3. The applets
can play MIDI music if they run in an up-to-date environment.
Another
pretty good solution is Macromedia’s Flash because SWF files can also
contain embedded MIDI information. This may be the best solution for
some environments where Flash is used extensively, but I’m a programmer,
not a graphic designer. QuickTime also includes a MIDI synthesizer, so
it might make sense for some applications.
The Java world, though, struck me as the best because it comes with a
solid, full-featured programming
language, an efficient distribution mechanism, a wide range of open
source tools, and a well-engineered music API. It also seems
likely that Java will get
wider distribution now that Sun has won many of its legal
battles with Microsoft. It may not be perfect, but nothing is.
Architecture
There are two parts to my system: the server application and the
applet at the client. Information about the state of the market
travels between them formatted as an HTML file. The server
application puts together a summary every few minutes and writes it to
the file system. The client requests the file via HTTP, and Apache’s
static system handles the request. The current version has no servlet.
The dynamic work is done by a crontab daemon running a Java job every n
minutes.
The size of the HTML file is minimal. The current version is several
hundred bytes, but it could be made much smaller by stripping away
some of the information that makes it readable. The current version
includes HTML to display the data in a browser, a technique that
makes it a bit easier to debut. It also includes longer descriptions of
the data. These could be embedded in the client to save space in a
production implementation.
Here’s a sample copy of the file:
<html><head></head><body>
Current as of Mon Jun 23 10:02:27 EDT 2003
DJ Aerospace=-1.11%
DJ Automobile=-1.35%
DJ Biotech=-1.16%
DJ Chemicals=-1.44%
DJ Construction=-0.40%
DJ Financial=-0.70%
DJ Healthcare=-1.04%
DJ Energy=-0.46%
DJ Titan Retail=-1.03%
</body></html>
The applet is also nothing fancy. A production version would let the
user customize the tempo, timbre, instrument, and
information included in the sound. This version just displays the raw
information and plays the sound. No neat controls are
available.
If you want to roll your own applet, the best place to begin is with
Sun’s basic demonstration.
While the API has many features, the steps are pretty basic:
- You begin by creating a Sequence
object. - Each Sequence contains a number of Tracks.
I use only one, but you may want to have multiple instruments in
your composition. - Notes are created by adding two MidiEvent
objects to a Track. The first one starts a note and the second one ends
it. - The Sequence can be played by handing it to a Sequencer.
The standard one for a system can be found by calling
MidiSystem.getSequencer(). - The Synthesizer
object actually plays the notes but you can’t just give the Sequence to
it. Well, you could, but there’s a special object known as a Transmitter
designed to make your life a bit easier. You give the Sequence to the
Transmitter, which parcels it out to the Synthesizer. Most of the work
is actually hooking up the APIs so the data flows in the right
direction. You can see the code in Applet to understand how to do it.
That’s it. The API has a great deal of flexibility, although it’s not
clear how much of it is available in each platform. Some Windows
include different MIDI synthesizers and they may or may not offer all
of the features.
The music
Composing music is an art, and I don’t pretend that this version is
anything great. I simply reworked a piano exercise with chords to
represent the flow of the stock market. If the market is going up, the
notes get higher. If the market is going down, the notes get lower.
I thought of using a scale, but that seemed a bit too simple. This
version uses notes from the C major chord and arranges them in a
progression of chords. An uptick plays the three notes going up: C
then E then G. A downtick plays them going down: G then E then C. To
add some variety, I invert the chords after each tick. Two upticks in a
row would be played CEG pause EGC. The second C is one octave higher.
Down downticks in a row would be played GEC pause ECG. The second G
would be one octave lower.
This technique produces a composition that reflects the tone of the
market. If all of the prices are rising, then the progression of chords
will get higher and higher. A mixed market will stay in the center of
the range. A falling one will generate a song that plunges into bass
notes.
This approach only scratches the surface of what can be done. It might
be neat to increase the tempo as well as the pitch if the prices are
climbing. Or maybe the tempo could indicate volume and the pitch could
indicate price. Many other details could be encoded in the other
parameters, although it’s not clear how well the human brain could
process them. Perhaps some composers will become interested in the area
and try to create algorithms that generate beautiful and instructive
music.
Future work
There are a number of different ways this concept can be extended and
improved. Giving the applet the ability to change the tempo of the
music and the instrument would be a simple way to provide more control.
A better solution might let the user pick and choose which stock prices
to include in the song.
The server could also avoid some of the problems with compatability by
rendering the song into different files. Unfortunately the Java Sound
API does not make it easy to convert a MIDI file into a WAV or AU file
despite the fact that the API includes the ability to play and record
MIDI, WAV, and AU files. Some of the pipes just don’t connect for some
reason. One Sun employee suggested with some chagrin that you can
connect line-out to line-in with a wire.
The server could also spit out pure MIDI files in the right format for
QuickTime and other players. If your compostions aren’t too involved or
challenging, the MIDI files may even be smaller than the text files
used in this example. The user, though, will have little control over
the music except perhaps to set the tempo. The applet can change the
composition and customize it for the user.
Peter Wayner ‘s latest books include Translucent
Databases and Disappearing Cryptography.
