What kind of information can a program recieve from input audio, and how would the code for that stuff look?
Is there an article anywhere about audio processing, (preferably including the limitations of cheap microphones), as well as, perhaps, a refresher on what each thing being determined means?
I'm thinking that it could be a really interesting area in interface design because of how much information there is in sound (stereo recording?) which can be gathered transparently. Voice recognition obviously doesn't work great, but that's just a really ugly implementation that tries too hard, actively seeking a shape that something as rigid as a computer system can understand. Things do not always need such strict shape -- particularly games, because the ultimate output of those is to a human in the present.
Potentially, (in the ideal fantasy world), the stuff could be as cool and fun as multitouch, although a few thousand dollars cheaper.
I suppose your touching on what the DS does with a limited microphone, where even a handheld 'learns' words that a user can re-iterate. It does have intersting gaming considerations....
You touched on stereo input, would this be going as far to do amplitude comparisons (with stereo separated mic(s)) to gauge how close/far someone was... ? Probably not, but what about using amplitude of sound input to determine someone's position....
I'm thinking a few things, actually.
The first thing that comes to mind is of course a RockBand killer. The next thing is a puzzle or platformer that heavily utilizes sound as a method of control. (Another thing that comes to mind, when thinking of such an abstract concept, is Daniel's blobby effects stuff).
Stereo input would be good for the rockband killer, of course.
What I like about audio is that it lets the player create his own controller, just like we can get with optical multitouch. Want to make different tones with a rubber tube? Go right ahead. Feel free, however, to use a clarinet, a tuba or maybe even a fake guitar. The only thing that matters here is that the visual feedback shows what is going on, acting as a teaching mechanism.
Without any software to test it, I can't be sure, but I think the right visualizations could work as suitable feedback. Subtle adjustments to the real instrument change the sound it makes in different ways, and that change is reflected by an incremental change on the screen. In the case of a RockBand killer, one sees how he lines up with the real thing and can gradually figure out the correct patterns. The actual software would be very soft-coded, adapted to any situation. It is up to the player to find the best way to fit it, however, since the game does not dictate the controller. Realistically, to match a guitar one would probably need a guitar, but may be able to hold his own with a few elastic bands in at least one of the fields. The fun comes from invention, and exists beyond just the software, which is something I think would be really amazing.
Such an interface would make the instrument become more like a controller, learned by ear and perfected by practice. No, it won't create the next Hendrix or Elvis, but it could be really fun.
The difference is that it's a cool controller, like the Wii controller in that it is a direct link to the real world; one that must be weilded, learned and customized in ways far more exciting than any d-pad and buttons ever were.
Of course, that's adaptable far beyond a flexible RockBand killer. I think a sound-based platformer could be quite intriguing, and while maybe not selling many copies, could get some recognition for accessibility ;)
All in all, rich sound input could be an interesting experiment!
you touched on a good point about accessability, sound controlled games for special needs schools is a worthwhile cause, and there is a market for it. Requirements might take some sussing out mind you....