Talk Funnel

Image: Wikipedia

In the early 1990’s, I got involved with a local arts group called George Coates Performance Works (GCPW). The productions combined opera, theater, music, live projection, videoconferencing, and a lot of new technology — mostly donated by companies like Apple and SGI, but also some lab oddities scavenged around Silicon Valley.

In one of the more popular pieces — Box Conspiracy — an actor stood on stage and used a pen-tablet to interactively control a large ‘all-seeing’ 3-D eye that was projected on the screen behind him. Making this work involved hooking up a prototype pen tablet to a wireless modem that was connected to an SGI workstation back in the control room. The SGI used the pen inputs to drive a custom animation program that generated left-eye/right-eye images in 3D. This was then projected onto the screen behind the actor and the audience (wearing polarized 3-D glasses) got to experience the synthesis of live and generated worlds in real 3-D space.

This was way before pen tablets and way, waaaaay before wireless was ubiquitous. GCPW productions were often a combination of artistry and super-high tech. On any given day, opera singers and actors mingled with 3-D designers from LucasArts and NASA engineers. It was an interesting experience.

From a technical point of view, it all had to be hand-rigged, with custom client, networking, and 3-D rendering software, along with private protocols synchronized across multiple platforms. The effect was pretty cool and the show ended up being pretty popular (less due to the special effects, I think, than its timely lampooning of the ‘5000 interactive TV channels’ meme).

The pen-tablet interface always bothered me a little. The thing we really wanted to do was to use the actor’s body as the input to the software and synchronize the movement with the live 3-D scene projected around him. There were a number of technical challenges. The first was how to establish a person’s body position without hindering their movement.

Once we had that, we would need to figure out where the arms and feet stood relative to the trunk. There were a number of Rube Goldberg devices out there, mostly for gaming and 3DVR applications that involved wearing body-suits or harnesses, with heavy wires tracking behind them (the sort of technology later used to generate the fluid motions of the Gollum in Lord of the Rings). The state-of-the-art in motion tracking back then was the Data Glove. Full-body tracking was a pretty tall order, but it seemed like it was an easier problem than tackling the body’s position.

The first step was to see if we could snip the wires. The wireless modem we used with the pen tablet was a lab prototype donated by a hardware vendor (can’t remember their name, but I don’t think they ever commercialized it). It was pretty lightweight and could be clipped to the waist and it could run for a few hours on a single battery charge. All it did was act like a standard RS-232 serial port (remember those?) so you could use it like a pretty long cable and transmit simple position data. The software at the other end could read it and use it to render the imagery.

But then how to establish the actor’s position and movement?

The solution I ended up pursuing (but ultimately abandoning as unworkable) was to combine a gyroscope with a set of accelerometers and a touch-sensor. The touch sensor could establish a ‘home’ position on the stage while the gyroscope and accelerometer (carried by the actor) could record relative position and motion from that point on. A small computer (the pen-tablet?) could gather the data and transmit it through the modem back to the SGI. The whole contraption could be fitted inside a harness or small backpack.

I tried finding out as much as I could about gyroscopes and accelerometers (this is pre-Google). Not much there. Fortunately, there was a big electronics and embedded system trade conference coming up in San Jose, so I headed down to see what I could find.

The problem that ultimately sank the whole plan was a condition called Gimbal Lock. A gyroscope is made of a rotor spinning inside three inter-locking rings, each connected to another via a tiny pin. Sensors on each ring can calculate relative motion in X/Y/Z (aka pitch, roll, and yaw) coordinates. The problem is with those tiny pins. If you move the whole contraption far enough, the spinning rotor eventually comes pretty close to where the pins sit. If you’re not careful and the ring actually hits the pin, the angular velocity of the rotor will send the whole thing spinning around wildly and everything goes bat-shit crazy.

In airplanes, where this technology was mostly used, gyroscopes were often used as part of an Inertial Navigation System and connected to the auto-pilot. If the plane made a sudden motion and the gyro hit Gimbal Lock, it was considered a Very Bad Thing.

The only way to get around Gimbal Lock was to shut the whole gyro down, wait until it stabilized then spin it back up again — a process that could take several minutes since the gyros often ran at thousands of RPMs. There were some techniques to work around this problem (like using a fourth gimbal) but this was not available in the smaller, low-powered gyroscopes.

At the conference, I remember asking several vendor reps if they could think of a solution (the booth staff in these types of conferences were always heavy-duty PhDs who could actually answer technical questions). They would ask me what the application was. I remember explaining what I was trying to do, fully expecting them to gently scoot me away to make room for the serious customers. Amazingly, they would all get excited — I like to think it reminded them why they got into engineering in the first place.

They’d sit down with pen and paper and try in-earnest to tacke the problem. Product manuals would be consulted. Charts and scientific calculators would be whipped out. But alas, the universal conclusion was that given the state of the technology at that time, there was no way to put together something that was a) cost-effective, b) not susceptible to periodic locking, c) could be carried by a human being (even in a back-pack), or d) all of the above. But they all agreed that it was a very unique application, seeing as how most of their customers bought their hardware for use in spaceships.

After a whole day of this I gave up on the idea and eventually went with the pen-tablet set-up.

What made me reminisce about all this was my tinkering today with the solid-state accelerometer built into the iPhone. It can tell when you’re moving it around all three (X/Y/Z) axes and has pretty good resolution. The iPhone also has built-in WiFi, so two things out of three I needed to solve that problem — all of which would require a backpack and a sturdy back — are built into this little hand-held device.

The only thing missing is a solid-state gyroscope and a wayback machine so I can go back and use it on that stage.