Threshold Wave

Working with innovative new media company 55 degrees ltd, Square Box Systems Ltd helped develop the software for the Threshold "Wave" video installation at Perth Concert Hall in Scotland.

Visitors to Perth's new state-of-the-art Concert Hall are greeted with groundbreaking interactive visual art works on the 22-screen Threshold "Wave" system.

The Threshold system was conceived of as an interactive art installation by artists Alex Hamilton and Richard Ashrowan. The installation provides a series of multi-channel digital canvases to display specially commissioned works of art in the entrance foyer of Perth's new concert hall.

The main display area is the 22-screen "Wave" itself which greets visitors when they first enter the building, together with two projectors on perpendicular walls in a corner of the foyer, and a screen in each of the toilet areas. Threshold also features an advanced underfloor sound system, arranged in different audio 'zones', and interactivity is provided by sensors including a contact that reports when the front door opens and cameras directed at different parts of the foyer to detect the level of movement and crowd activity.

Technical Details

Threshold is implemented using a bank of Apple Mac Mini computers, one feeding each 40-inch screen (manufactured by NEC). Although intended for consumer use rather than industrial applications, the Mac Mini was chosen because of its extreme compact size and low cost (should one unit ever fail it would be cheap and easy simply to swap in a replacement for example).

Who says Mac Minis aren't designed to be rack mounted? All it takes is a few metal shelves and some rubber sheeting, plus a little ingenuity - though having an air-conditioned machine room helps too! (Hardware and integration provided by Derek Kemp and Audio Visual Consultants of Edinburgh.)

In a regular "video wall" system a single video source, such as from a DVD player, is blown up to fill a large area, and for this off-the-shelf hardware solutions are readily available. By contrast, the 22 screens of the Threshold Wave are hung on the wall in an highly unconventional, linear horizontal arrangement, giving an extreme 37:1 aspect ratio (with a total of 28160x768 pixels to play with) - possibly the ultimate in "wide screen" and "high definition" television! This, coupled with the interactivity requirements, meant that a fully custom software solution was called for.

The solution adopted is relatively straightforward, but effective. A "show" is defined, consisting of various media elements (movies, still images, and "generative" objects such as coloured areas or text) which are positioned on a large virtual canvas. The position of these objects (as well as their size, transparency, and so on) can be varied over time, so you might have a background video playing on each screen while a foreground object runs up and down the Wave. (Perhaps the screens represent a giant virtual fish tank that individual fish can roam around in, with the amount of activity mirroring the number of people milling about in the foyer.)

Each of the 22 Mac Minis is identical, and each has its own copy of both the show and the media files. They all play the same show at the same time, the only thing that differs is that each computer is configured to show a different "window" on the virtual canvas. Taken together, the output from all the computers combines to show the complete canvas, while each node is responsible only for rendering a small part of it.

Going into slightly more technical detail, Threshold uses QuickTime 7 to play back media files recorded in one of many common media formats. The preferred formats are MPEG-4, DV, or M-JPEG for movies, and JPEG, PNG or TIFF for still images, though other formats are supported too. The visual elements are composited at run time using OpenGL. A 3D transformation matrix, together with an opacity measure, gives full control over how they are combined and mapped onto the canvas. The transformation may be animated, so movies and stills can be moved around over each other, stretched and scaled to different sizes, fade in and out, or be rotated and flipped so they appear to tumble in three dimensions. This behaviour is under the artist's control, either in a predetermined manner or based on interaction with the audience through the sensor inputs.

Combining many smaller media elements dynamically, rather than trying to play back one single, extremely wide, video file across the whole system, is both more efficient and more convenient. More efficient because nodes don't waste time decoding and moving around pixels that aren't displayed on that node, and more convenient because keeping the different media elements independent allows them to be controlled interactively, and also corresponds to how the artist is likely to generate a show in the first place. Not even the most expensive wide-screen HDV camera in the world shoots 28160x768 video (nor is it ever likely to!), so the show will almost inevitably be constructed from many separate elements. This approach also has other advantages, such as allowing a family of related shows to share the same media elements but combine them in different ways. The entire system appears as one continuous canvas and there is no requirement for media files to be broken up artificially just to fit screen boundaries.

A show is defined by an XML text file that lists the media elements, where on the timeline they occur, and what transformation should be applied to it (as well as how the transformation is to be animated over time). In the current version of the software, shows are hand crafted, based on common templates and some simple scripts to aid in the creation of repeated elements, but for the next version a timeline-based, drag-and-drop authoring tool is planned.

Software Components

The software is written in Java (with some C++ native code), relying on QuickTime and OpenGL to provide excellent native playback performance on quite modest hardware.

There are four separate components to the software:

The Node Playback Engine runs on each node and interprets the XML show file to play the media files that fall in its window on the canvas.
The Master Controller communicates with all the nodes using a socket protocol over the local area network, telling them which show to play, synchronising the playback, and querying the status of each node. It has a simple graphical user interface to control which show is played when and display the state of each node using traffic light status lights: green for show playing normally, amber to mean starting up or busy, and red for an error.
The Signal Processor processes the input from the cameras in the foyer and sends the results of its calculations, representing "busy-ness" of the scene, as sensor values to the Master Controller (which in turn broadcasts them to all the nodes to modify the playback of the show).
The Playback Simulator is a standalone program that artists can use to try out a show they are authoring on their own computer. For performance reasons, it's clearly not possible to play back an entire show intended for 22 computers in real time on just one machine, but several options are available to preview a show: the entire canvas can be rendered in wire-frame form, a scaled down still snapshot of the entire canvas at any given time can be shown, or a selected subset of the canvas can be rendered at near real-time.

The Master Controller allows the operator to schedule a playlist of shows and monitor the operation of the system.

In addition to this specially written software, Apple Remote Desktop is used to manage the network of Mac Mini nodes, allowing them to be administered remotely without requiring a keyboard, mouse and accessible display. It also permits media files and shows to be efficiently replicated onto all the nodes in one operation. (Copying many gigabytes of video data to each node in turn would be very cumbersome in comparison.)

The Master Controller, Signal Processor and Playback Simulator can all run on any machine with Java installed, such as a Mac or Windows PC. The Node Playback Engine uses a plug-in architecture to do the actual rendering, and there are currently two renderer implementations. One uses QuickTime 6 and JOGL (the Java OpenGL for Gaming API) and would in theory work under either Mac or Windows, though to date it has only been tested on the Mac, while the other uses the newer QuickTime 7 OpenGL Context API, which gives better performance but is only available under Mac OS X 10.4.

Conclusion

The Threshold architecture is highly flexible. Although initially designed with the 22-screen "Wave" at Perth Concert Hall in mind, the architecture is inherently scalable. Any number of nodes and screens, of any resolution, can be arranged on a canvas as required, with a simple configuration file governing the placement of the screens. Several different canvas arrangements can be defined and selected from the Master Controller, for example with or without a mullion gap between screens, with the toilet and projector screens as separate canvases or linked to the main canvas, or even optionally splitting the main canvas into two smaller areas and playing different shows on each. Various options (such as stretching, cropping or letterboxing the image) are available if a show doesn't match the aspect ratio of the canvas it's being shown on. The Threshold software is extensible, for example new renderers can be written for different types of output hardware, and the XML show syntax could be extended to support new capabilities.

The system is also designed to be robust. Once started, each node carries on playing its show without direct input from the controller, apart from an occasional 'time synch' command to ensure the playback on the different screens doesn't drift out of synchronisation relative to each other (by more than a few milliseconds). If the software on one particular node should crash for any reason, a monitor process will automatically relaunch the playback engine and quickly resume playing from the point the rest of the system has reached. Likewise, the controller software can be shut down and relaunched, on another machine even, and regain control of the running system without interrupting any show that is playing.

For further details about Threshold, please contact Russell Henderson at 55 degrees ltd.