The International Symposium on Augmented Reality (ISAR '01) is taking place at the end of this month at Columbia University in New York. Since visual Augmented Reality is virtual 3D that is overlaid onto the real world and is often transmitted via the Internet or the Web, we decided to take a closer look at this fascinating field.

First we will give you an overview of Augmented Reality along with some examples of what is being done in AR and finally we will give you some tools and resources for learning more about AR.

Augmented Reality (also called Mixed Reality) refers to the use of computers to overlay virtual information onto the real world. If you look at Virtual Reality and true reality as two ends of a spectrum, Augmented Reality would fall somewhere between the two. Although Virtual Reality (VR) receives a lot of media attention, Augmented Reality (AR) may ultimately prove to be just as useful, especially with the added range of information supplied from such sources as the Internet and GPS.

Augmented Reality is really about the augmentation of human perception: supplying information not ordinarily detectable by human senses. AR supplements the real world instead of replacing it as with VR, In visual AR, by using a see-through Head-Mounted Display the user can see the real world around him, with computer graphics superimposed or composited with the real world.

A good example of a technology that uses VR, AR and true reality is the Magic Book . In the real world, a Magic Book looks like a children's storybook with colorful pages and simple text. When readers look at the same pages wearing lightweight head-mounted displays (HMD), the pictures pop off the page and come to life as three-dimensional animated virtual objects (VRML) almost like pop-up book would. You can still see the real world book, as well as everything that is going on around you but a little virtual scene sits on the page. And by simply touching the corner of the page, readers can also fly into the immersive VR world and freely explore the scene via an avatar. Several readers can gather around a single Magic Book and experience it together. Wearing HMDs, each reader can view AR scenes from their own perspective or fly into the immersive world and see each other represented as avatars in the same virtual scene. Readers that remain in the AR setting have a God's eye view of their fellow readers as miniature avatars in the virtual scene before them. The readers who have entered the immersive world can look up into the sky and see the AR readers staring down at them like giants. Magic Books are very entertaining to play with, especially with a child.

Another entertaining use of AR is the Contact Water game. Participants, wearing a see-through head-mounted display (HMD) and paddle devices on one hand, can see each other as well as a pool of virtual water that has formed on each of their hands. From a virtual pond in the center, a tiny virtual sea animal jump into each participant's hand. The dolphins or whales play and cavort and participants can cause them to jump into other hand-pools or play in their own pool by making hand motions. The animals are very life-like and fun.

But beyond entertainment, there are many practical uses for AR. Yoshinori Kobayashi's EnhancedDesk is a novel interface system that allows users to perform various kinds of tasks in a regular office environment. The key component of the Enhanced Desk is its capability to monitor user's activities using computer vision techniques. By using an infrared camera and an image processing hardware, the system can successfully identify user's fingertips quickly even in a complicated background. In addition, the Enhanced Desk can recognize small objects on a desktop by using an additional pan-tilt camera that is tracking a user's fingertip.

You can also have your desktop be anywhere you look. The Everywhere Displays use a rotating mirror to project information onto any surface in an environment. User interaction is detected by a video camera so no physical contact with any computer device is required, allowing the transformation of every surface in a space into a projected "touch screen". A video camera is used to detect hand/body activity on the projected area, so people can interact with the projected image by simply touching the surface.

Another practical use of AR is the Mobile Augmented-Reality Systems (MARS). What MARS attempts to do is not only superimpose graphics over a real environment in real-time, but also change those graphics to accommodate a user's head- and eye- movements, so that the graphics always fit the perspective all while the user is moving physically.

This is achieved using a see-through head-mounted display (with orientation tracker), a tracking system a differential GPS system and a mobile computer incorporated into one unit housed in a belt-worn device that wirelessly relays information to an HMD display. This allows the user to move freely and, for instance, take a tour of a university campus wearing MARS and receive additional information about the buildings they are passing.

Touch, hearing, and even smell are also being augmented with great success.

Haptic feedback companies like SenseAble Technologies augment the sense of touch to do many thing, including allowing the user to do freeform 3D modeling on the computer.

An even more interesting use of SenseAble's haptic feedback devices are tools like the NanoManipulator DP-100, an interactive visualization and control system that enhances the power of scanning probe microscopes. Users of this system can interactively view "nanoscale" objects such as carbon nanotubes and individual viruses in 3D, feel the surface of the structures, and interactively manipulate them. The possibilities for real-time exploration and manipulation of atomic-sized structures makes the system ideal for research and development in a wide variety of areas, including nano-electro mechanical systems, material science, physics, biochemistry and genomics.

Haptic feedback is also being used in distance surgery , in medical and in the veterinary training and even to play virtual musical instrument.

Audio AR is being used in many ways, not only to suplement visual AR but in it's own right. Guided by Voices: An Audio Augmented Reality System is one example of audio AR that could be used by the visually handicapped in the future.

Although augmented smell is still a very new technology, companies like Digiscents are working to make a business out of scent AR.

These technologies are already changing the way we interact with both the real and the virtual world, and in the future those changes will become even more dramatic.

Tools And Resources:

Jim Vallino's Augmented Reality Page

Although some of the links are a little dated, this is the best page we found for technical information on Augmented Reality.

AR Toolkit

ARtoolKit is an open source software library that has been used in Shared Space projects. The toollkit can be used to calculate camera position and orientation relative to physical markers in real time. This enables the easy development of a wide range of augmented reality applications.

The Kalman Filter

Course Notes From An Introduction to the Kalman Filter

According to many sources, the Kalman Filter is the best AR filter available for estimation of Multi-Mesurments. It is also free.