On TV, you’ve doubtless seen those arrows and circles overlaid on and around players in a cricket match. “Now this fielder here,” says Boycott in his loovly Yorkshire accent, and a red circle appears around the fielder. “Cooms oop to long-on,” he continues, and an overlay arrow appears, pointing in the direction of the shift.
So here you have two silly examples of a field of research we’ll soon name. But a not-so-silly question: what do the balloons and the arrows have in common?
Figuring the answer requires no great stretch of the imagination. In both cases, information to augment your visual focus is being brought in: information to aid you in understanding what you’re looking at.
Now let’s stretch our imaginations a bit: if information can be overlaid, why not other things? Take your favourite FPS around with you in the real world! Overlay the game-with all its monsters-on what the “real” world is providing your eyes-buildings and all! So as you’re walking to work or college, monsters abound-one on the road, one behind your office building…
Notice we said “real” in quotes. That’s because the lines are getting blurred-the lines between reality and virtuality. We’re going to talk about “Augmented Reality,” where you’re in the real world and your experience of it is enriched in a useful way. Embedding virtual objects in reality-as in the FPS example-could be called Augmented Reality as well, but it’s more like “mixed reality.” Or should that be “Augmented Virtuality”?
The Terms-And-Definitions Mess
This is not a textbook, and we hate terms and definitions as much as you do. But we do need to get some things straight. Real Reality (hopefully) needs no explanation, but when it does, a professional philosopher needs to step in. We don’t have one in-house, so we’ll leave that out.
Virtual Reality (VR) is total immersion in a world divorced from the reality around you. You’re sitting in your room with goggles on, which make it seem exactly as though you were on Mars. There’s no overlaying of anything anywhere. Period. And that’s where we’ll leave it at in what follows.
Augmented Virtuality is concerned with systems that primarily recreate virtual environments while introducing elements of reality into the simulation. It’s a concept not too many folks are bothered with, so let that be as it is.
Now, there’s Augmented Reality (AR); that apart, let’s coin a term-Virtual Imagery Superimposed on Top of Actuality (VISTA). It’s a nice acronym, because it does mean new vistas of experience… and this is a nice place to get started.
More Real Than Real
You’ve seen it on TV, like we said: now imagine being in the stadium-somewhere in the back seats-when a cricket match is on. In the year 2020, our favourite year of the future. You’re wearing comical-looking goggles, dangling cables and all. The bowler does his run-up; you see an information titbit next to the bowler that states his name. When he delivers his ball, you see the speed in kmph. You’re seeing all this with your eyes, because your goggles are tuned in to various information sources-and they are displaying the “extra” information, the primary information being the plain scene of the ground. (Everything is information, right?) That’s AR.
Then, look at the scene below. If you haven’t been to our Fair City, you might not know the monument on the left is The Gateway Of India-so your goggles tell you that bit. You can also see whether rooms are available at the Taj just opposite, and more… That’s the vision some have for AR. Well, not the vision; it’s an example application that can aid tourists. The actual number of applications for AR are very, very many.
Although Augmented Reality has not been entirely ignored, science fiction writers tend to emphasise the Virtual Reality experience. It’s cooler
An aside: we’ve overdone the goggles. AR can also be done using Head-Mounted Displays-the way it’s being done in most research labs now. It could be a cell phone with a camera. It could even be implants in your eyes.
If you’ve been brought up normally, you’re probably wondering how AR can possibly work. Where does the extra information come from? How can it be overlaid? In fact, those happen to be two core questions. The first is to get the information; the second is to overlay it so perfectly on what your eyes see that it becomes one blended scene. Yes, there can be augmented hearing and smell and all, but let’s keep it simple and only talk about augmented vision.
A lot depends on the complexity of what is to be achieved. Think of a relatively simple application, like helping a newbie repair computer hardware. When he looks at a motherboard through his AR system, it could label the ports, slots, jumpers, and such. What this demands is an information system that knows the motherboard in its entirety, well enough to label it. Why it’s relatively simple is because the system need not know where the person is.
Extend this to the picture on the previous page. What needs to be known? Obviously, your physical location, and a database of things at that location. It’s easy to imagine the former being supplied by a GPS device. (Easy to imagine, but hard to achieve-imagine GPS to the accuracy of a millimetre!) It’s also easy to imagine the latter being supplied by a database created by a tourism company. An ad agency could supply additional information to that database-that’s how you’d get the price of the car and where it’s available.
Things could get more advanced: picture someone walking down the street wearing Guess jeans, and an ad popping up telling you what store is discounting them! Now that does sound far out, but it’s not impossible. Think, for instance, of a world where everything had RFID tags on them.
What is called “registration,” however, is a rather hard problem. It’s about the extra information being properly overlaid on the real scene. When you move your eyes to the left, the tags need to shift appropriately so they still point to the correct objects-and that requires a complex head-tracking mechanism. We cannot here go into the details of how that is achieved; in any case, we’re trying here to tickle your imagination, not go into boring technical details!
AR At Work
Although AR has not been entirely ignored, science fiction writers tend to emphasise the VR experience. It’s cooler. But AR is as much of an exciting possibility, if not more so, than VR. With an appropriate technique, AR can enhance any scene. Scour the Web and you’ll see what countless people have envisioned. Here’s a little sampler.
A mechanical engineer is working with a piece of machinery he’s never touched before. As he wields his gas cutter, he sees, neatly labelled, the parts of the object he’s not familiar with.
A doctor needs to make a cut in an open brain, and his AR system shows him precisely where it is safe to make the cut.
A road is being dug up, and water pipes are to be avoided in the process. Say there’s a database of the pipes in the city. You can then imagine a camera-based AR system that allows the diggers to “look” at the pipes under the road.
We haven’t even begun to scratch the surface. Those examples were just to get you thinking.
Who’s Doing What
At the University of Cambridge Department of Engineering, they’re developing what they call “Robust Model-based Tracking for Outdoor Augmented Reality.” You point the AR device at an object, and the device detects the edges. Any labelling-as in augmenting-can be done only after the edges have been determined.
Edge detection-seem simple? Actually, the technicalities are way too heavy to go into here! Even for this starting point-detecting edges-you need gyroscope measurements, measurement of gravity and magnetic fields, and more!
Nokia’s Mobile Augmented Reality Applications (MARA) project uses camera-equipped mobile devices for “video see-through mobile AR,” meaning that it can overlay information right on the cell phone’s screen, depending on what the camera is being pointed at. We learnt from http://tinyurl.com/32tmkx about such a prototype phone: it is equipped with MARA software, and hardware including a GPS, an accelerometer, and a compass. Conceptually, it’s simple. As the phone captures a video stream, MARA collates information from the aforementioned hardware to pinpoint the location and orientation of the phone. It then looks in a database to determine what objects should be visible when the camera is at that location. (The database could be on the phone, or accessed remotely.) Some pattern matching is involved here, and once it’s been established what objects are being viewed by the camera, MARA highlights them and provides information about them (and hyperlinks, if available in the database). You can imagine the rest: if a restaurant is being viewed and is also in the database, MARA could (potentially) display the menu and wait time. If there were a hyperlink as well, you’d be able to visit the restaurant’s Web site!
Watch the videos at http://tinyurl.com/y883yj. 
VISTA: The Middle Ground
What we were calling VISTA (Virtual Imagery Superimposed on Top of Actuality)-the infusion of virtual elements into a real scene-is, as we said, something like mixed reality. Strictly speaking, according to many people’s definitions, this is AR-because it’s all about augmenting a scene. We’ve coined the term VISTA because we think AR enhances the reality you’re experiencing, while VISTA is adding virtual elements that shift focus away from the “reality.”
Consider these useful applications:
- An interior designer can have a view of the before-and-after scene. He sees a room as it is, and when he presses a button on his Head-Mounted Display, his planned additions-like a sofa-appear as though they were in the room.
- Here’s a gem of an idea we found on the Net: “Don’t like your office environment? Add plants, waterfalls, and hummingbirds to your office with the ‘sounds of nature’ software module. Is your significant other unbearably ugly? Overlay their natural face with any character using the ‘augmented people’ module.”
Then, from http://tinyurl.com/35jpxo, we learn that a system has been developed for tourists at sites such as Pompeii to be able to see the people who lived there before the eruption of Vesuvius destroyed the city. They’re calling it a “breakthrough prototype AR system”; it can add “digital people” and other computer-generated, animated elements in real-time to what you “actually” see. Again, “actually” is in quotes, because… oh, well.
Turns out someone’s actually designed an AR-read VISTA-version of Quake, in which the monsters are overlaid on the real scene in front of you. Read about the ARQuake Project-and watch the videos-at http://tinyurl.com/c6459.
We said it, and we’ll say it again-the applications are limitless. The US military has been doing VISTA-oriented research for many years now. The possibilities? How about this one: there is data on the construction of a building, with its entrances and exits; a soldier with a Head-Mounted Display would see all these at once instead of just the one entrance he’s facing. Then, think about training: soldiers could “face” troops and tanks and all, while actually situated on real ground, and perform the appropriate actions.
Closing Thoughts
Some aver that AR is a potentially dangerous technology. When it becomes commonplace, you’d have to get your “broadcasts” from somewhere-for example, we said ad agencies could send their information to the information servers so that the ads get displayed along with tourist-aiding tags. Now if those broadcasts are to be received from some kind of central agency-either a government, or a large corporation-it could result in the ultimate in mind control. The reality you perceive would be controlled by someone else! If the idea is not clear, think about how much of our minds is controlled by Google today. With that as a real-life example, think about “subscribing” to AR channels, and how they’d influence your perception of reality.
Continuing along the lines of “subscribing to channels,” think of how exactly you could use AR. As a tourist in Kenya, you’d subscribe to the Kenyan tourism broadcast. On a trek, you’d tune in to “Nature”-and trees and shrubs and little furry animals could get tagged with their names. Just for fun, at a high-profile business meeting, you’d be able to figure how expensive the business suits around you are. Talking to someone who doesn’t speak English? Use translation AR, and you’d “listen” to the person in English.
We’ll leave the fantasising to you. But we need to reinforce that AR and VISTA aren’t fads; research is red-hot and people are dead-serious about it. (Please remember, only we coined “VISTA”!) But it might just lead to nothing much, going the way of VR-if you remember how wild people were about VR a decade and a half ago.
Ultimately, it’s about potential. To use a cliché and an obvious phrase, we’ll end by saying that Augmented Reality has the potential to fundamentally change our life-experience.