Notable archievements in this space
Augmented Reality dates back to the 1950s when Morton Heilig, Father of Virtual Reality, invented the Sensorama Stimulator. He dubbed it the “Experience Theater”. The idea behind it was to simulate an experience with visuals, sound, vibration, and smell, without actually being subjected to the “possible hazards of particular situations”. It was designed as a cinematic experience: it blew wind at you, vibrated the seat you sat on, played sounds to your eyes and projected a form of a stereoscopic 3D environment to the front and sides of your head; kind of like those 5D movies that we watch in theaters today. However, it was a mechanical device shaped like an arcade machine from the 80s. Oddly enough in retrospect, the project didn’t get financial backing. The first augmented reality head-mounted display system was invented by Ivan Sutherland in 1968. The device was too heavy-to-handle and was hung suspended from the ceiling and very appropriately called The Sword of Damocles.
NOTABLE ARCHIEVEMENTS IN THIS SPACE
From the earliest attempts at augmentation in the form of Experience Theaters to newer milestones like Google Glasses, we chronicle some of the pivotal moments in this space
Augmented Reality dates back to the 1950s when Morton Heilig, Father of Virtual Reality, invented the Sensorama Stimulator. He dubbed it the “Experience Theater”. The idea behind it was to simulate an experience with visuals, sound, vibration, and smell, without actually being subjected to the “possible hazards of particular situations”. It was designed as a cinematic experience: it blew wind at you, vibrated the seat you sat on, played sounds to your eyes and projected a form of a stereoscopic 3D environment to the front and sides of your head; kind of like those 5D movies that we watch in theaters today. However, it was a mechanical device shaped like an arcade machine from the 80s.
Oddly enough in retrospect, the project didn’t get financial backing. The first augmented reality head-mounted display system was invented by Ivan Sutherland in 1968. The device was too heavy-to-handle and was hung suspended from the ceiling and very appropriately called The Sword of Damocles. While its graphical prowess was fairly limited, it was the first step to usable-AR. It was only in the 1990s that the tech was finally christened, when a Boeing researcher Tom Caudell coined the term “augmented reality”. Julie Martin created a theatrical production in Australia called “Dancing in Cyberspace”, which used Silicon Graphics computers to let the dancers and acrobats manipulate virtual objects in real time projected into the same performance space.
For most of the 20th century, AR remained a toy for the scientists. Which meant that the common man didn’t even know it existed? Bulky equipment with expensive software kept far from the consumer; until Hirokazu Kato of Nara Institute of Science and Technology, Japan released the ARToolKit to the open source community. The ARToolKit was a platform for creating augmented reality applications. It used video tracking capabilities and virtual object interaction, and provided 3D graphics models that could be overlaid on any OS platform. Smartphones were not in the picture at the time; the Flash based AR applications required a simple handheld device with a camera and a stable Internet connection. Prof. Bruce H. Thomas of the Wearable Computer Lab at the University of South Australia created the first ever Augmented Reality video game, ARQuake. You needed your laptop in a backpack, GPS, gyroscopes, a head-mounted display and voila! The world around you changed into a game arena, with monsters roaming about in your vicinity in real time. You didn’t need a joystick; you could move about in the real world and destroy the virtual monsters. The game still exists as a prototype to this day and unfortunately, no plans to commercialise it exist as of now.
A few years later in 2008, Augmented Reality apps came to your smartphones, where you could finally enjoy this mindblowing technology. With applications like Wikitude and Layar which pioneered the smartphone app market, the environment around you was suddenly made “clickable”. These AR apps use your smartphone/tablets’ camera, GPS, compass and other sensors to overlay information on the camera feed to your device. When ARToolkit was ported to Adobe Flash, Augmented Reality was made possible through your desktop browser and the webcam.
In February 2009, at the TED conference, Pattie Maes and Pranav Mistry presented their augmented-reality system- SixthSense. Their prototype consisted of a pocket projector, a mirror and a camera contained in a pendant-like device that a user wore around his neck. Since the camera is positioned on the user’s chest, it augments everything in sight. SixthSense is a remarkable device; it uses these off the shelf components and costs approximately $350 to build (not including the computer). The micro-projector and the camera are connected to a mobile computing device placed in the user’s pocket. The mini projector projects visual images onto your surroundings- walls, hand, table, etc., enabling them to be used as interactive screens. The camera and mirror are used to recognize and track a user’s hand gestures, colors and physical objects using image-processing based techniques.
The software on the computing device processes the video stream data captured from the camera using simple computer-vision techniques, gathers GPS coordinates and pulls up data from the Internet. The user may also wear colored caps on the fingers which are used to interact with the projected image; even fingers with different colors of nail polish would work. The movements and arrangement of these colored markers (also known as fiducials) and hand gestures serve as interaction instructions for the projected application interfaces. SixthSense also supports multi-touch interaction letting you zoom in, zoom out or pan using many of the intuitive hand movements that we are now familiar with. The SixthSense prototype implements a number of demonstrative applications:
• The map application lets you navigate a map displayed on a nearby surface using hand gestures.
• The drawing application lets you draw on any surface by tracking the fingertip movements of the user’s index fingers.
• SixthSense also lets you draw symbols in the air using the tip of your index finger and recognizes those symbols as interaction instructions. Drawing a magnifying glass symbol takes you to the map application or drawing a ‘@’ symbol lets you check mail; Draw a circle on your wrist and SixthSense projects an analog watch on it.
• SixthSense recognizes your complex hand gestures. The SixthSense system takes photos of the scene you are looking at by detecting the ‘framing’ gesture. You can stop by any surface or wall and flick through the photos you’ve taken.
• The SixthSense system also augments physical objects in your vicinity by overlaying more information on them and making them interactive. For example, a newspaper can show live video news or dynamic information provided on a regular piece of paper. Touch any object and learn so much more about it than appears to the eye. For more details on SixthSense log on to: http://dgit.in/Xn9EZw.
Pranav Mistry with SixthSense
Steven Spielberg’s 2002 movie Minority Report envisaged life in 2054, and shows the exceptional fullfledged Augmented Reality at work. For sci-fi fans, the pertinent memory is that scene in the movie when Tom Cruise’s character Chief John Anderton strolls through a mall while everything advertised itself by bombarding him with personalized marketing messages. A Guinness billboard calls him by his name and tells him he could use a drink; the Gap store has holograms of assistants who ask him if he’s enjoying his previous purchase; an American Express advertising displays a giant 3D credit card embossed with his membership details and so on; you get the idea.
SixthSense’s Visual Dialpad
This “advertising nirvana” is based upon the possible existence of retinal scanners; also, computer monitors and displays are transformed into holograms. With the virtual retinal display technology, the image is scanned directly into the retina of human eye. The viewer sees what appears to be a conventional display floating in space. This fancy technology is far from being a figment of your imagination and is currently under development at the University of Washington’s Human Interface Technology Laboratory.
AR in Hollywood; a scene from Minority Report
Augmented reality is changing the way we see the world. Someday you’ll be walking down the street with augmented-reality displays, which will eventually look much like a normal pair of glasses; informative graphics will appear overlaid on your field of view in real time with synced audio.
Sergey Brin with Google Glasses
These enhancements will be refreshed continually to reflect the movements of your head. This remarkable technology blurs the line between what’s real and what’s computer generated by augmenting what we see, hear, feel and smell. In the AR wars, Google is a step ahead of everyone. Google’s Project Glass is developing a wearable spectacles like device that would overlay information on the eye’s view and could interact with the Internet via natural language voice commands. The device will have Google’s Android operating system. The prototype has been worn by Sergey Brin, co-founder of Google, to several occasions. He demoed the glasses at the Google I/O presentation in June 2012 where skydivers, abseilers, and mountain bikers wore the glasses and live streamed their point of view to a Google+ Hangout, which was also shown live at the Google I/O.
Google Glasses have also been compared to Steve Mann’s EyeTap. An EyeTap is also a type of wearable computing gear for one eye invented by Steve Mann in 1980s. The design allows the user’s eye to operate as both a monitor and a camera. The first prototype consisted of a computer in a backpack wired up to a camera and its viewfinder which in turn was jerry-rigged to a helmet. With time, the technology evolved and the size of the device shrinked. The current version consists of an eyepiece that mediates the reality, a keypad with which the user can interface with the EyeTap and a small processor which can be attached to almost anything, even a WiFi device for pulling up data from the Internet. Delving into the technology behind thisv amazing gear, we find that an EyeTap uses something called a “beam splitter” to send the same image to both the eye and a camera.
The camera then digitizes that reflected image and sends it to a computer, which processes it. The result of all this processing is data that is sent to a projector that will then be overlayed over the scene that you are viewing
Steve Mann with EyeTap
Interactive eyemaker Vuzix will also reveal their competitor to Google Glasses- Smart Glass M100, in CES 2013. The Vuzix eyepatch will have Android IceCream Sandwich and a 720p camera along with Wi-Fi, Bluetooth and head-tracking sensors. You can take stills, capture video (with playback and picture viewing options), answer the phone with a visual address book, read emails and texts and run various AR applications. This “modernized monocle” is supposedly iOS and Android compatible. Vuzix, which spe- cializes in technofancy optics, created the first video eyewear to support stereoscopic 3D for the PlayStation 3 and Xbox 360. It also created other Augmented Reality eyewear, including theWrap 920AR, Raptyr and STAR 1200.
Not lagging behind in the race, Microsoft has also patented Augmented Reality glasses for “live events”.
Vuzix Smart Glasses M100
Here, “live event” is a broad term encompassing sports, singing and other things that are happening around you, including action replays and lyrics. Microsoft has recently demonstrated live language translation to and from Chinese. These AR glasses would enhance the stream of video by overlaying informative graphics. The device is expected to cover a wide market with its “live event” idea. According to Patently Apple, Apple too has got patents for its own retinized version of Google Glasses, granted in February 2010, which would work with a “handheld communication device”. Although the sketch in the patent is not very impressive, it’s an established fact that Apple never disappoints with its design. With glasses and head-mounted displays in the making, companies are also focussing on developing AR contact lenses. These lenses have embedded in them integrated circuits, LEDs and antenna for wireless
Apple’s Sketch for AR glasses
communication. They let the wearer focus on two fields of view, the close-to-the-eye AR images and distant real world objects at the same time. Innovega, the company that makes them is trying to emulate display screens akin to what Arnold Schwarzenegger’s character saw in the movie Terminator. These lenses, called iOptik are very much in demand by the US military. Now we might have agents like Ethan Hunt from Mission Impossible 4: Ghost Protocol, wearing these AR lenses and pulling up information about everyone in sight. News on the block is that even Microsoft is trying to build AR lenses with help from the University of Kansas.
iOptik Contact Lenses
Apart from the wearable computing devices that use Augmented Reality, the technology is being exploited in many other areas. One of the great examples is the AR sandbox, created by researchers at UC Davis University of California, which overlays a virtual, colored topographic map on a sand surface in real-time. As you change the topography of sand, make riverbeds and mountains, the 3D visual image adheres to the changes in terrain configuration. You can actually see the water flowing in the channel and fill up the basin. The image is overlaid by a Kinect 3D camera which is connected to a projector that overlays virtual information on the sandbox. The system is powered by an Intel Core i7 CPU and a Geforce 580 graphics processing iOptik Contact Lenses unit. Such AR systems can be used for educational purposes, for example for projects in science exhibitions.
Another cool device based on this remarkable technology is the Augmented Reality glove for your ipad. Ironically, now you can operate your ipad, a touchscreen device, without touching it. With this AR glove called T(ether) you can create virtual environments on the tablet. It lets you control ipad objects-pick them up, drop them down, push them aside or even create a new object by tracing the shape’s outline- with simple hand gestures. T(ether) can be used by a number of people to collaboratively work on the same virtual environment for the editing and animation of 3D virtual objects. By using Vicon motion capture cameras, the position and orientation of the tables, user’s heads and hands, are tracked and spatially annotated in real-time. Watch the video on the following link to see how it works (http:// dgit.in/Uo6x2P).
T(ether) AR glove for ipad
With man still trying to perfect the above mentioned AR devices, Augmented Reality has managed to find a foothold in the world of apps for your smartphones and tablets. It will be a while before you own a pair of AR glasses or an HMD like the SixthSense, but you can taste this sweet fruit via your smartphones and tablets. iPhones and Android devices have a vast market of AR apps for almost every aspect of human indulgence, from finding restaurants to recognizing landmarks and looking up satellites in the sky. More information on various popular AR apps is given in chapter 10 of this booklet. These AR apps use your device’s GPS for looking up geotagged information, use your camera to receive relevant information of the world around you, and so much more.
And now AR is even making its way into cars. It builds on the “heads up display” (HUD) which is a transparent display that presents data to the user without him having to look away from the road. This technology was developed for military aviation for the convenience of fighter pilots and began to make its way into high-end cars. These HUDs can be used to project directions or speed and other relevant information onto the windscreen in the driver’s line of sight so the driver need not take his eyes off the road. While all this sounds very fascinating, augmented reality could allow these displays to get even better. Various car manufacturing companies such as BMW, Toyota, Mercedes and GM have shown off their prototypes of looking glass integrated with AR technology. The Mercedes Dice prototype conjures up information about passing places of interest and shows icons of friends driving past the car on the windshield with their social network status. Toyota has shown off their version of “enhanced car windows” that will allow users to zoom in on places of interest that they pass by. The enhanced vision system developed by General Motors goes one step ahead and uses an array of sensors and cameras mounted inside and outside the vehicle to monitor the on-road conditions. Apart from acting as a navigation guide, it helps you drive in foggy conditions and generate warnings about possible dangers. The display adjusts according to the driver’s head movements.
Every one of us have different expectations from this technology. The near future can be seen but in the long term, this technology might always surprise you. One interesting perspective is that of the InteraXon Co-Founder Ariel Gartern. The company is working on a brain-sensing headband that receives your brainwaves to monitor concentration levels, reduce anxiety, improve your memory and much more. They look forward to a future where brainwaves and AR work in tandem. The head mounted displays and AR glasses provide the opportunity to collect brainwave signals while the user continuously wears the device.
This collaboration allows the computer system to present contextually aware overlays by noting your likes and needs. The system could continuously register and process your level of distress throughout your workday and present information accordingly. These systems would be “context aware” and would not only show relevant information but also present it in the way you want. For example if it senses that you’re sleepy, it would show you the hotels in the vicinity to spend the night. The real-time neuro feedback will let it know your brain state and try to optimize your experience. This is just one futuristic view; there is so much more to come. The bright side is that developers already have a far-sighted view and plans to exploit this technology and research is being done to make it a reality.