Augmented reality (AR)
by Chris Woodford. Last updated: August 15, 2023.
Not so long ago, future-minded people enjoyed making a sharp distinction between "reality" (the often tiresome, problematic world around us) and "cyberspace" (the promising new "virtual" world inside computers). Back in 1996, when the Web was still a novelty and relatively few people were online, John Perry Barlow famously declared that cyberspace is "a civilization of the Mind... both everywhere and nowhere, but it is not where bodies live."
Some years down the line, it's clear that black-and-white distinctions between real and virtual were incredibly wide of the mark. What most people actually want from computers is much more pragmatic: intuitively easy-to-use technology that will enhance their busy, everyday, real lives—to help them find a decent coffee shop, to make shopping less of a chore, to meet new friends who share their interests, or whatever it might be. Now computers are smaller and more portable than ever, and you can go online almost anywhere on the planet, using online information to enhance ("augment") real life is where the smart money is heading. Augmented reality, as it's known, is something we'll be hearing an awful lot more about in the next few years. Let's take a closer look at how it works!
Photo: Augmented reality: What's that plant? Just point your tablet computer at the pesky thing so the digital camera can take a photo, press a button, and the plant-finder app automatically identifies what you're looking at. In seconds, it's called up the Wikipedia entry (left) and a selection of photos of the species on Flickr (right) so you can double-check. This is only one of the possible, future applications of augmented reality, which means blending computer-generated information with the world you can see in front of you. (Simulated image.)
Sponsored links
Contents
What is augmented reality?
Augmented reality sounds horribly theoretical and abstract, but it's actually very simple: real life plus topical, relevant, background information equals something massively more useful. A few quick examples will make the idea clearer:
- You're watching a tennis grand-slam on TV and there's a controversial call from one of the line judges. Was that serve in or out? The TV station runs an instant replay with a computer animation showing the exact trajectory of the ball and where it landed—just outside the line. Then a little table comes up on the screen showing how many serves have been in or out for each player and how the figures have changed over the course of the match.
- You're walking the streets of London, England and you suddenly come across an amazing bit of architecture. What is this fantastic building? Who was the architect? Is that really titanium? You're dying to find out more, but the building is closed and there's no information about it at all. So you hold your cellphone up and take a quick photo. The phone uses its built-in GPS (satellite navigation) system to figure out roughly where you are, then quickly searches Google Images to find similar photos taken in the same neighborhood. In a couple of seconds, it's identified the building and brought up a Wikipedia page telling you all about it.
- You're a fighter pilot flying over a warzone with anti-aircraft fire shooting up at you. You really have to concentrate and looking down at all the gauges on your instrument panel is a distraction you can do without. Fortunately, you're wearing what's called a heads-up display (HUD), a set of goggles with built-in, miniaturized computers that automatically project instrument readings so they "float" in front of your eyes. You can find out everything you need to know without taking your eyes off the sky.
- You're driving down the freeway sometime in 2020 and you start to feel hungry. Wink your right eye twice and a computer display overlays your windshield with a list of eating places in nearby towns. Wink your eye to select the one that looks most promising and your sat-nav system reads out directions for how to get there. You can email an order in advance so it'll be ready when you arrive.
You can see that augmented reality is actually a mixture of real life and virtual reality, somewhere in between the two, so it's often referred to as mixed reality. The key point is that the extra information it gives you is highly topical and relevant to what you want to do or know in a certain place and time.
Photos: 1) A relatively cumbersome, prototype helmet-mounted display developed by NASA in the early 1990s: computer generated images are blended with what the pilot sees through his eyes. Photo courtesy of NASA Langley Research Center (NASA-LaRC).
2) A more recent prototype developed for the US Air Force in 2008. Here the augmented reality display is incorporated into an ordinary pair of wraparound eyeglasses. Photo by Jonathan Snyder courtesy of US Air Force.
How does AR tracking work?
If you're out and about in the real world with your laptop, netbook, or cellphone, it's easy enough to get information: just bring up Google and type in some words. In the brave new world of augmented reality, it's even easier: you get the extra information automatically. That means your portable computing device needs some automatic way of finding out where you are or what you're looking at—a problem known as tracking.
The simplest form of tracking is for the device to use GPS (or some other satellite navigation system) to figure out your position automatically, which is fine if you want broad, background information about a place you're visiting (a local street map, a list of nearby coffee bars, directions to the nearest hotel, or whatever). It's relatively easy to use tracking information from Wi-Fi hotspots as well. But what if you're somewhere like an art gallery or museum and what you actually want is information about each picture or exhibit automatically coming up on your cellphone as you walk through the building? GPS isn't (yet) precise enough for an application like that, so what could we do instead? Broadly speaking, there are two different solutions known as marker-based and markerless tracking.
- Markerless tracking: You could point your phone at each picture or exhibit and have some kind of pattern recognition or feature-detection system try to identify it. That's how our own perceptual systems work, after all: our eyes see things and our brains figure out what we're looking at, then "call up" background information. Our brains are amazingly good at this and make it seem very easy, but it's a much harder problem for a computer to tackle, not least because the best computer vision systems are only a fraction as good as our own.
- Marker-based tracking: A simpler option would be for the gallery or museum to print small, two-dimensional barcodes (also called data-matrix codes) next to each item on display. Then you'd simply point your phone's camera at one of them, your phone would turn the barcode into a web address, and its browser would call up an appropriate web page with further information. AR systems can be designed to read all kinds of other markers (or fiducial markers) as these "added reference points" are called.
Marker-based tracking is currently proving to be the more popular option, largely because it's so simple to implement. But in the longer term fully automatic, markerless tracking seems certain to win out because that's how our own visual systems work—and it's what most users prefer. After all, we recognize our friends automatically without them having to walk around with barcodes printed on their foreheads!
Photo: Marker-based tracking: The Inkhunter app (described below) demonstrates what a tattoo would look like anywhere on your body. Before it can do that, you need to help it figure out where to do its stuff by drawing a "square smile" on your skin. This is a simple example of marker-based tracking.
Displaying augmented reality
Virtual reality has taken off in a much more limited way than many people supposed. One reason for this is the difficulty of "immersing" people in a computer-generated world: ideally, you need to wear a sophisticated headset that completely blocks out the real world and replaces it with a computer-drawn equivalent. With augmented reality, the problem is different: what we need is a way of overlaying computer-generated information on what we can see already. When people first started talking about augmented reality, there was a feeling we'd all end up walking around in heads-up displays, like fighter pilots.
Photo: Some forms of augmented reality require complex and elaborate headsets, like this US Air Force model, which tracks the pilot's head movements and adjusts the display accordingly. The main photo is a closeup of the headset; the inset photo (top right) shows how the pilot wears it. The rest of us can experience AR using nothing more than a cellphone screen. Photo by Mercedes Porter courtesy of US Air Force and DVIDS.
Now Net-connected cellphones with cameras are commonplace, they've become the focus for AR—although that may change in future. Already, researchers are working on prototype contact lenses with built-in computer displays, which would superimpose things like Web pages onto our field of vision. Another possibility is wearing eyeglasses with miniature, forward-facing cameras built into the frames and transparent screens added to the lenses.
Sponsored links
What kind of AR applications already exist?
Links in this section take you mostly to YouTube demos of the various apps, which I think is the best way of giving you a quick taste of what's available.
When the creators of the James Bond films came up with the movie title "The World is Not Enough," they might have been talking about augmented reality. Lots of things we do everyday rely on us simultaneously interacting with the world outside our bodies and sneaking a glance at the digital world inside our computers; enhancing the real world is where augmented reality comes into its own. So what sort of applications are already available?
If you want to change your world but you're not sure how to go about it, augmented reality's happy to help. Take home remodeling for example. It's often quite hard to imagine what your rooms will look like painted a different shade of red, blue, yellow, or gold—which will look best, and what if you make a mistake? Augmented reality home decorating apps like the Dulux Visualizer allow you to look through your smartphone camera and change the colors of your walls at will. Also designed with home makeover buffs in mind, IKEA's Catalog app lets you try out pieces of virtual furniture in your real-world home. If it's your body you want to change, rather than your home, how about one of the many clothes and shoe samplers? They let you see what a new pair of sneakers (or shorts) would look like without going anywhere near a shop. In much the same vein, Inkhunter is a tattoo app that work similar magic on your skin, allowing you to check what a new design would look like on any part of your body before going anywhere near a needle!
Photo: Decorating with AR: What would your home look like painted purple, green, or yellow? Decorating apps like Dulux Visualizer, shown here, give you an instant impression.
If all this sounds a little bit frivolous, it's worth remembering that augmented reality has lots of serious applications in everything from medical education to parcel transportation. Doctors use AR both for teaching (medical students can use AR apps to explore organs in 3D, then project them onto their own bodies) and for "previewing" to patients how scary treatments might affect them; a company called Orca MD makes several "decide" apps (Eyedecide, Knee Decide, and Foot Decide to name just three) that allow people to explore what bits of their body look like from the inside when they're suffering a variety of medical conditions. In 2020, doctors from Johns Hopkins University used augmented reality to help them perform spinal surgery more effectively.
We're not all doctors and rocket scientists in the making, but AR can help all of us, in one way or another. DHL, the delivery company, has trialled augmented reality headsets that help its warehouse staff to find and pack parcels more quickly. And where heads-up displays used to be the preserve of fighter pilots, now they're heading for the mainstream in car windscreens and motorcycle visors (though the long-anticipated, crowd-funded Skully AR motorbike helmet failed to make it into production).
Classroom education's another area where reality is well worth augmenting. If you want to teach people about the stars and planets, for example, what better way than to use a smartphone or tablet that can help them label the things they can see in the sky above their heads? Sun Seeker and Star Walk are just two of many AR applications giving students an inspiring new way to find their way around the world. Satellites and passing flights can be tracked the same way; check out the app version of Flight Radar for example.
Photo: Augmented-reality simulations, pioneered by companies such as Miller, are an efficient way to teach welding in the classroom: they're safer, cut gas and material costs, and save time and money. Photo by Austin Siegel courtesy of US Air Force and DVIDS.
Augmented reality is endlessly entertaining—and fun too. Who could ever forget the global buzz of excitement when the most popular AR game to date, Pokémon Go, hit the streets in 2016. By 2019, it had been downloaded over a billion times, making it the most popular and accessible introduction to augmented reality so far. Slightly less exciting, but no less useful, there are apps like Layar that add digital information to existing real-world stuff: you could scan a magazine to see the adverts leap to life, for example, or point your smartphone at a textbook to watch some handy, related videos.
If you're interested in developing your own AR apps, you don't necessarily have to start from scratch. Wikitude is probably the best known AR-creating software for mobile apps, with built-in object recognition and tracking.
Photo: As with virtual reality, a great deal of augmented reality research and development is driven by military applications. This soldier is testing a US Army system called Integrated Visual Augmentation System, based on the Microsoft Hololens "mixed reality" system. Photo by Chandler Coats courtesy of US Army and DVIDS.
