2.1 Types of Augmented Reality Systems

One of the most important parts of augmented reality is the ability of the user to see his or her environment. However, the augmented reality device also has to “see” it, and that involves a computer-based vision system (Fig. 2.1).

Fig. 2.1
figure 1

Vision-based augmented reality

Full size image

A camera combined with a display is an appealing configuration. Such a setup provides vision-based feedback that effectively closes the loop between the localization process and the display. This also reduces the need for heavy calibration procedure.

But, what should, or will be the display device. We have several choices, as described in the following sections.

All the displays used in wearable augmented reality systems (which excluded by definition mobile devices such as smart phones, tablets, and notebooks) are commonly referred to see-through, near-to-the-eye displays, or NEDs.

2.2 The Taxonomy of Augmented Reality

Augmented reality is an amazingly diverse, robust, and complicated field, and if ever there was an industry where one size does not fit all, it is the augmented reality arena.

At high level, it starts with two main categories: wearable and non-wearable (portable or stationary devices).

Wearable includes headsets, helmets, and one day, contact lenses.

Non-wearable includes mobile devices (smartphone, tablets, notebooks, weapons, etc.), stationary devices (TVs, PCs, plays, etc.), and head-up displays (integrated or retrofitted).

The diagram in Fig. 2.2 (Sect. 2.2) outlines the taxonomy of the augmented reality field.

Fig. 2.2
figure 2

Taxonomy of augmented reality

Full size image

I will attempt to define and give examples of the various devices for these categories, which has been challenging. At the time of this writing there were 80 companies making one type or another augmented reality device, and I doubt I found all the military ones. This book is not meant to be a survey of suppliers as they will come and go, and hopefully the book will remain relevant and useful beyond the churn in the industry.

Ron Padzensky, who runs a blog-site on augmented reality called, Augmera [1], re-classified the above and created a taxonomy which is strictly hardware oriented (Fig. 2.3).

Fig. 2.3
figure 3

Taxonomy of augmented hardware (Ron Padzensky)

Full size image

And yet another segmentation or taxonomy is the device itself. Augmented reality can be experienced in or on dedicate augmented reality devices, or non-dedicated devices such as TVs, mobile phones, tablets, and PCs.

Within dedicated visual see-through augmented reality systems, there are seven classes:

  • Contact lens

  • Helmet

  • Head-up Display (HUD)

  • Headset (Smart-glasses)

    • Integrated

      • Indoors

      • Outdoors

    • Add-on display and system for conventual, sun, or safety glasses

      • Indoors

      • Outdoors

  • Projectors (other than HUD)

  • Specialized and other (e.g., health monitors, weapons, etc.)

The following is a list (in alphabetical order) of examples of the primary classes of visual see through dedicated augmented reality devices.

Ronald Azuma who is credited with defining the three main elements of augmented reality has yet another taxonomy [2]:

  1. 1.

    Head-Mounted-Displays (HMD)

    1. (a)

      LCD-based, head-worn

    2. (b)

      Virtual retinal displays

  2. 2.

    Handheld displays

    1. (a)

      Flat panel LCD displays with an attached camera

  3. 3.

    Projection displays

    1. (a)

      Project the virtual information directly on the physical objects

    2. (b)

      Head-worn or fixed projector in the room objects with special surface reflecting the light

    3. (c)

      Projected images only visible along the line of projection

Presumably 3.c could include HUDs.

While Steve Mann in his book: Intelligent Image Processing (Wiley 2001) suggests the following:

Augmented Reality must be:

  1. (a)

    Orthospatial (capable of mapping rays of virtual light collinearly with rays of real light);

  2. (b)

    Orthotonal;

  3. (c)

    Orthotemporal (interactive in real-time).

Says Mann, “An ideal reality mediator is such that it is capable of producing an illusion of transparency over some or all of the visual field of view, and thus meets all of the criteria above.”

The following are some examples of types of products that make up the augmented reality market.

2.3 Contact Lens

Contact lenses for augmented reality are still in development and there are no commercial products available yet. The information on these intriguing devices can be found in the section, “Contact lens,” Sect. 2.3.

2.4 Helmet

In the case of helmets, I classified a device as a helmet if it covered the user’s ears, total head and most of the face (Fig. 2.4).

Fig. 2.4
figure 4

This is a helmet (Source: Daqri)

Full size image

Some of the devices I put in the integrated smart-glasses category are quite large and look like a mini helmet so everyone may not agree with my classification (Fig. 2.5).

Fig. 2.5
figure 5

This is not a helmet, and is an integrated smart-glasses AR HMD (Source: Caputer)

Full size image

As in all categorization, distinctions can blur and overlap, which can at times cause confusion for someone new to the industry. Or in some situations it may just suit the developer, or the user to refer to the device in a certain way for easier communication. There are no hard rules in these classifications, just generalizations.

2.5 Head-Up Display

In the case of head-up displays, I only considered add-on or retrofit systems, not factory installed systems by vehicle manufacturers. The adoption of head-up displays in cars is rapid and difficult to track. The high-end cars from Audi, BMW, Cadillac, Lexus, Mercedes, and others have sensitized the consumers, as well as legislators who are proposing it be mandated.

A retro-fit head-up display is usually a low-cost device that either connects to an automobile’s on-board diagnostics system (OBD2) connector and/or connects via bluetooth to a smartphone. Such devices sit on the dashboard, and project the vehicle speed, engine speed, water temperature, battery voltage, instantaneous fuel consumption, average fuel consumption, mileage measurement, shift reminders (if needed), and other warning conditions and project them to the inside surface of the windshield (Fig. 2.6).

Fig. 2.6
figure 6

Kshioe’s Universal 5.5″ Car A8 head-up display (Source: Amazon)

Full size image

Another example is one that uses the smartphone’s application that displays vehicle speed and navigation information, available for the Android and iOS operating systems. In the following image, you can see the user (Fig. 2.7).

Fig. 2.7
figure 7

Hudway Glass which offers navigation and a speed information from a smartphone (Source: Hudway)

Full size image

There is additional information on head-up displays for aircraft in the section on “Aircraft,” section “Aircraft”, and on automotive in the section, “Walking and Driving,” section “Walking and driving”.

It should be noted, HUDs are not limited to automobiles, and are being deployed in buses, trucks, and even ships.

2.6 Smart-Glasses

As mentioned earlier, I have subdivided the smart-glasses suppliers into integrated and add-on, and both of those categories are further subdivided regarding their ability or design to be used indoors, or outdoors. Obviously indoors is an implied subset of outdoors, however, there are some consumer smart-glasses that are integrated with sunglasses and would be inappropriate to wear indoors.

2.6.1 Integrated Smart-Glasses

The distinction between an add-on augmented reality device, and an integrated device may seem arbitrary. The distinction I made was if the device could be attached to, or worn with regular glasses, it was an add-on device. If it included and integrated lenses and other elements (such as a microphone, camera, or earphones) then it is considered as integrated smart glasses, augmented reality headset.

2.6.1.1 Indoors and Outdoors

I have further subdivided integrated smart-glasses into consumer and commercial, and those into indoors and outdoors. It’s an important distinction. Consider a sports headset like Intel’s Recon. However, for augmented reality glasses that have a time-of-flight depth/distance measuring technology, like Microsoft’s Hololens , the sensors depend on non-visible light that does not agree with UV outdoors (and can also be affected indoors by sunlight coming through windows). In other cases, the display may not be bright enough to overcome the ambient light from outside.

2.6.1.2 Consumer

Smart-glasses can vary widely from the newer consumer type of spectacles offered by companies like GlassUp ’s UNO, Laforge Shima, and Ice ’s Theia. These devices have a small real time OS on board and communicate via BTLE (Bluetooth low energy ) with any smartphone (or tablet) running Android or iOS. It displays to the user any kind of message that the smartphone has already elaborated, and uses the smartphone’s GPS, gyro, accelerator, and other sensors (Fig. 2.8).

Fig. 2.8
figure 8

GlassUp’s UNO consumer augmented reality glasses (GlassUp)

Full size image

Consumer smart-glasses are mobile battery-powered, and look close to if not exactly like regular glasses. Some consumer versions of smart-glasses have a small box or package, typically the size of a smartphone, connected by a thin cable to the glasses. The box is worn on the belt or in a pocket.

Included in the consumer category are smart-glasses designed for specific functions such as sports and exercise such as Intel ’s Recon, and Kopin Solos.

I do not include Google Glass or its imitators as consumer since they are conspicuous and call attention to themselves and the wearer, just the opposite of what is desired to be worn in public.

2.6.1.3 Commercial

Other smart-glasses are designed for commercial, scientific, and engineering uses and are more powerful, and usually tied to a computer via a tethered umbilical cord. Companies like Atheer Labs, Meta , Osterhout , and many others are in this class.

And some companies like GlassUp make both consumer and commercial smart-glasses.

Also, included in this category are special smart-glasses for people with eye disorders or partial blindness such as LusoVU Eyespeak (Fig. 2.9).

Fig. 2.9
figure 9

This is an integrated augmented commercial reality head-mounted device (Source: Atheer Labs)

Full size image

The Atheer Labs head-mounted display is an integrated smart-glasses example because it has ear temples on both sides and has built-in lenses.

2.6.1.4 Binocular Verses Monocular

Some integrated smart-glasses have only one display (and all add-on smart-glasses only have one display). Binocular provides much better, more natural viewing. Often, monocular devices can cause headaches, have screens that obstruct the view rather than overlay, etc. Binocular also provides greater field of view than most monocular glasses.

2.6.1.5 Prescription Smart Glasses

Some companies have taken one additional step in the development of their smart glasses, and added prescription ground lenses. One of the questions often asked of smart glasses manufacturers is, “Can I use it with my glasses?”

As is mentioned later (Laforge, Sect. 6.1.4), as of this writing, only a few companies are offering consumer-class augmented reality eyewear with a normal look.

Rochester Optical , a renowned optical company founded in 1883 by William H. Walker [3] (1846–1917) has provided prescription lenses that can be used with other firms augmented reality glasses. In November, 2015, Hong Kong Optical Lens Co., LTD established a partnership with Rochester Optical in offering Smart Solutions for five types of smart glasses and providing optical corrections users. Rochester Optical had designed its Smart Solutions optimized for people who felt discomfort while wearing smart glasses. The company developed two types of prescription glasses. Very thin and light frames that can be used under augmented reality headsets (and virtual reality head-mounted displays too), and custom frames that can accept add-in augmented reality displays such as Recon, Vuzix, Epson, Atheer, and others.

Jins Mem also offers type of smart glasses in a conventional frame with prescription lenses. However, they do not have a camera, and connect to a companion application in a smartphone via Bluetooth. They’re equipped with a gyroscope, six-axis accelerometer, and electrooculography (EOG) sensors for tracking eye movement. The Meme smart glasses are designed for fitness tracking and can measure posture and identify when fatigue is setting in.

2.6.1.6 Blue Light Filters

Smartphones, tablets, and displays in augmented reality headsets are common sources of blue-violet light. Close proximity to users’ eyes intensifies the impact, and too much blue light exposure can contribute to eye fatigue, and potentially cause early-onset macular degeneration [4].

To combat this problem, lenses have been designed that can block harmful blue-violet light emitted by digital devices. Rochester Optical has made this a feature of their lens offerings. Laforge is also adding blue light to their lenses, in addition to UV filtering.

2.6.2 Add-On Smart-Glasses

Add-on augmented reality display devices, like the Garmin Varia-Vision , can be attached to sunglasses or prescription glasses (Fig. 2.10).

Fig. 2.10
figure 10

This is an add-on augmented reality display device (Source: Garmin)

Full size image

Add-on displays, or retro-fit displays, are usually limited to monocular presentation. Interestingly, the suppliers seem to favor the right eye, possibly influenced by Google Glass. Steve Mann’s original EyeTap however, was worn over the left eye.

The other augmented reality devices such as contact lenses are discussed in the “Smart contact Lens,” section on Sect. 8.7.5.3.

2.7 Projection

The use of projected specially treated light that can be seen through some type of a viewer has been worked on since the early 2010s. Advertised as holo-something, with the implication that it is some form or type of a hologram, and inspired by the famous image in the original Star Wars movie (now Episode 4) with a hologram of Princess Leia projected by R2D2 (Fig. 2.11).

Fig. 2.11
figure 11

Princess Leia’s hologram projected into open space (Courtesy: Lucasfilm Ltd.)

Full size image

To project an image requires some type of a reflective device or medium to send the light (photos) to the viewer’s eyes; free space is not such a medium. However, special glasses, or a viewer (such as a tablet) can be. The glasses/viewer can see the basic scene, and receive the virtual information, classic augmented reality. It’s not holographic, its mixed or projected reality.

To create the illusion, a display system needs two pieces of information: the exact position of the projection surface in 3D space, and the position of the viewer’s eyes in the same 3D space. Together, these two provide enough information to set up the correct perspective projection.

Realview Imaging, Yokneam, Israel, introduced a 3D volumetric imaging system in 2012 targeted at the medical market for visualization by clinicians called the Holoscope. Based on acoustical interference, it was effective, but small, and expensive.

Since then the company has developed an augmented reality headset for viewing the models (Fig. 2.12).

Fig. 2.12
figure 12

Realview’s Deep perception live holography system (Realview)

Full size image

Realview creates multiple depth planes at multiple distances in real time, projected simultaneously. Using multiple depth plans, says the company, eliminates the vergence confusion. Like other firms (Microsoft, Hololamp, CastAR, etc.) the term holograph is used casually, and incorrectly, holograms don’t have focal planes. However, in the case of Realview, which is using depth planes, it may be definitionally correct. A holographic image of a real 2D image is still a holographic image, just not one of a 3D object. As Dr. Oliver Kreylos of the University of California at Davis points out, composing a holographic image from multiple slices of a 3D object is an approximation of creating a holographic image of the full 3D object, but it is still a real holographic image.

According to the Wikipedia entry for computer-generated holography, one of the simpler algorithms to generate the required interference patterns, Fourier transform, is only able to create holograms of 2D images. Another method, point source holograms, can create holograms of arbitrary 3D objects, but has much higher computational complexity.

To create the illusion, a display system needs two pieces of information: the exact position of the projection surface in 3D space, and the position of the viewer’s eyes in the same 3D space. Together, these two provide enough information to set up the correct perspective projection.

A more affordable system, suitable for home use, is the one developed by Hololamp . Founded in Paris France in early 2016, the company introduced their augmented reality projector in early 2017. The Sony (Microvision laser-beam scanning) projector makes it possible for augmented reality to interact with real objects, and no special glasses or smartphone are required to see its animated images (Fig. 2.13).

Fig. 2.13
figure 13

Hololamp glasses free augmented reality projector (Hololamp)

Full size image

HoloLamp is a structured light projection system that creates a 3D point-cloud of the local area to generate the 3D objects in the field of view. The objects are then used as markers for the superimposed virtual objects, and registered accordingly so they behave physically correctly. The point of HoloLamp is to project images of virtual 3D objects onto arbitrary surfaces. HoloLamp uses an additional set of cameras looking upwards to identify and track the viewer’s face, using face tracking algorithms. Based on that, the software can project 3D objects using one or more projection matrices. The effect is monoscopic, and can only work for a single user.

Hololamp describes their approach as Spatial Augmented Reality.

2.7.1 Spatial Augmented Reality

Spatial augmented reality (SAR) is a term, and concept pioneered Oliver Bimber while at Bauhaus-University , Weimar, and Ramesh Raskar at Mitsubishi Electric Research Laboratory , Cambridge, MA, in 2004 [5].

Special augmented reality is a branch of augmented reality based on projectors that deliver a glasses-free and hands-free augmented reality experience. It exploits the way our eyes perceive 3D objects to allow you to perceive a 3D experience. Spatial augmented reality enables moving images on any surface by mapping the surface, tracking the user and then projecting an image that is warped so that from the user’s point of view it is what they would expect to see as a 3D effect.

The Hololamp concept it is somewhat similar to the AR Sandbox , i.e., a set of cameras that scan a projection surface and a viewer’s face, and a projector drawing of a perspective-correct image, from the viewer’s point of view, onto the projection surface. It’s also similar to what CastAR does.

The CastAR glasses project 3D images in front of the wearer’s eyes and feel as if one is seeing a virtual layer on top of the real world, or feel like being immersed inside a game world. It uses shutter glasses with a forward facing (into the room) set of tiny projectors, and a reflective sheet-like material called retro-reflective. Theoretically one could use the reflective sheets to animate an entire room.

The U.S. National Science Foundation (NSF) funded AR sandbox —a multi-university project designed to develop 3D visualization applications to teach earth science concepts. It is a hands-on exhibit combining a real sandbox, and virtual topography and water created using a closed loop of a Microsoft Kinect 3D camera, powerful simulation and visualization software, and a data projector. The resulting augmented reality (AR) sandbox allows users to create topography models by shaping real sand, which is then augmented in real time by an elevation color map, topographic contour lines, and simulated water (Fig. 2.14) [6].

Fig. 2.14
figure 14

Left: Sandbox unit when turned off. The Kinect 3D camera and the digital projector are suspended above the sandbox proper from the pole attached to the back. Right: Sandbox table when turned on, showing a mountain with a crater lake, surrounded by several lower lakes

Full size image

The goal of this project was to develop a real-time integrated augmented reality system to physically create topography models which are then scanned into a computer in real time, and used as background for a variety of graphics effects and simulations. The final product is supposed to be self-contained to the point where it can be used as a hands-on exhibit in science museums with little supervision.

2.7.2 CAVE

A classic CAVE Automatic Virtual Environment (better known by the recursive acronym CAVE) is an immersive virtual reality theater where 3D images are rear projected to between three and six of the walls of a room-sized cube. The first CAVE was conceived in 1991 and built by Professors Dan Sandin and Tom DeFanti at the University of Illinois at Chicago’s Electronic Visualization Laboratory where grad student Carolina Cruz-Neira wrote the first software drivers for the CAVE. Since 2009, various CAVEs have been made from LCD and OLED panels with passive polarization, eliminating the need and space for projectors.

A CAVE can show a video projection of the outside world, so in once sense it is an obstructed view augmented reality system. Consider the giant CAVE at Stoney Brook University in New York (Fig. 2.15).

Fig. 2.15
figure 15

Reality Deck” CAVE at Stony Brook Univ (2012) 40′ × 30′ × 11′ high room containing 308 LCD display—1.25 billion pixels

Full size image

Combining augmented reality environments with immersive virtual environments can reduce interaction gaps and provide embodied interactions in distributed collaborative works. It can also provide many opportunities and degrees of freedom for collaboration in both real and virtual worlds.

Inside a CAVE there can be multiple people viewing the projections. They wear stereoscopic glasses (typically polarized and untethered and usually have reflectors on them for location sensing within the cave.

CAVEs do not have to be six-sided rooms. The University of California, San Diego built the WAVE display, and true to its name, it is shaped like an ocean wave, with a curved wall array of 35, 55-inch LG commercial LCD monitors, that end in a ‘crest’ above the viewer’s head and a ‘trough’ at his or her feet (Fig. 2.16).

Fig. 2.16
figure 16

University of California, San Diego’s WAVE CAVE

Full size image

WAVE is an acronym for Wide-Angle Virtual Environment, it was designed and built in-house by Qualcomm Institute’s Director of Visualization Tom DeFanti, Professor of Visualization and Virtual Reality Falko Kuester and Senior Design Engineer Greg Dawe. The WAVE, a 5x7 array of HDTVs, is 20-feet long by nearly 12-feet high.

A CAVE can be as small as three walls, and as large as Stony Brook’s.

2.8 Specialized and Other

Glasses, designated by the developer as smart-glasses, but with limited or no informational display capability are offered for fitness tracking and health monitoring, and in some cases navigation. Some of the glasses use audio as the information presentation (including navigation). Others, again designating themselves as offering augmented reality smart-glasses merely offer an embedded camera in a glasses frame.

Weapons with augmented reality mechanisms adapted to them are used bythe military (e.g., shipboard defenses) and consumers (see Augmented Reality in Hunting, section “Augmented reality in hunting”).

2.8.1 Watermarking Augmented Reality

In 2000, the Digimarc company in Portland OR, developed a water marking technology to thwart counterfeiting. As the web was spreading into all corners of commerce the company realized their technology could be used as a quasi-marker embedded in images and messages. Digimarc’s process of embedding a digital watermark into an image involved dividing the image into blocks of pixels. Then the watermark is independently embedded in each of these blocks. That allowed the watermark to be detected from an image region as small as one block. Spread spectrum techniques are used to make the signal imperceptible and to combat the effect of image manipulation and filtering [7].

The reader reverses the operation of the embedder by extracting the synchronization signal from the frequency domain of the image. It uses the signal to resolve the scale, orientation, and origin of the watermark signal. Finally, it reads and decodes the watermark signal.

Digimarc developed a product they called the MediaBridge, which created a bridge between traditional commerce and electronic commerce (see Fig. 2.17, Sect. 2.8.1).

Fig. 2.17
figure 17

Digimarc’s watermarking technology was used to embed digital watermarks in printed images such as magazine advertisements, event tickets, CD covers, goods packaging, etc

Full size image

When a person used a digital camera or scanner to produce a digital image version of a printed MediaBridge image, the MediaBridge reader application detects and reads the embedded watermark. The embedded watermark represents an n-bit e.g., the Digimarc server. That index was used to fetch a corresponding URL from the database. Then the URL is used by the Internet browser to display the related Web page or start a Web-based application—it created a bridge between the printed material and the Internet.

Digimarc subsequently licensed the technology to several firms and it’s in use in various places today.