Skip to main content
SpringerLink
Log in

Reducing the gap between Augmented Reality and 3D modeling with real-time depth imaging

  • SI: Mixed and Augmented Reality
  • Published:
Virtual Reality Aims and scope Submit manuscript

Abstract

Whereas 3D surface models are often used for augmented reality (e.g., for occlusion handling or model-based camera tracking), the creation and the use of such dense 3D models in augmented reality applications usually are two separated processes. The 3D surface models are often created in offline preparation steps, which makes it difficult to detect changes and to adapt the 3D model to these changes. This work presents a 3D change detection and model adjustment framework that combines AR techniques with real-time depth imaging to close the loop between dense 3D modeling and augmented reality. The proposed method detects the differences between a scene and a 3D model of the scene in real time. Then, the detected geometric differences are used to update the 3D model, thus bringing AR and 3D modeling closer together. The accuracy of the geometric difference detection depends on the depth measurement accuracy as well as on the accuracy of the intrinsic and extrinsic parameters. To evaluate the influence of these parameters, several experiments were conducted with simulated ground truth data. Furthermore, the evaluation shows the applicability of AR and depth image–based 3D modeling for model-based camera tracking.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  • Azuma RT (1997) A survey of augmented reality. Presence Teleoperators Virtual Environ 6:355–385

    Google Scholar 

  • Bastian J, Ward B, Hill R, van den Hengel A, Dick A (2010) Interactive modelling for ar applications. In: 9th IEEE international symposium on mixed and augmented reality (ISMAR), 2010, pp 199–205

  • Besl P, McKay N (1992) A method for registration of 3-D shapes. IEEE Trans Pattern Anal Mach Intell 14(2):239–256

    Article  Google Scholar 

  • Bleser G (2009) Towards visual-inertial slam for mobile augmented reality. PhD thesis, TU Kaiserslautern

  • Bleser G, Pastamorv Y, Stricker D (2005) Real-time 3d camera tracking for industrial augmented reality applications. In: WSCG, pp 47–54

  • Bleser G, Becker M, Stricker D (2007) Real-time vision-based tracking and reconstruction. J Real Time Image Proc 2:161–175

    Article  Google Scholar 

  • Bosché F (2008) Automated recognition of 3d cad model objects in dense laser range point clouds. PhD thesis, University of Waterloo

  • Bosché F (2010) Automated recognition of 3D cad model objects in laser scans and calculation of as-built dimensions for dimensional compliance control in construction. Elsevier J Adv Eng Inform 24(1):107–118

    Article  Google Scholar 

  • Bosché F, Teizer J, Haas CT, Caldas CH (2006) Integrating data from 3d cad and 3d cameras for real-time modeling. In: Proceedings of joint international conference on computing and decision making in civil and building engineering, pp 37–46

  • Engelhard N (2010) KinectAutoCalibration. https://github.com/NikolasE/KinectAutoCalibration

  • Engelhard N, Endres F, Hess J, Sturm J, Burgard W (2011) Real-time 3d visual slam with a hand-held rgb-d camera. In: Proceedings of the RGB-D workshop on 3D perception in robotics at the European robotics forum, Vasteras, Sweden

  • Franke T, Kahn S, Olbrich M, Jung Y (2011) Enhancing realism of mixed reality applications through real-time depth-imaging devices in x3d. In: Proceedings of the 16th international conference on 3D web technology. ACM, New York, NY, USA, Web3D ’11, pp 71–79

  • Gall J, Rosenhahn B, Seidel HP (2006) Robust pose estimation with 3D textured models. In: Pacific-Rim symposium on image and video technology (PSIVT), pp 84–95

  • Garland M, Heckbert PS (1997) Surface simplification using quadric error metrics. In: Siggraph 1997, pp 209–216

  • Georgel P, Schroeder P, Benhimane S, Hinterstoisser S, Appel M, Navab N (2007) An industrial augmented reality solution for discrepancy check. In: ISMAR 2007: proceedings of the 6th IEEE and ACM international symposium on mixed and augmented reality, pp 1–4

  • Georgel P, Schroeder P, Navab N (2009) Navigation tools for viewing augmented cad models. IEEE Comput Graph Appl 29(6):65–73

    Article  Google Scholar 

  • Georgel P, Benhimane S, Sotke J, Navab N (2009a) Photo-based industrial augmented reality application using a single keyframe registration procedure. In: ISMAR 2009: proceedings of the 8th IEEE and ACM international symposium on mixed and augmented reality, pp 187–188

  • Haller M (2004) Photorealism or/and non-photorealism in augmented reality. In: Proceedings of the 2004 ACM SIGGRAPH international conference on virtual reality continuum and its applications in industry, VRCAI ’04, pp 189–196

  • Huhle B, Jenke P, Straßer W (2008) On-the-fly scene acquisition with a handy multi-sensor system. Int J Intell Syst Technol Appl (IJISTA) 5(3/4):255–263

    Google Scholar 

  • Kahn S, Wuest H, Fellner DW (2010a) Time-of-flight based scene reconstruction with a mesh processing tool for model based camera tracking. In: 5th international conference on computer vision theory and applications (VISAPP), vol 1. pp 302–309

  • Kahn S, Wuest H, Stricker D, Fellner DW (2010b) 3D discrepancy check and visualization via augmented reality. In: 9th IEEE international symposium on mixed and augmented reality (ISMAR), pp 241–242

  • Klein G, Murray D (2009) Parallel tracking and mapping on a camera phone. In: Proceedings of the eigth IEEE and ACM international symposium on mixed and augmented reality (ISMAR’09), Orlando, pp 83–86

  • Kolb A, Barth E, Koch R, Larsen R (2009) Time-of-flight sensors in computer graphics. In: Proceedings of the eurographics (state-of-the-art report), pp 119–134

  • Lepetit V, Fua P (2005) Monocular model-based 3D tracking of rigid objects: a survey. In: Foundations and trends in computer graphics and vision, vol 1, pp 1–89

  • Lindner M, Kolb A, Hartmann K (2007) Data-fusion of pmd-based distance-information and high-resolution rgb-images. In: Proceedings of the international symposium on signals, circuits and systems (ISSCS), session on algorithms for 3D TOF-cameras, vol 1, pp 121–124

  • MesaImaging (2011) Mesa imaging. http://www.mesa-imaging.ch

  • Newcombe R, Davison A (2010) Live dense reconstruction with a single moving camera. In: IEEE conference on computer vision and pattern recognition (CVPR), 2010, pp 1498–1505

  • Oggier T, Lustenberger F, Blanc N (2006) Miniature 3D ToF camera for real-time imaging. In: Perception and interactive technologies, pp 212–216

  • OpenNI (2011) Openni framework. http://www.openni.org/

  • OpenSG (2011) OpenSG. http://www.opensg.org

  • Pan Q, Reitmayr G, Drummond T (2009) Proforma: probabilistic feature-based on-line rapid model acquisition. In: Proceedings of the 20th British machine vision conference (BMVC), p 11

  • Panagopoulos A, Samaras D, Paragios N (2009) Robust shadow and illumination estimation using a mixture model. In: IEEE conference on computer vision and pattern recognition, 2009. CVPR 2009, pp 651 –658

  • Schiller I, Beder C, Koch R (2008) Calibration of a pmd-camera using a planar calibration pattern together with a multi-camera setup. In: The international archives of the photogrammetry, remote sensing and spatial information sciences, vol XXI. ISPRS Congress, pp 297–302

  • Shi J, Tomasi C (1994) Good features to track. In: IEEE conference on computer vision and pattern recognition (CVPR’94), pp 593–600

  • Van den Hengel A, Hill R, Ward B, Dick A (2009) In situ image-based modeling. In: Proceedings of the 2009 8th IEEE international symposium on mixed and augmented reality, ISMAR ’09, pp 107–110

  • Webel S, Becker M, Stricker D, Wuest H (2007) Identifying differences between cad and physical mock-ups using ar. In: ISMAR 2007: proceedings of the sixth IEEE and ACM international symposium on mixed and augmented reality, pp 281–282

  • Wuest H (2008) Efficient line and patch feature characterization and management for real-time camera tracking. PhD thesis, TU Darmstadt

  • Wuest H, Wientapper F, Stricker D (2007) Adaptable model-based tracking using analysis-by-synthesis techniques. In: Kropatsch W, Kampel M, Hanbury A (eds) Computer analysis of images and patterns, lecture notes in computer science, vol 4673, Springer, Berlin, pp 20–27

    Chapter  Google Scholar 

  • Zhou F, Duh HBL, Billinghurst M (2008) Trends in augmented reality tracking, interaction and display: a review of ten years of ismar. In: ISMAR 2008: IEEE/ACM international symposium on mixed and augmented reality, pp 193–202

Download references

Acknowledgments

This work was partially funded by the German BMBF project AVILUSplus (01IM08002).

Author information

Authors and Affiliations

  1. Fraunhofer IGD, Darmstadt, Germany

    Svenja Kahn

Authors
  1. Svenja Kahn
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Svenja Kahn.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kahn, S. Reducing the gap between Augmented Reality and 3D modeling with real-time depth imaging. Virtual Reality 17, 111–123 (2013). https://doi.org/10.1007/s10055-011-0203-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10055-011-0203-0

Keywords

Search

Navigation