ABSTRACT
This paper reports findings from a between-subjects experiment that investigates how different learning content representations in virtual environments (VE) affect the process and outcomes of learning. Seventy-eight participants were subjected to an immersive virtual reality (VR) application, where they received identical instructional information, rendered in three different formats: as text in an overlay interface, as text embedded semantically in a virtual book, or as audio. Learning outcome measures, self-reports, and an electroencephalogram (EEG) were used to compare conditions. Results show that reading was superior to listening for the learning outcomes of retention, self-efficacy, and extraneous attention. Reading text from a virtual book was reported to be less cognitively demanding, compared to reading from an overlay interface. EEG analyses show significantly lower theta and higher alpha activation in the audio condition. The findings provide important considerations for the design of educational VR environments.
Supplemental Material
Available for Download
Spreadsheet containing the full list of questions used
- Pavlo D. Antonenko and Andreas Keil. 2017. Assessing Working Memory Dynamics with Electroencephalography. Routledge, 93--111. DOI: http://dx.doi.org/10.4324/9781315296258--7Google ScholarCross Ref
- Pavlo Pasha Antonenko, Fred Paas, Roland Grabner, and Tamara Gog. 2010. Using Electroencephalography to Measure Cognitive Load. Educational Psychology Review 22 (12 2010), 425--438. DOI: http://dx.doi.org/10.1007/s10648-010--9130-yGoogle ScholarCross Ref
- Paul Ayres. 2006. Using subjective measures to detect variations of intrinsic cognitive load within problems. Learning and Instruction 16 (6 2006), 389--400. DOI: http://dx.doi.org/10.1016/j.learninstruc.2006.09.001Google ScholarCross Ref
- Robert J. Barry, Adam R. Clarke, Stuart J. Johnstone, Christopher A. Magee, and Jacqueline A. Rushby. 2007. EEG differences between eyes-closed and eyes-open resting conditions. Clinical Neurophysiology 118, 12 (2007), 2765 -- 2773. DOI: http://dx.doi.org/10.1016/j.clinph.2007.07.028Google ScholarCross Ref
- C. Berka, D. J. Levendowski, M. N. Lumicao, A. Yau, G. Davis, V. T. Zivkovic, R. E. Olmstead, P. D. Tremoulet, and P. L. Craven. 2007. EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. Journal of Aviation Space and Environmental Medicine 78, 5 (May 2007), B231--244.Google Scholar
- Johanna Bertram, Johannes Moskaliuk, and Ulrike Cress. 2015. Virtual training: Making reality work? Computers in Human Behavior 43 (02 2015), 284--292. DOI: http://dx.doi.org/10.1016/j.chb.2014.10.032Google ScholarDigital Library
- Doug A. Bowman, Ajith Sowndararajan, Eric D. Ragan, and Regis Kopper. 2009. Higher Levels of Immersion Improve Procedure Memorization Performance. In Proceedings of the 15th Joint Virtual Reality Eurographics Conference on Virtual Environments (JVRC'09). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 121--128. DOI: http://dx.doi.org/10.2312/EGVE/JVRC09/121--128Google ScholarCross Ref
- Fabio Buttussi and Luca Chittaro. 2017. Effects of Different Types of Virtual Reality Display on Presence and Learning in a Safety Training Scenario. IEEE Transactions on Visualization and Computer Graphics PP (01 2017), 1--1. DOI: http://dx.doi.org/10.1109/TVCG.2017.2653117Google ScholarCross Ref
- Leidy Castro-Meneses, Jan-Louis Kruger, and Stephen Doherty. 2019. Validating theta power as an objective measure of cognitive load in educational video. Educational Technology Research and Development (06 2019). DOI: http://dx.doi.org/10.1007/s11423-019-09681--4Google ScholarCross Ref
- Luca Chittaro and Fabio Buttussi. 2015. Assessing Knowledge Retention of an Immersive Serious Game vs. a Traditional Education Method in Aviation Safety. Visualization and Computer Graphics, IEEE Transactions on 21 (04 2015), 529--538. DOI: http://dx.doi.org/10.1109/TVCG.2015.2391853Google ScholarDigital Library
- Gabriele Cierniak, Katharina Scheiter, and Peter Gerjets. 2009. Explaining the split-attention effect: Is the reduction of extraneous cognitive load accompanied by an increase in germane cognitive load? Computers in Human Behavior 25 (03 2009), 315--324. DOI: http://dx.doi.org/10.1016/j.chb.2008.12.020Google ScholarDigital Library
- Cody Cox, Andrew House, Alex Lopez, and Gregory Pool. 2017. The Need for Conceptual Models of Technology in Training and Development: How Immersive Does Training Need to Be? Industrial and Organizational Psychology 10 (12 2017), 696--701. DOI: http://dx.doi.org/10.1017/iop.2017.78Google ScholarCross Ref
- Tilman Dingler, Kai Kunze, and Benjamin Outram. 2018. VR Reading UIs: Assessing Text Parameters for Reading in VR. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI EA '18). ACM, NY, NY, USA, Article LBW094, 6 pages. DOI: http://dx.doi.org/10.1145/3170427.3188695Google ScholarDigital Library
- Poul Dourish. 2001. Where the Action is: The Foundations of Embodied Interaction. MIT Press, Cambridge, MA, USA.Google Scholar
- EngageVR. 2019. EngageVR. https://engagevr.io/. (2019). [Online; accessed 01-09--2019].Google Scholar
- Chris Fowler. 2014. Virtual reality and learning: Where is the pedagogy?: Learning activities in 3-D virtual worlds. British Journal of Educational Technology 46 (02 2014). DOI: http://dx.doi.org/10.1111/bjet.12135Google ScholarCross Ref
- Anthony Gallagher and Christopher Cates. 2004. Virtual Reality Training for the Operating Room and Cardiac Catheterisation Laboratory. Lancet 364 (10 2004), 1538--40. DOI: http://dx.doi.org/10.1016/S0140--6736(04)17278--4Google ScholarCross Ref
- Sarah Garcia, Ronald Kauer, Denis Laesker, Jason Nguyen, and Marvin Andujar. 2019. A Virtual Reality Experience for Learning Languages. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA '19). ACM, NY, NY, USA, Article INT039, 4 pages. DOI: http://dx.doi.org/10.1145/3290607.3313253Google ScholarDigital Library
- D.M. Halliday, J.R. Rosenberg, A.M. Amjad, P. Breeze, B.A. Conway, and S.F. Farmer. 1995. A framework for the analysis of mixed time series/point process data-Theory and application to the study of physiological tremor, single motor unit discharges and electromyograms. Progress in Biophysics and Molecular Biology 64, 2 (1995), 237--278. DOI: http://dx.doi.org/10.1016/S0079--6107(96)00009-0Google ScholarCross Ref
- Daniel Harley, Aneesh P. Tarun, Daniel Germinario, and Ali Mazalek. 2017. Tangible VR: Diegetic Tangible Objects for Virtual Reality Narratives. In Proceedings of the 2017 Conference on Designing Interactive Systems (DIS '17). ACM, NY, NY, USA, 1253--1263. DOI: http://dx.doi.org/10.1145/3064663.3064680Google ScholarDigital Library
- Lasse Jensen and Flemming Konradsen. 2018. A review of the use of virtual reality head-mounted displays in education and training. Education and Information Technologies 23, 4 (2018), 1515--1529. DOI: http://dx.doi.org/10.1007/s10639-017--9676-0Google ScholarDigital Library
- Cheryl Johnson and Richard Mayer. 2012. An Eye Movement Analysis of the Spatial Contiguity Effect in Multimedia Learning. Journal of Experimental Psychology 18 (02 2012), 178--91. DOI: http://dx.doi.org/10.1037/a0026923Google ScholarCross Ref
- Mina Khan, Fernando Trujano, Ashris Choudhury, and Pattie Maes. 2018. Mathland: Playful Mathematical Learning in Mixed Reality. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI EA '18). ACM, NY, NY, USA, Article D108, 4 pages. DOI: http://dx.doi.org/10.1145/3170427.3186499Google ScholarDigital Library
- Pankaj Khanna, Insu Yu, Jesper Mortensen, and Mel Slater. 2006. Presence in Response to Dynamic Visual Realism: A Preliminary Report of an Experiment Study. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology (VRST '06). ACM, NY, NY, USA, 364--367. DOI: http://dx.doi.org/10.1145/1180495.1180569Google ScholarDigital Library
- David Kirsh. 2013. Embodied Cognition and the Magical Future of Interaction Design. ACM Transactions on Computer-Human Interaction 20, 1, Article 3 (April 2013), 30 pages. DOI: http://dx.doi.org/10.1145/2442106.2442109Google ScholarDigital Library
- George H. Klem, H. Lüders, Herbert H. Jasper, and C Elger. 1958. The ten-twenty electrode system of the International Federation. The International Federation of Clinical Neurophysiology. Electroencephalography and clinical neurophysiology. Supplement 52 (1958), 3--6.Google Scholar
- Wolfgang Klimesch. 1996. Memory processes, brain oscillations and EEG synchronization. International Journal of Psychophysiology 24, 1 (1996), 61 -- 100. DOI: http://dx.doi.org/10.1016/S0167--8760(96)00057--8 New Advances in EEG and cognition.Google ScholarCross Ref
- Wolfgang Klimesch. 1999. EEG alpha and theta oscillations reflect cognitive and memory performance: a review and analysis. Brain Research Reviews 29, 2 (1999), 169 -- 195. DOI: http://dx.doi.org/10.1016/S0165-0173(98)00056--3Google ScholarCross Ref
- Labster. 2019. Labster Simulations. https://www.labster.com/simulations/. (2019). [Online; accessed 01-09--2019].Google Scholar
- Colin Lankshear and Michele Knobel. 2003. New Literacies: Changing Knowledge and Classroom Learning (1 ed.). Open University Press.Google Scholar
- Marc Erich Latoschik, Daniel Roth, Dominik Gall, Jascha Achenbach, Thomas Waltemate, and Mario Botsch. 2017. The Effect of Avatar Realism in Immersive Social Virtual Realities. In Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology (VRST '17). ACM, NY, NY, USA, Article 39, 10 pages. DOI: http://dx.doi.org/10.1145/3139131.3139156Google ScholarDigital Library
- Robb Lindgren, Michael Tscholl, Shuai Wang, and Emily Johnson. 2016. Enhancing Learning and Engagement through Embodied Interaction within a Mixed Reality Simulation. Computers & Education 95, C (April 2016), 174--187. DOI: http://dx.doi.org/10.1016/j.compedu.2016.01.001Google ScholarDigital Library
- BF Lu, KT Lim, JM Zheng, and YY Cai. 2004. Learning Molecular Biology by VR Playing. In Proceedings of the 2004 ACM SIGGRAPH International Conference on Virtual Reality Continuum and Its Applications in Industry (VRCAI '04). ACM, NY, NY, USA, 168--172. DOI: http://dx.doi.org/10.1145/1044588.1044622Google ScholarDigital Library
- Guido Makransky, Stefan Borre-Gude, and Richard E. Mayer. 2019a. Motivational and cognitive benefits of training in immersive virtual reality based on multiple assessments. Journal of Computer Assisted Learning (2019). DOI: http://dx.doi.org/10.1111/jcal.12375Google ScholarCross Ref
- Guido Makransky and Gustav Bøg Petersen. 2019. Investigating the process of learning with desktop virtual reality: A structural equation modeling approach. Computers & Education 134 (2019), 15 -- 30. DOI: http://dx.doi.org/https://doi.org/10.1016/j.compedu.2019.02.002Google ScholarDigital Library
- Guido Makransky, Thomas S. Terkildsen, and Richard E. Mayer. 2019b. Role of subjective and objective measures of cognitive processing during learning in explaining the spatial contiguity effect. Learning and Instruction 61 (1 6 2019), 23--34. DOI: http://dx.doi.org/10.1016/j.learninstruc.2018.12.001Google ScholarCross Ref
- Guido Makransky, Thomas S. Terkildsen, and Richard E. Mayer. 2019c. Adding immersive virtual reality to a science lab simulation causes more presence but less learning. Learning and Instruction 60 (2019), 225 -- 236. DOI: http://dx.doi.org/10.1016/j.learninstruc.2017.12.007Google ScholarCross Ref
- Guido Makransky, Philip Wismer, and Richard E. Mayer. 2019d. A gender matching effect in learning with pedagogical agents in an immersive virtual reality science simulation. Journal of Computer Assisted Learning 35, 3 (2019), 349--358. DOI: http://dx.doi.org/10.1111/jcal.12335Google ScholarCross Ref
- Anne Mangen and Theresa Schilhab. 2012. An embodied view of reading: Theoretical considerations, empirical findings, and educational implications. Skriv! Les! (2012), 285--300.Google Scholar
- Richard E. Mayer. 2014. The Cambridge Handbook of Multimedia Learning (2 ed.). Cambridge University Press. DOI: http://dx.doi.org/10.1017/CBO9781139547369Google ScholarCross Ref
- Richard E. Mayer and Richard B. Anderson. 1992. The Instructive Animation: Helping Students Build Connections between Words and Pictures in Multimedia Learning. Journal of Educational Psychology 84, 4 (1992), 444.Google ScholarCross Ref
- Zahira Merchant, Ernest T. Goetz, Lauren Cifuentes, Wendy Keeney-Kennicutt, and Trina J. Davis. 2014. Effectiveness of Virtual Reality-based Instruction on Students' Learning Outcomes in K-12 and Higher Education. Computers & Education 70 (Jan. 2014), 29--40. DOI: http://dx.doi.org/10.1016/j.compedu.2013.07.033Google ScholarCross Ref
- Oliver A. Meyer, Magnus K. Omdahl, and Guido Makransky. 2019. Investigating the effect of pre-training when learning through immersive virtual reality and video: A media and methods experiment. Computers & Education 140 (2019), 103603. DOI: http://dx.doi.org/10.1016/j.compedu.2019.103603Google ScholarDigital Library
- Tassos A. Mikropoulos and Antonis Natsis. 2011. Educational Virtual Environments: A Ten-year Review of Empirical Research (1999--2009). Computers & Educaction 56, 3 (April 2011), 769--780. DOI: http://dx.doi.org/10.1016/j.compedu.2010.10.020Google ScholarDigital Library
- Roxana Moreno and Richard E. Mayer. 2002. Verbal Redundancy in Multimedia Learning: When Reading Helps Listening. Journal of Educational Psychology 94, 1 (2002). DOI: http://dx.doi.org/10.1037/0022-0663.94.1.156Google ScholarCross Ref
- Johannes Moskaliuk, Johanna Bertram, and Ulrike Cress. 2012. Training in virtual environments: Putting theory into practice. Ergonomics 56 (12 2012). DOI: http://dx.doi.org/10.1080/00140139.2012.745623Google ScholarCross Ref
- Matthias Oberhauser and Daniel Dreyer. 2017. A Virtual Reality Flight Simulator for Human Factors Engineering. Cogn. Technol. Work 19, 2--3 (Sept. 2017), 263--277. DOI: http://dx.doi.org/10.1007/s10111-017-0421--7Google ScholarDigital Library
- Daria Osipova, Atsuko Takashima, Robert Oostenveld, Guillén Fernández, Eric Maris, and Ole Jensen. 2006. Theta and Gamma Oscillations Predict Encoding and Retrieval of Declarative Memory. Journal of Neuroscience 26, 28 (2006), 7523--7531. DOI: http://dx.doi.org/10.1523/JNEUROSCI.1948-06.2006Google ScholarCross Ref
- Fred Paas. 1992. Training Strategies for Attaining Transfer of Problem-Solving Skill in Statistics: A Cognitive-Load Approach. Journal of Educational Psychology 84 (12 1992), 429--434. DOI: http://dx.doi.org/10.1037/0022-0663.84.4.429Google ScholarCross Ref
- Paul R Pintrich, David A F Smith, Teresa Garcia, and Wilbert James McKeachie. 1991. A Manual for the Use of the Motivated Strategies for Learning Questionnaire (MSLQ). National Center for Research to Improve Postsecondary Teaching and Learning (1991).Google Scholar
- Sébastien Puma, Nadine Matton, Pierre-Vincent Paubel, Eric Raufaste, and Radouane Yagoubi. 2017. Using theta and alpha band power to assess cognitive workload in multitasking environments. International journal of psychophysiology : official journal of the International Organization of Psychophysiology 123 (10 2017). DOI: http://dx.doi.org/10.1016/j.ijpsycho.2017.10.004Google ScholarCross Ref
- Pei-Luen Patrick Rau, Jian Zheng, Zhi Guo, and Jiaqi Li. 2018. Speed reading on virtual reality and augmented reality. Computers & Education 125 (2018), 240 -- 245. DOI: http://dx.doi.org/10.1016/j.compedu.2018.06.016Google ScholarDigital Library
- W. J. Ray and H. W. Cole. 1985. EEG alpha activity reflects attentional demands, and beta activity reflects emotional and cognitive processes. Science 228, 4700 (May 1985), 750--752.Google ScholarCross Ref
- Giuseppe Riva. 2008. From virtual to real body: Virtual reality as embodied technology. Journal of Cyber Therapy and Rehabilitation 1 (01 2008), 7--22.Google Scholar
- Katja Rogers, Jana Funke, Julian Frommel, Sven Stamm, and Michael Weber. 2019. Exploring Interaction Fidelity in Virtual Reality: Object Manipulation and Whole-Body Movements. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, NY, NY, USA, Article 414, 14 pages. DOI: http://dx.doi.org/10.1145/3290605.3300644Google ScholarDigital Library
- Gavriel Salomon. 1984. Television is easy and print is tough. The differential investment of mental effort in learning as a function of perceptions and attributions. Journal of Educational Psychology 76 (1984), 647--658. DOI: http://dx.doi.org/10.1037/0022-0663.76.4.647Google ScholarCross Ref
- Christian Scharinger. 2018. Fixation-related EEG frequency band power analysis: A promising methodology for studying instructional design effects of multimedia learning material. Frontline Learning Research 6 (12 2018), 57--71. DOI: http://dx.doi.org/10.14786/flr.v6i3.373Google ScholarCross Ref
- Jinsil Hwaryoung Seo, Brian Michael Smith, Margaret E. Cook, Erica R. Malone, Michelle Pine, Steven Leal, Zhikun Bai, and Jinkyo Suh. 2017. Anatomy Builder VR: Embodied VR Anatomy Learning Program to Promote Constructionist Learning. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '17). ACM, NY, NY, USA, 2070--2075. DOI: http://dx.doi.org/10.1145/3027063.3053148Google ScholarDigital Library
- Daniel Sjölie, Kenneth Bodin, Eva Elgh, Johan Eriksson, Lars-Erik Janlert, and Lars Nyberg. 2010. Effects of Interactivity and 3D-motion on Mental Rotation Brain Activity in an Immersive Virtual Environment. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, NY, NY, USA, 869--878. DOI: http://dx.doi.org/10.1145/1753326.1753454Google ScholarDigital Library
- Charles Spence. 2009. Explaining the Colavita visual dominance effect. In Attention, Narayanan Srinivasan (Ed.). Progress in Brain Research, Vol. 176. Elsevier, 245 -- 258. DOI: http://dx.doi.org/10.1016/S0079--6123(09)17615-XGoogle ScholarCross Ref
- Misha Sra, Xuhai Xu, Aske Mottelson, and Pattie Maes. 2018. VMotion: Designing a Seamless Walking Experience in VR. In Proceedings of the 2018 Designing Interactive Systems Conference (DIS '18). ACM, NY, NY, USA, 59--70. DOI: http://dx.doi.org/10.1145/3196709.3196792Google ScholarDigital Library
- John Sweller. 1994. Cognitive load theory, learning difficulty, and instructional design. Learning and Instruction 4, 4 (1994), 295--312. DOI: http://dx.doi.org/10.1016/0959--4752(94)90003--5Google ScholarCross Ref
- Rustin Webster. 2015. Declarative knowledge acquisition in immersive virtual learning environments. Interactive Learning Environments (01 2015), 1--15. DOI: http://dx.doi.org/10.1080/10494820.2014.994533Google ScholarCross Ref
- William Winn. 2003. Learning in Artificial Environments: Embodiment, Embeddedness and Dynamic Adaptation. Technology, Instruction, Cognition and Learning 1 (01 2003).Google Scholar
- Mihye Won, M. Mocerino, Kok-Sing Tang, David Treagust, and Roy Tasker. 2019. Interactive Immersive Virtual Reality to Enhance Students' Visualisation of Complex Molecules. Springer, Singapore, 51--64. DOI: http://dx.doi.org/10.1007/978--981--13--6998--8_4Google ScholarCross Ref
- Anna Wong, Wayne Leahy, Nadine Marcus, and John Sweller. 2012. Cognitive load theory, the transient information effect and e-learning. Learning and Instruction 22, 6 (2012), 449 -- 457. DOI: http://dx.doi.org/10.1016/j.learninstruc.2012.05.004Google ScholarCross Ref
Index Terms
Investigating Representation of Text and Audio in Educational VR using Learning Outcomes and EEG
Recommendations
Pedagogical Agents in Educational VR: An in the Wild Study
CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing SystemsPedagogical agents are theorized to increase humans’ effort to understand computerized instructions. Despite the pedagogical promises of VR, the usefulness of pedagogical agents in VR remains uncertain. Based on this gap, and inspired by global efforts ...
VR sickness measurement with EEG using DNN algorithm
VRST '18: Proceedings of the 24th ACM Symposium on Virtual Reality Software and TechnologyRecently, VR technology is rapidly developing and attracting public attention. However, VR Sickness is a problem that is still not solved in the VR experience. The VR sickness is presumed to be caused by crosstalk between sensory and cognitive systems [...
Electroencephalographic (EEG) Correlates of Visually Induced Motion Sickness (VIMS) in the Virtual Reality (VR) Based Simulations
Human-Computer Interaction – INTERACT 2023AbstractIn the presented study, the possibility of detecting Visually Induced Motion Sickness (VIMS) in Virtual Reality (VR) simulations using electroencephalographic (EEG) recordings was investigated. 31 adult participants were tested with VR and EEG in ...
Comments