Remediating learning from non-immersive to immersive media: Using EEG to investigate the effects of environmental embeddedness on reading in Virtual Reality

Highlights

• Study examined if traditional written learning content is experienced and cognized differently when embedded in immersive VR.

• EEG and objective measures were used to assess how environmental embeddedness affects cognition during reading.

• Results show that reading in VR yields higher transfer, demands more cognitive engagement, and is less time efficient.

• Findings suggest that environmental embeddedness is a powerful VR affordance to promote learning.

• Study urges future initiatives to strategically target unique VR affordances when designing immersive learning content.

Abstract

Virtual Reality (VR) has the potential to enrich education but little is known about how unique affordances of immersive technology might influence leaning and cognition. This study investigates one particular affordance of VR, namely environmental embeddedness, which enables learners to be situated in simulated or imagined settings that contextualize their learning. A sample of 51 university students were administered written learning material in a between-subjects design study, wherein one group read text about sarcoma cancer on a physical pamphlet in the real world, and the other group read identical text on a virtual pamphlet embedded in an immersive VR environment which resembled a hospital room. The study combined advanced EEG measurement techniques, learning tests, and cognitive load measures to compare conditions. Results show that the VR group performed significantly better on a knowledge transfer post-test. However, reading in VR was found to be more cognitively effortful and less time-efficient. Findings suggest the significance of environmental embeddedness for learning, and provide important considerations for the design of educational VR environments, as we remediate learning content from non-immersive to immersive media.

Introduction

In her visionary book The Hamlet on the Holodeck, written at the time of analogue television, Murray (1997) argued that the holodeck – a universal fantasy machine from Star Trek– is not at all a farfetched imagination of reality; and optimistically predicted that in the very near future we will all be able to enter “an illusory world that can be stopped, started, or turned off at will, but that looks and behaves like the actual world and includes parlor fires, drinkable tea, and characters, who can be touched, conversed with, and even kissed” (p. 15). Only a few decades later, echoing the central thesis of Murray's book, we see our media landscape being evermore densely populated with a wide choice of accessible head mounted displays (HMDs), real-time 3D content creation engines, sophisticated data gloves, and physical manipulation devices. These technologies all offer immense possibilities to represent simulated and imagined wonderlands, expanding our physical and sensory capabilities, and allowing us to experience the inexperienced in a wide breadth of fields, disciplines, and aspects of everyday life (Brooks, 1999; Sherman & Craig, 2018).

Considering such a multifaceted nature of Virtual Reality (VR), it is not surprising that in the last decade we have also witnessed a surge of interest in immersive technologies in education (Howard, 2019; Jensen & Konradsen, 2018; Merchant et al., 2014; Mikropoulos & Natsis, 2011; Muller Queiroz et al., 2018; Raditanti et al., 2020; Snelson & Hsu, 2019). In the last few years, we have moved away from understanding VR as a complex and costly simulator, sporadically available for training of pilots and medical staff (Lee, 2017; Seymour et al., 2002), to conceiving it as an affordable tool that is accessible to an everyday learner (Jones, 2018). Exemplifying this, a recent review by Radianti et al. (2020) identified 18 application domains of VR for education, indicating that there is profound interest in using immersive technologies in different learning scenarios. Some, for instance, have explored VR for training practical skills and procedural knowledge in the domains of engineering, public safety and technical personnel training (Butussi & Chittaro, 2018; Feng et al., 2018; Çakiroğlu & Gökoğlu, 2019). Others, have attempted to use it for teaching declarative and conceptual knowledge in various STEM related disciplines (Jang et al., 2017; Lindgren et al., 2016; Makransky et al., 2020), as well as in the humanities such as arts, language learning, and history (Alfadil, 2020; Checa & Bustillo, 2020; Degli Innocenti et al., 2019). VR is also used to develop and inspire communication, collaboration and other soft skillsets that are at the core of pedagogy and learning (Howard & Gutworth, 2020). Nevertheless, despite this initial excitement surrounding immersive education, most existing applications of VR have been experimental in nature, and little is known about how effective this new technology is for learning (Jensen & Konradsen, 2018; Radianti et al., 2020). On the one hand, incipient research has started to suggest its benefits for student engagement, development of deep knowledge, and knowledge transfer to the real world (Makransky et al., 2019a). Others have suggested that, if not designed appropriately, VR-based learning activities can also lead to unnecessary overload on our cognitive and perceptual systems (Makransky et al., 2019b; Meyer et al., 2019; Moreno & Mayer, 2002; Parong & Mayer, 2018).

In the light of recent global challenges, it has become evident that VR has yet to be implemented in daily teaching routines (Radianti et al., 2020). While many educators acknowledge the benefits of using immersive tools to aid online learning, existing virtual educational content is rather limited, and is often hard to customize and integrate into standard educational contexts (Johnston, 2018). Additionally, instructional designers and software development companies conceiving these tools have few guidelines and frameworks for designing this new technology (Johnson-Glenberg, 2019; Makransky et al., 2020). Present VR applications are, therefore, often based not on specialized learning content and dedicated pedagogy methods, but rather ground themselves in routine and elementary translations of longstanding educational materials into the virtual space. As such, given that to this day the most common way to represent learning content is still through written and spoken language (Lankshear & Knobel, 2003), most educational VR tools are also being designed around these traditional forms of representation (see Fig. 1). In some educational VR environments, for example, learning content as we know it is embedded in immersive simulations of real situations that are hard to access in real life (e.g. (EngageVR, 2019; Labster, 2019)). In others, written or spoken learning information is accompanied by exquisite human-scale visualizations of non-existent, or otherwise invisible, micro-scale phenomena (e.g. (Bowman et al., 2009; Webster, 2016). However, in both cases, even if learning takes place in an immersive medium, we are still acquiring knowledge from traditional information representations, established in preceding media formats (e.g. books, webpages, etc.).

Bolter and Grusin (1999) coin this process of translating elements from one media system to another as remediation, and define it as a natural phenomenon that persists with the evolution of media technologies. They propose that as we move from one media to another, there is a certain point in time when two media systems grow and co-evolve together (Bolter & Grusin, 1999). In this overlap they borrow elements, standards and ways of thinking from one another. As we transition from traditional learning devices into the realms of VR, it is therefore not surprising that we are borrowing and integrating elements, guidelines and criteria from media that we are familiar with into new systems that we know little about. According to Richards (2000), however, with new media this shift is different, and it requires a major “readjustment of the alphabet”. New media is predominantly visual as well as dialogical, which involves symbolic extensions of the human body as well as “spatialization” of the human thought (Richards, 2000) – aspects that have not been explored in traditional media before. As this shift is a significant one, there is a need and a certain sense of responsibility for us as designers and educators to question if and how this move affects the ways we cognize and make sense of the world – the fundamental aspects in which learning and education are rooted. Specifically, whether these new modes of representation influence the way we experience and consume traditional spoken or written learning content; or whether VR is a mere technological delivery platform, a new kind of interface for learning, that concerns itself only with the technical features of immersion in a simulated world?

With this aim in mind, in the present study we intend to investigate what happens to our learning and cognition as we remediate learning content from traditional non-immersive media to immersive VR. In particular, we continue the work by Makransky and colleagues (2019b), who have previously examined direct learning content remediation from a desktop computer-enabled system to immersive VR. In their study, the authors showed that with direct remediation educational efficacy of a medium suffers. However, they posed an open question that it perhaps could be reconciled if learning content were to be designed considering the unique affordances of VR for learning. This idea has also been explored in a study by Lindgren et al. (2016), wherein the authors investigated learning effectiveness of an immersive Mixed Reality (MR) system, comparing it to a non-immersive computer simulation. Even though students were presented identical learning content in both systems, their study found that MR was superior in terms of conceptual understanding, student engagement and attitudes towards learning. The authors suggested that this was due to the fact that the MR environment was designed to capitalize on one unique affordance of immersive media – embodied interaction – a representation mode that is not afforded by 2D media. A similar notion had been earlier suggested by Regian (1997), who conducted two studies comparing training transfer in the same navigation task between VR and computer-assisted instruction. In one of the studies the author found no difference between immersive and non-immersive representation. In the second study, however, VR was found to be superior to computer-assisted instruction after modifying specific instructional design parameters to benefit from the affordances of VR (i.e. changing perspectives from birds-eye view to immersed view). In the current study we further these investigations of remediation by capitalizing on one particular affordance of VR – environmental embeddedness. Specifically, we want to investigate if traditional learning content is experienced and cognized differently when we embed it in an immersive virtual environment afforded by VR. According to Winn (2003), in immersive environments “learning is the result of adaptation of the learner to the environment and the environment to the learner”, indicating that environmental embeddedness might influence the way we learn in VR. Similarly, Dlgarno and Lee (2010) have previously identified this affordance as one critical characteristic of 3D virtual learning environments to determine the efficacy of learning. According to the authors, environmental simulations have the ability to contextualize learning and facilitate learning tasks that might lead to improved knowledge and skills transfer when learning in VR (Dlgarno & Lee, 2010).

Extending the work of Makransky et al. (2019b), in this study we want to offer more control for learning content used during remediation. Learning content previously investigated by the authors was a virtual simulation, which consisted of various content representations such as written textual information, spoken auditory information, visual illustrations, and animations. While this provided a good first step for general understanding of content remediation from non-immersive to immersive media, echoing the semiotic view of remediation proposed by Sherman and Craig (2018), herein we argue that all of these content representations can be considered as idiosyncratic media systems. In this direction, a recent study has highlighted that substantial differences in learning and cognition exist when processing auditory versus written information formats in VR (Baceviciute et al., 2020). As such, even though combining several content representation forms might have ecological value, we propose that it might be that each of these systems is affected by remediation to immersive VR in distinct ways. In this study we therefore choose to investigate one specific content representation format, offering less variability and more control in understanding learning during remediation. Aiming to find a prototypical use case for remediated content based on industry trends, as well as comprehending that to this day reading is one of the most common ways of accessing knowledge (i.e. articles, blogposts, social media posts, books, websites, etc.), we focus on this process through the lens of reading written learning content representations (see Fig. 1). In particular, in this study we aim to investigate the continuous reading process, which we believe to be more relevant to studies of learning, as opposed to isolated word or sentence reading scenarios, which are predominant in literary and cognitive investigations. Similarly, due to the imposed educational context, we are specifically interested in novel expository texts (i.e. texts that hold new factual, non-narrative information), allowing for didactic meaning-making processes to take place during reading.

Finally, in the current work the intent is to move away from the existing focus of assessing educational value of VR through prototype evaluations and user attitude ratings (Cox et al., 2017; Jensen & Konradsen, 2018; Moskaliuk et al., 2013), and to incorporate a more rigorous experimental multi-method research strategy that combines subjective, behavioral, and cognitive learning outcomes. To account for the objective online experience during the process of learning, in this approach we employ advanced methods within electroencephalography (EEG) research, which allows us to gain an understanding of how the experience of reading written learning content might be influenced by remediation from traditional learning contexts to immersive VR at the cognitive level of analysis.

The aforementioned incipient research on unique affordances of VR, current trends in the immersive educational technology industry, and the lack of objective empirical studies investigating the relations between the learner, the environment and the learning content, motivates the central research question of this article - how does the affordance of environmental embeddedness influence learning during the remediation of written learning content from non-immersive media to immersive VR (R1). Additionally, this study examines how such environmental embeddedness influences cognitive processing as compared to a traditional reading setting in order to better understand how distinct media differ in cognitive load demands (R2). In other words, we are specifically focused on understanding whether we are overwhelming learners by embedding them in a virtual environment, possibly straining their cognitive apparatuses (Makransky et al., 2019b; Moreno & Mayer, 2002; Parong & Mayer, 2018); or whether these external representations act as cognitive aids that allow us to link knowledge and experience, which can be used for aiding reflection and imagination (Kuzmicova, 2015; Mangen & Schilhab, 2012)? In an attempt to evaluate these questions empirically, we set up a cross-media between-subjects comparison experiment, where we contrast reading didactic text from a physical booklet while being situated in a real world, with reading an identical text, but presented on pages of a virtual booklet in VR, embedded in a virtual environment. We combine knowledge from embodied cognition and latest findings from educational VR and EEG research to drive our research methods and experimental design choices. Our approach is in line with the integrative framework proposed by Mangen and van der Weel (2016), wherein the authors have expressed a growing need to establish dialogues across diverse disciplines and paradigms in future reading and learning inquiries.