Wearing a Hololens: A New Dimension to Remote Presence in Education Sebastian Hahta University of Turku Turku, Finland joseha@utu.fi Maryam Teimouri University of Turku Turku, Finland mtebad@utu.fi Tomi "bgt" Suovuo University of Turku Turku, Finland bgt@utu.fi Selma Auala Namibia University of Science and Technology Windhoek, Namibia selmaauala15@gmail.com Erkki Rötkönen University of Turku Turku, Finland rotkonenerkki@utu.fi Jason Mendes Namibia University of Science and Technology Windhoek, Namibia Jaxmendes2@gmail.com Naska Goagoses Carl von Ossietzky University of Oldenburg Oldenburg, Germany naska.goagoses@uni-oldenburg.de Heike Winschiers-Theophilus Namibia University of Science and Technology Windhoek, Namibia hwinschiers@nust.na Erkki Sutinen University of Turku Turku, Finland erkki.sutinen@utu.fi ABSTRACT Based on a prior developed platform, we have implemented a Mixed Reality immersive 3D telepresence technology using Microsoft’s Hololens. We have used this technology in a one week co-design workshop in Namibia, as well as in 3-day trial between Namibia and Finland, to explore and to derive future development strate- gies in regard to the technological capabilities as well as integra- tion into educational systems. The contemporary teleconferencing provides a very narrow set of modalities, which hinders engage- ment and effective teaching, especially of primary students who require the teacher to nurture them to learn. Through 3D immer- sive telepresence, as we envision it, this gap could be narrowed do wn.Participants in our demonstration can observe this system in operation and see how it works in practice with a Hololens 2 head- set. The 3D capture part of the system will also be demonstrated in a proximate location. CCS CONCEPTS • Human-centered computing → Mixed / augmented reality; Participatory design; • Information systems → Internet com- munications tools. KEYWORDS Mixed Reality, hololens, depth camera, immersion, education ACM Reference Format: Sebastian Hahta, Maryam Teimouri, Tomi "bgt" Suovuo, Selma Auala, Erkki Rötkönen, Jason Mendes, Naska Goagoses, Heike Winschiers-Theophilus, and Erkki Sutinen. 2023. Wearing a Hololens: A New Dimension to Remote Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). C&T23, June 2023, Finland © 2023 Copyright held by the owner/author(s). https://doi.org/10.48340/ct2023-2822 Presence in Education. In ACM International Conference. ACM, New York, NY, USA, 4 pages. https://doi.org/10.48340/ct2023-2822 1 INTRODUCTION In an ever more connected world, striving for global education, studying technologies to support remote presence as a concept, its use and affordances, becomes more relevant than ever. While mixed reality (MR) wearables are available on the market, user experiences in educational settings are currently poorly studied, especially concerning the Microsoft HoloLens 2 [4, 5]. We maintain that an improved understanding of how the technology can and should be used in education is essential to inform further research and development agendas. We further postulate that while MR live video streaming is at an early technical stage for wide adoption, we need to involve students and teachers as well as educational and technology experts to explore new ways of technology enhanced learning [10]. Thus under a four-year interdisciplinary research project, we explore among other aspects, the co-creation of im- mersive MR video affordances and applications with children and teachers in Namibia and Finland. By working with students and teachers in diverse school contexts, we draw on their imaginations and social demands to inform further steps in addressing social and emotional deficiencies with existing technologies that are used for shared remote education. We use Lidar cameras and Hololenses with depth camera based capture, aiming for a practical portable, low-latency systemwith reasonable network requirements on band- width and latency, in order to support distributed teaching and learning activities. In this paper, we first introduce current developments in ex- tended realities, then the project context and recent activities. We describe the technology and demo setup before concluding with future developments. https://doi.org/10.48340/ct2023-2822 https://doi.org/10.48340/ct2023-2822 C&T23, June 2023, Finland Hahta et al. 2 EXTENDING REALITIES User interfaces are in fast development. From the times of punch- cards to text terminals, then graphical user interface and now ex- tended reality in head mounted displays. Computer mediated com- munication has followed this development from e-mail to 2D video conferencing, which is depicted in Figure 1a. We are gazing at the dreams presented by science fiction in form of manifesting holographic images of people participating in discussions with full spatial affordances to interact in a shared space. Figure 1: Different contemporary methods for experiencing MR. Virtual Reality (VR) headsets are common and provide an easy way to test new immersive technologies allowing users to experi- ence the virtual environment in a more natural way. As depicted in Figure 1b, this method blocks the natural reality (NR) from the user’s view, replacing it with the highly immersive VR. However they are not as practical in Mixed Reality (MR) contexts, depicted in Figure 1c, where virtual content is to be overlaid or otherwise mixed with the NR, as the headset usually completely blocks the user’s vision to their surroundings. Smart glasses have been avail- able for long, but as heads-up display they lack similar immersion to VR headsets because the virtual content is not fixed relative to the real environment. Latest MR headsets, such as the Microsoft’s Hololens, have VR like immersion without limiting user’s vision to the real environment. The applicability of VR headsets in communication has been studied in the past, but their inherent limitations are problematic: in addition to blocking the user’s view to their surroundings, the headset will also cover the user’s face from other participants, re- mote or local. MR headsets do not suffer from these limitations in the same extent and therefore provide interesting opportuni- ties by making natural interaction with both a local and virtual environment possible. There are multiple projects working on Mixed Reality telepres- ence with creating a 3D representation of people and transmitting it over the network. In 2016 Microsoft’s demonstrated their Holopor- tation system consisting of a Hololens and a capture system based on multiple pairs of stereo cameras [2]. Alternative approaches for 3D capture are also actively researched, for example Meta’s photorealistic animated avatars for AR/VR, which uses a machine learning model to generate 3D representation from normal 2D camera images [7]. 3 PROJECT CONTEXT The Beyond the Imaginable Technologies for Sustaining Remote Life (BIT::TIP) project aims at removing the critical research hurdles that are holding back the use of sensory immersive 3D video as an alternative to ordinary videoconferencing. Quality and perfor- mance are key targets, but so is an improved understanding of how the technology can and should be used, as well as providing the evidence that industry needs to invest. By working with children in a challenging school context, we utilise their imaginations and social demands to drive the final steps in solving the numerous so- cial and emotional deficiencies with existing technologies that are used for shared remote work, school and in our personal lives. The wellbeing of a digital and remote society will rely upon improved technologies if it is to be sustained in the future. Over the past one-and-a-half years, the project was concerned with establishing desirable affordances for wearable MR technolo- gies to be used in distributed educational settings. From a peda- gogical perspective, we deemed it necessary to frame our work with fitting theories and past empirical work, as well as integrating educational constructs that are considered vital for learning, yet have not received so much attention within the development of new educational technology. This includes constructs such as the classroom climate and academic engagement, which were chosen as starting points. Classroom climate is defined as “every aspect of the school ex- perience, including the quality of teaching and learning, school community relationships, school organization, and the institutional and structural features of the school environment.” (p. 315) [11]. Classroom climate is thus seen as a multidimensional construct, with one major component being the community domain, which focuses around relationships with teachers and classmates, social- emotional support, and feelings of connectedness [12]. As a first step within the project, we conducted a systematic literature re- view about the creation of a social classroom climate in online and technology-enhanced learning environments in school settings [1]. Although including 29 articles in the thematic synthesis, we found that most research was utilizing and adapting available technolo- gies, and none worked with immersive technologies. We found that the introduction of technologies with different features provided new opportunities for social relationships in and beyond the class- room. Specifically, relationships were fostered between students that would normally not mingle with each other, students had more Wearing a Hololens: A New Dimension to Remote Presence in Education C&T23, June 2023, Finland choices on how to express themselves and participate, and infor- mal exchanges between students and teachers were possible. The systematic review also highlighted limitations in current research, such as a lack of (1) clear theoretical frameworks, (2) the inclusion of students’ voices, and (3) a diverse set of participants. In a next phase of the project, we focused on the construct of aca- demic engagement, which is defined as “the quality of a student’s connection or involvement with the endeavor of schooling and hence with the people, activities, goals, values, and place that com- pose it” [9, p. 494]. Based on the Self-System Model of Motivational Development [8], and conceptual distinctions between the types of academic engagement (i.e., emotional, behavioral, agentic engage- ment, [6]), we run a one-week co-design workshop with Namibian primary students and teachers, consisting of distinct sessions [5]. The aim of the workshop was to explore opportunities, desires and needs from within a real educational setting. By utilising a co- design approach, we ensured that we include the perspective and ideas of students and teachers. Furthermore, co-design enables the end-users to get involved in early stages of the conceptualization and design of new technology. In the workshop, students and teachers in Namibia were first acquainted with the Hololens. They wore the Hololens and saw the holograms of other students/teachers mixed with the surrounding realities. Focus groups interviews and role-plays centered around academic engagement were conducted. Participants were also given Legos and cardboard boxes to build conceptual models for how holographic students and teachers should be facilitated in their in- teractions with physically present students and teachers. The study focused on attaining first insights into students and teachers per- spectives on academic engagement when holographic technology is integrated into classrooms. Recently, we conducted an evaluation case study, in which teach- ers in Namibia, wearing a Hololens, gave short engaging lessons to a set of students in the same physical space, as well as a Hololens- wearing student from Finland. This setup, as depicted in Figure 2, was then repeated with a teacher in Finland, and students from Finland and Namibia. A preliminary analysis and observation of the sessions revealed the need for a number of technical improvements ranging from audio to visuals, as well as shared spaces to enhance teacher-student engagement. 4 TECHNOLOGY DESCRIPTION The goal of the technology is to produce live 3D holographic telep- resence experience. The concept is similar to what is demonstrated in Microsoft’s Holoportation demonstration [2], but we focus on having a light weight and portable system using commercial off- the-shelf hardware. The system consists of three main software components: live 3D capture and transmission, rendering and holographic display and a web service providing streaming and configuration. 3D capture is implemented with a depth camera, Intel Realsense L515, which provides both color and depth video at 30 frames per second. Both color and depth are then encoded with a HEVC codec. Depth is encoded in YUV channels, allowing the use of standard color video codecs [3]. In addition, the required bandwidth is significantly reduced by masking the background pixels, leaving Figure 2: Hololens and a depth camera in a case study in Finland and Namibia only the foreground objects visible and to be transmitted. The overall processing latency is approximately 50ms, in addition to any network latency. The client renders virtual viewpoints from the color and depth images using our custom CUDA renderer. The viewpoints are then displayed on a Microsoft HoloLens 2 MR headset. Hololens is inte- grated to the system with Unreal Engine plugin, which provides the interface between our software stack and the Hololens headset. The headset will overlay the hologram on the local environment through its transparent stereoscopic display. The exact positioning of the hologram can be done in advance to fix the hologram in specific location in the physical space. 5 DEMO SET-UP 3D capture and holographic display will be demonstrated at the conference. The demo is implemented using a single depth camera and a Hololens 2 headset, both on the conference site. A person appearing on the camera will be rendered as hologram for another person who is wearing the headset. A video stream from the headset camera, which includes the overlaid virtual content, is also shown on a screen to give another point of view for other observers. This setup allows participants to get an impression how the system works in practice, considering both the capture and the display. 6 FUTURE DEVELOPMENT Current machine learning based alternatives for 3D capture, which multiple research groups are working on, are not yet capable of gen- erating personalized avatars without offline training. As our goal is to include flexible groups in our case studies, in addition to people the system also needs to be able to consider the physical scene in addition to avatars. Achieving a good hologram image quality is challenging with a depth camera based 3D capture. Inherent noise in the data and occlusions in the scene appear as artifacts in the ren- dered image, becoming more noticeable as the viewpoint diverges C&T23, June 2023, Finland Hahta et al. from camera’s origin. Improving the visual quality of the hologram by addressing challenges in 3D capture remains an area of develop- ment on which we continue to work on. Equally the involvement of stakeholders from the educational sector remains a concern in early development decisions of technologies, thus we will reinforce the engagement of teachers and students, considering pedagogical theories and devising adequate methods of participation. ACKNOWLEDGMENTS The project is funded by the Academy of Finland (343364), with additional funding by the Finish Cultural Foundation. Edtech design activities in Namibia are supported by MTC Namibia. REFERENCES [1] Naska Goagoses, Tomi "bgt" Suovuo, Heike Winschiers-Theophilus, Calkin Suero Montero, Nicolas Pope, Erkki Rötkönen, and Erkki Sutinen. 2023. A sys- tematic review of social classroom climate in online and technology-enhanced learning environments in primary and secondary school. Education and Informa- tion Technologies (2023). https://doi.org/10.1007/s10639-023-11705-9 [2] Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh Kowdle, Yury Degtyarev, David Kim, Philip L Davidson, Sameh Khamis, Ming- song Dou, et al. 2016. Holoportation: Virtual 3d teleportation in real-time. In Proceedings of the 29th annual symposium on user interface software and technology. ACM, New York, 741–754. [3] Fabrizio Pece, Jan Kautz, and TimWeyrich. 2011. Adapting standard video codecs for depth streaming.. In EGVE/EuroVR. 59–66. [4] Nikolaos Pellas, Ioannis Kazanidis, and George Palaigeorgiou. 2020. A systematic literature review of mixed reality environments in K-12 education. Education and Information Technologies 25, 4 (2020), 2481–2520. [5] Nicolas Pope, Sebastian Hahta, Erkki Rötkönen, Naska Goagoses, Jason Mendes, Erkki Sutinen, and Heike Winschiers-Theophilus. 2022. Remote Presence: Live Holograms for a Social Classroom. In ACM International Conference on Interactive Media Experiences. 249–252. [6] Johnmarshall Reeve. 2013. How students create motivationally supportive learn- ing environments for themselves: The concept of agentic engagement. Journal of educational psychology 105, 3 (2013), 579. [7] Alexander Richard, Colin Lea, Shugao Ma, Jurgen Gall, Fernando De la Torre, and Yaser Sheikh. 2021. Audio-and gaze-driven facial animation of codec avatars. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. IEEE, New Jersey, 41–50. [8] Ellen Skinner, Carrie Furrer, Gwen Marchand, and Thomas Kindermann. 2008. Engagement and disaffection in the classroom: Part of a larger motivational dynamic? Journal of educational psychology 100, 4 (2008), 765. [9] Ellen A Skinner, Thomas A Kindermann, and Carrie J Furrer. 2009. Amotivational perspective on engagement and disaffection: Conceptualization and assessment of children’s behavioral and emotional participation in academic activities in the classroom. Educational and psychological measurement 69, 3 (2009), 493–525. [10] Calkin Suero Montero, Naska Goagoses, Heike Winschiers-Theophilus, Nico- las Pope, Tomi Suovuo, Erkki Rötkönen, and Erkki Sutinen. 2022. Supporting Academic Engagement through Immersive Technologies. Education and New Developments. Vol 2. (2022). [11] Ming-Te Wang and Jessica L. Degol. 2016. School climate: A review of the construct, measurement, and impact on student outcomes. Educational Psychology Review 28, 2 (2016), 315–352. [12] Ming-Te Wang, Jessica L. Degol, Jamie Amemiya, Alyssa Parr, and Jiesi Guo. 2020. Classroom climate and children’s academic and psychological wellbeing: A systematic review and meta-analysis. Developmental Review 57 (2020), 100912. https://doi.org/10.1016/j.dr.2020.100912 https://doi.org/10.1007/s10639-023-11705-9 https://doi.org/10.1016/j.dr.2020.100912 Abstract 1 Introduction 2 Extending realities 3 Project Context 4 Technology Description 5 Demo Set-Up 6 Future Development Acknowledgments References