Despite all the hype in recent years about the potential for virtual reality in education, an emerging technology known as mixed reality has far greater promise in and beyond the classroom.
Unlike experiences in virtual reality, mixed reality interacts with the real world that surrounds us. Digital objects become part of the real world. They’re not just digital overlays, but interact with us and the surrounding environment.
If all that sounds like science fiction, a much-hyped device promises some of those features later this year. The device is by a company called Magic Leap, and it uses a pair of goggles to project what the company calls a “lightfield” in front of the user’s face to make it look like digital elements are part of the real world. The expectation is that Magic Leap will bring digital objects in a much more vivid, dynamic and fluid way compared to other mixed-reality devices such as Microsoft’s Hololens.
What’s in the box?
Even in the sometimes overhyped world of technology, Magic Leap has received an unusual level of gushing speculation. This summer the company announced that it will ship its first Magic Leap One: Creator Edition device by the end of September. The Creator Edition is intended for developers to encourage them to build applications and experiences.
While we don’t know what exactly will be in the box, here is what we do know. Magic Leap One, often described as a pair of futuristic punk glasses, will be tethered to a wearable computer called a Lightpack. The latter provides much of the computational power for the device and can be clipped to your pocket. The glasses will come in two sizes, standard and large, along with nose, temple and forehead pads that are meant to ensure a comfortable fit.
The Magic Leap headset will come with a handheld controller that includes haptic feedback. The device will also include sensors that can detect a user’s movements, and it will support eight gestures that can be used to control motion within mixed reality applications.
Magic Leap One will also have external-facing cameras and microphones that are used to collect information about the environment, speakers that deliver spatialized audio, and inward-facing cameras that can determine where the user is looking. One of the questions Magic Leap has consistently refused to answer is the field of view. In his hands-on review, Rolling Stone author Brian Crecente said, "The viewing space is about the size of a VHS tape held in front of you with your arms half extended."
It should be noted that unlike some VR headsets, the device will not accommodate users who wear glasses. Magic Leap is reportedly working with a partner to eventually offer a prescription lens add-on. The headset is designed for indoor use which suggests that, like Hololens, the device will not handle bright outdoor light.
As with all emerging technologies it is not just the device, but the partnership and ecosystem that will shape our experiences. In Magic Leap’s case, the company has so far announced a series of partnerships, including Disney’s Lucasfilm and a number of game, film and design studios. In early July, they announced that AT&T will be the exclusive wireless distributor for network access and content distribution once the consumer edition of the device is launched.
It’s most likely that the first applications of the device involve games and entertainment, but I would not be surprised if some informal learning applications are included as well. More importantly, the question is what are we going to create and do in mixed reality in the learning environment.
What will mixed reality learning experiences look like?
If mixed reality is to transform education, Magic Leap One is only part of the opening act. It will be exciting to finally test the new device, and its debut will likely push Microsoft and other providers to release their next generation devices as soon as possible. Indeed, we may soon hear more about the rumored Augmented Reality devices from Apple and Google both of which may include mixed reality elements.
To start creating for mixed reality calls for a new set of opportunities and design considerations. Think about the times you learned about the force of gravity and law of physics or the way Renaissance masters used perspective in art and architecture. For some of us this may have been through the pages of the book, but others of us may have taken a trip outside to experiment with gravity or study Renaissance masterpieces in person.
Now think about all the other things you wished you had learned this way and imagine a dynamic digital display that transforms your environment and even your living room or classroom into an immersive learning lab. It is learning within a highly dynamic and visual context infused with spatial audio cues reacting to your gaze, gestures, gait, voice and even your heartbeat, all referenced with your geo-location in the world. Unlike what happens with VR, where our brain is tricked into believing the world and the objects in it are real, MR recognizes and builds a map of your actual environment.
For instance, when you throw a digital tennis ball, it will bounce back depending on how far away you’re standing from the nearest physical wall. If you shrink or enlarge the ball, it will respectively make the ball bounce back on a different trajectory suitable to its new scale. The tennis ball will come back if it hits the physical wall but will continue to roll if it goes through the door.
This will open a whole new world for experimentation across many fields of study. The interplay of sensors, machine learning and visuals, when done right, may feel like magic, as objects react to the users’ actions and respect the physics of the real world.
With Microsoft’s current Hololens, we’ve seen innovative learning experiences in healthcare, engineering, art and other fields. Mixed reality that includes vivid graphics and a broad spectrum of gestural interaction could lead to the creation of more deeply immersive interactive experiences.
The ability to manipulate data in 3D spaces makes it easier and more efficient to detect patterns and gain insights. For instance, think about an urban simulation where students will be able to observe the flow of traffic, weather, people and their impact on cities in a realistic 3D environment. In mixed reality, students may be able to look at urban systems that could be compressed down to a tabletop size. Students could explore individual elements of the scene from multiple angles and then bring the environment closer by expanding the size to virtually step into it at street-level. And with the gesture of a hand, they could plunge deeper into the model, moving directly inside one of the buildings and possibly meeting a guide or peers there.
In mixed reality, virtual characters will be contextually aware, able to sit on the real chair next to you and know when you are talking to them, or to somebody else in the room. That might let users join a musical performance with a virtual symphony and play along. Collaboration comes naturally in shared physical spaces, and I expect mixed reality to offer us compelling new ways to connect and create together.
How do we design for mixed reality?
As with other technologies, we will first draw upon what we already know from making earlier educational games, simulations and videos. They will feel like mini simulations with branching learning paths and narratives.
But the full potential will emerge only as we design a new language for mixed reality. We should not simply rely on existing instructional-design methods but look into bringing user experience designers, artists, creatives and students into the development process. We will need a new visual and gestural language that will work in mixed reality. Natural user interfaces and sensors will make leveraging spatial computing ubiquitous. Sensors will collect data that will provide us with new ways to assess learning. In the future, MR will open portals to immersive stories in your living room.
In mixed reality we will work with fewer pixels than in VR, and digital objects will need to respect the physics and in some cases social aspects of the world. In mixed reality, designing the interactions is as important if not more than designing the visual graphics. Fundamentally, if VR is about transporting you to a magical world, MR is about infusing your world full of magic.
What does the future hold?
There is still a dose of scepticism about what Magic Leap will accomplish with its device. The initial release of Magic Leap One will be a luxury item for many education institutions with little accomodation for people with disabilities or digital equity. Yet in the future, immersive technologies may hold a greater potential to solve these issues compared to our current technology tools.
Further down the line, as Moore’s law drives the technology’s power up and prices down, expect the fidelity and interactivity of these experiences to only get better in the next three years. It is likely that we will see mixed reality glasses at a variety of price points that will eventually replace our phones, screens and desktops.
Mixed reality is about breaking the boundaries between our physical and virtual worlds. It will offer us a canvas to imagine new modes of learning and new ways to collaborate with others beyond the frame of the screen. Ultimately discovering the potential of mixed reality does not depend on any single device, but on the journey we take as humans in understanding how we can learn, connect and enrich our lives.
