XR is the name for all the possible virtual and augmented realities which can be invented, now and in the future. The term gets rid of the messy language about VR and AR. When UK pioneer Robin McNicholas talked XR at AIDC2020, he promised to show us some journeys into remarkable possibilities.
Ironically he spoke just days before the coronavirus changed all our relationships with the real world.
Robin McNicholas is a founder of Marshmallow Laser Feast, a UK collective of fertile imaginations playing with screen, interaction, computers and the human sensorium. They are interested in the creative outlet rather than the technology.
We are all used to the goggles and our separation from the world to enter virtual reality. But McNicholas quotes Presstube to say that ‘VR is the Segway for your face,’ suggesting that the technology will melt into something which integrates the electronic reality with the world and does away with all that bulky technology.
That is really what Marshmallow Laser Feast is inching towards. I am going to tell their story mostly with clips.
The company built itself on brilliant digital eye candy, and created an income from commissions and advertisements. Here is their 2014 showreel.
'What we've learned from doing these insane commercials is firstly to deal with people under really high pressure,' said McNicholas. 'We are dealing with a lot of egos. But also we are trying to choose the commercials to benefit the team internally, as a tool, to connect us to a new entity or introduce us to a new way of thinking.'
Just making XR experiments is a brain-wrenching conceptual landscape of its own as software becomes visual, integrates with twisty cameras, crosses into compositing and bumps into the human sensorium. They engage with the enormous potential of interactivity, a term which is bleedingly obvious in games but a high art passion and a trap if creators want to engage with mainstream storytelling.
Here are two experiments, now part of Marshmallow history and their evolving take on a rarified technical ferment.
This is essentially a music piece which is telling the images what to do.
Meet Your Creator is a light show in which the creator controls the shafts of light in a dance, or a story.
This project for Saatchi and Saatchi was a lot of fun but essentially traditional. It shows us the possibilities of large scale quadcopter spectacles, which have now become town-sized spectacles, like fireworks crossed with Christmas lights.
Threaded through all this is the way that the relationship between performer and technology is cutting loose. In those U2 scenes, for instance, the band movements are controlling the images. I move, it responds, we dance together.
Somewhere in here there was an explosion in the hivemind of the collective, maybe in quiet slow motion. Perhaps it started in Grizedale Forest in 2015. The collective was challenged to produce a VR work that could be viewed in amongst the trees. They decided to create a way of seeing the ecology through the eyes of different creatures: a mosquito, a dragonfly, a frog and an owl. A mossie, for instance, experiences (or sees) carbon dioxide which it uses to find warm-blooded creatures in order to suck their blood as placental fluid.
So the environment was recorded using different technologies like Lidar, and turned into consistent experiences. The whole system had to be run off a backpack which also carried haptic devices so the user could feel elements like beating wings along their spines.
Here is the result:
There are videos about how this is built at iteota.com.
This raises the whole question of sensorium. We are using computation and external data to create the experience of looking and hearing at a particular time and place, so that the story is unique to the user. I stand somewhere different and move differently and see a unique trajectory.
They have also built ways in which a normal audience can watch outside the singular space of a headset, so audiences can see a performer like a band or a dancer move and sound in a way that controls the visual experience. They make the world up in the real world as they go along, like a shared game.
What happens when Marshmallow delves deeply into the nature of perception, which is built from signals going into the body, connecting to stored brain material which assembles the world we see? Our reality is all repurposed data in a myriad of different ways, and the transformations affect each other in our heads.
For instance, our visual sense of colour is directly affected by sound. This enquiry led to the Canadian/British collaboration, A Colossal Wave in which the experience inside the headset is affected by outside activity which varies from singing to dropping a huge ball or the movement of the subject.
Here is the trailer.
The little creatures are very sweet.
Now the collective has built We Live in an Ocean of Air, described thus:
'We Live in an Ocean of Air is a multi-sensory immersive installation illuminating the fundamental connection between animal and plant. Step through the canvas and share a breath with the giants of the plant kingdom...
'Using a unique combination of technologies from untethered virtual reality, heart rate monitors and breath sensors to body tracking, visitors will be completely immersed in a world beyond human perception.
'Follow the journey of your own breath from body, to plant, to planet in an intricate 3D world, where the deeper systems and connections that intimately tie together all life on Earth are made visible.'
There is plenty of other fascinating work in XR. Forbes has published
its 50 favourite projects for 2019, though they tend to be more literal and less daring than Marshmallow Laser Feast. CNET describes
the Sundance New Frontiers program for 2019 to create some context while The Verge picks
the best which includes a recent Marshmallow Laser Feast project. It seems oddly low key for such an ambitious outfit.