Haven't we always lived in mixed reality?
Why I learn what I'm learning
Consider the terms: virtual reality, augmented reality. VR is a fully immersive experience, whereas AR superimposes computer graphics onto the physical space we see around us. Then there’s mixed reality, a term that seemingly implies that we can interact with the augmented reality. In some future, we’ll stop using those terms. Everything is reality. Or, is it?
On my walk today, I kept trying to find a better means of expressing the spatial experiences that will be offered through future display devices.
When I Zoom with a friend, an image of my friend appears on my computer monitor. She is talking, listening, and responding to me. Are we not experiencing reality?
I would think most people will say that Zooming with someone is reality despite the lack of physical presence where we can touch another, or even smell their scent.
More than a decade ago, when I lived in Buenos Aires, I would Skype with my mother who lived in Tennessee. She has since died, yet I still have memories of talking with her.
Those memories enter my mind as thoughts, sometimes at the most inopportune times. I know she’s gone but the memory is real. Or, perhaps I should say, my experience of a thought as a memory is a real experience for me. I know that memories are not accurate. Some memories are of a reality that never occurred but have become, over time, conflated in our minds with other situations. Yet, experiencing those thoughts are our reality. Is that a form of mixed reality? I think so. Mixed reality does not require a digital device. Mixed reality is a play upon the mind.
As we move further into the mid-century, we are creating mechanisms through advances in display technologies that are distorted mirrors projecting scenes onto our field of view. (See Meta’s prototype of Orion eyeglasses.)
With advances in all technology, comes responsibility to those wielding the capacity to create the worlds we experience. Likewise, each person stepping into this new form of mixed reality has responsibility for recognizing falsehood from reality. Educators often talk of digital literacy. The aim of all education should be a literacy of reality.
Connecting to My Life’s Work
I’m not studying artificial intelligence merely to know more. I want to teach more engaging courses on AI. I want people to make the connection between “science” and their lives, specifically with what they view as their life’s work.
In connecting AI to my own life’s work, I need more focus on embodied systems, e.g., avatars that play the role of characters in stories. Or in another context, how AI-enabled avatars serve as teachers, conveyors not only of information but aiding us in developing meaning and understanding of new topics.
The work of Murray Shanahan offers many insights for thinking about LLMs as components embodied (i.e., integrated) within an application. In my courses, I asked the students to write their answer to this yes/no question: is ChatGPT a LLM? In both courses, over 90% of the students said, ‘Yes.’ We then proceeded to talk about how ChatGPT is an application with functionality wrapped around the LLMs created by OpenAI.
Quote to Think About
"Our life is a faint tracing on the surface of mystery." Annie Dillard
A memory
In college, one of the books we read for my Religion class (a required course at Sewanee) was Pilgrim at Tinker Creek. What I remember about that book is nothing from the book itself but a place, a specific reading room in the library, where I sat while reading the book. One day, sitting near me was a guy from the same class. I’ve forgotten his name. I could vaguely describe him. We both sat there reading the same book. We started talking; he surely initiated the conversation since I was too shy. I remember his enthusiasm for the book. Why has the image of that brief conversation stayed with me all these decades? Whenever Annie Dillard’s name comes up, that moment in the library in 1985 is what comes to my mind.
I should reread that book, particularly since I live in a valley by the Blue Ridge Mountains near where Dillard wrote her book. This is a book I should give to my kid.
Helpful thoughts on “seeing” (by Maria Popova) from reading Pilgrim at Tinker Creek.
Why do we remember?
A book I purchased last week and started reading: Why We Remember: Unlocking Memory's Power to Hold on to What Matters by Charan Ranganath, which I learned about from the Finding Mastery podcast.
Reading a book is a passive experience. I love reading. The feeling conveyed through a love of reading, for me, is the feeling of being immersed in some other world. But, even after reading a book I really enjoyed, the finely crafted sentences in that book leave no more than “a faint tracing”.
My vivid memory of The Pilgrim at Tinker Creek was formed through an active association located in a specific physical environment that held meaning for me. How do I create virtual environments and experiences that serve as containers for stimulating the memory?
Why I learn what I learn
The other morning, I did a review of my year, so far, and what I’ve accomplished or, more appropriately, what I did not. There’s the life I want to return to next year: location independent while pursuing my life’s work in creating interactive stories with narrative paths and dialogue enhanced by AI.
This new genre of spatial experiences lack a decent name, though agonizing over what to call a genre is less important than creating works in that medium.
When it comes to my life's work in understanding how we tell and read stories (fiction and factual) in digital media (i.e., via hardware and software) in the mid-21st century, I want to form a lens through which to examine the matter. (Honestly, I just want a short phrase to tell myself every morning as a reminder about why I’m doing what I’m doing.) I want to articulate my lens in a simple way, e.g., if this tech exists, what experiences can be created? By "tech", I mean the AR/VR that is becoming possible over the next decade due to advances in AI, GPUs, computer graphics and related tech.
This lens focuses on the transformative potential of AR/VR, AI, and advanced graphics technologies to push beyond traditional boundaries of storytelling.
The core questions to guide my exploration:
Experience Design: What new types of immersive experiences become possible with the integration of AR/VR and AI-driven storytelling?
Emotional and Cognitive Impact: What emotional and cognitive experiences can be uniquely crafted with this technology that were not possible before?
By using this lens, I want to maintain a focus on the possibilities and implications of these technologies on storytelling and user experience, always asking: "Given this tech, what new kinds of stories can we tell, and how will they change the way we experience and understand narratives?"
I really need a strong sentence that I tell myself everyday and that I can convey to others as to what my life's work is all about.
What new types of immersive experiences become possible with the integration of AR/VR and AI-driven storytelling?
Experience Design:
Spatial Experiences are stories that unfold within immersive, interactive environments, allowing participants to move through and engage with the narrative as if it were a tangible world. These experiences blur the boundaries between the physical and the imagined, transforming how we perceive, explore, and connect with stories by placing us directly within them. They create a dynamic interplay between the environment and the participant, offering a profound and personal journey that reshapes our understanding of narrative and reality.
On my walk, I wanted to toss away the term spatial experiences as the term for defining the genre. Is not everything we do in the world a spatial experience? Perhaps “spatial interactive experience”, which could be called SIX. Catchy, no? But it makes no sense. I’m having a spatial interactive experience as I sit here typing on a keyboard.
I return to mixed reality as an intrinsic part of the human experience.
With these technologies, how can we craft stories that reshape our perception and engagement with the world?"
At the end of the day, while I’m wildly intrigued by the philosophy of these experiences, I’m more excited about the craft of spatial experiences.