The VR Perception Hurdle

I love seeing the advances in Virtual Reality, Augmented Reality and Mixed Reality. I believe as the world continues some version of its distributed workforce, these tools and the content creation skill sets required will become valuable ways that AV integration companies can continue to help their clients.

That being said, all of these Extended Realities have some hurdles to overcome still. Augmented Reality (AR) is the closest to becoming mainstream — as it typically just provides a digital overlay to a physical object, whether a visual asset or added data. Mixed Reality (MR) has a little further to go, as the digital assets have to live in the real world, which means they are more than an overlay, but an interplay between the digital and physical. The AI has to know where real-world objects are in relation to the digital ones, allowing them to interact with them and layer them in a spatial relationship. Virtual Reality (VR), in principle, is the easiest — as it replaces the world and doesn’t require interplay with it. However, motion sickness, lack of natural interfaces and limited haptic feedback still plague even some of the best VR experiences.

Even if the issues behind motion and control are fixed sufficiently, the question remains, “Will VR ever be perceived as a ‘real’ experience by the brain, or will we always know somewhere in the back of our minds that we’re immersed in a digital simulation?”

VR

Resolutions are increasing, color gamut is widening and contrast is deepening to make today’s displays potentially more realistic than ever, but there still may be a problem. What is that? Our brains are extremely adept at pattern recognition, which means that pixel structure is a problem.

To reinforce this point, take a look at what engineers did when digital started to replace film. They added noise. Why? because digital images suffer from color banding and expose pixel structure. Film is an organic medium with grain, but the grain is randomized meaning the position of the grains is not fixed frame after frame. Pixel structure never moves, and for that reason, it is perceived, even if subconsciously, by the brain, telling us what we’re looking at isn’t real.

Think of the way we perceive objects in the world. White light illuminates everything and objects absorb some of that spectrum of light and reflect others — giving them their color. Most light we see is reflected. There are few places in nature where we see emissive light, the sun and stars in the heavens, and fire, lightning, and bioluminescence in nature. If you think about primitive interpretations of these things, they were supernatural and magical. They were the source of superstition and wonder. In short, they stood out as “not of the natural world” until science sufficiently came about to explain the phenomena and start to recreate it through technology.

I contend that somewhere in our minds, emissive light still elicits that reaction. It’s “manufactured.”

Even technology that uses reflected light is different than nature as it is used to color objects, not to illuminate them to expose their natural hue. Projection technologies like DLP can reduce picture structure through increased aperture, due to the light reflecting off of rather than going through the optical engine. The issue with projection becomes the silhouetting, blending and contrast issues associated with it, making it good for VR only in highly contained and well-designed environments.

The emissive nature of our modern technologies, along with the inherent pixel structure of our 1080p, 2160p, and 4320p grids may, in the end, be too much to overcome.

This is why things like light field technology using silicon photonics were being explored by companies like Magic Leap to try to create a more natural interplay of physical and digital technologies and leverage the way eyes and brains interpret light.

I’m really excited about the future of our display technologies and all the possibilities extended realities provide us. In the end, I don’t think we’re at the point technologically to create The Matrix, and we may never be until we hack the optic nerve directly, but I don’t believe that it’s absolutely necessary to extract the immense value extended realities present.