Will latency doom smartphone-based virtual reality?

Advertisement

Job got you down? Has your love life lost its spark? Are you sick and tired of real reality? Lucky for you, all of that can change thanks to the modern miracle of virtual reality!

Okay, that sounds a little cheesy, right? Okay fine, it’s a lot cheesy. Virtual reality, however, is much less so and has the potential to revolutionize, well, everything! We’ve been trying to put virtual reality to use in more and more applications over the years, and now it’s poised to go beyond expensive simulators and make its bold entrance into our living rooms, and even our pockets!
In the past, virtual reality has been limited to specific applications: aircraft simulators, driving simulators, golf simulators, combat simulators, and (more recently) specialized medical equipment.

In each scenario the user is placed into an immersive environment, surrounded by scenery, sounds, lighting, and a very limited number of objects with which they can interact.

SIM City, EA Games

Watch it, buddy! I’m a Mayor in SIM City!

These simulators, unlike modern video games, are typically powered by many computers — sometimes dozens of them — each responsible for one area of the simulation. Video games are usually powered by just one computer, but they’re only responsible for showing you what’s on your screen. Anything beyond the display is basically ignored and requires almost no processing time. But virtual reality is different.

Unlike immersive games, VR has to be ready for everything that’s around you. When playing a game (whether on a console, computer, or mobile device), turning is generally done with your character’s entire body, bringing a new section of the map into view. In real life, however, we glance around a lot. We take cues from our peripheral vision. We need access to the “map” around us in ways that current games just can’t provide.

Let’s say you’re playing a game, but it’s very slow and laggy. One of the “tricks” to speed it up it is by lowering the resolution. Doing so cuts down on the number of pixels the graphics engine has to paint, and speeds up your gameplay. Of course that means it’s not going to look as good, but the goal is to find the right balance between the two. When we start talking about immersive configurations we suddenly jump up in the number of pixels — a lot! Instead of a simple 12-inch by 18-inch flat screen, we now have to consider a 360-degree panorama, horizontally and vertically.

Simulators

Airbus A320 flight simulator

It’s a bird! It’s a plane! It’s an Airbus A320 flight simulator!

Back to dedicated simulators.

These situations can’t afford not have the periphery already painted and displayed. That’s why they have so many separate computers powering them. You just need to move your head or flick your eyes to see the data on one of the other screens — screens that are stitched together so they look like one continuous display (well, almost).

VR does away with all those other screens, but the requirement for the data on those screens to be ready for you to “glance” at still remains. How fast are those glances, anyway?

Mobile devices are somewhere around 60fps. Movies are generally 24fps, but newer shows (like The Hobbit) were also released at 48fps. According to many so-called experts, the human eye can still “see” 60fps, so most computer monitors are around 75-90fps, though some are much faster.

The more frames displayed per second, the more data the GPU has to to process. Jumping from 24 to 48 (or 30 to 60) fps requires a doubling of processing power, and 60fps still isn’t “fast enough” to look “real”.

The real lag

Geez, Napoleon. Can't you peddle any faster?

Geez, Napoleon. Can’t you peddle any faster?

As if all that weren’t bad enough, now there’s an additional component to take into consideration: sensors.

Various studies have shown that we can detect a latency when it’s greater than 50 milliseconds. As soon as that threshold is surpassed and the user becomes aware of the latency (even on an unconscious level), the illusion of the artificial environment is lost.

We’ve gotten used to all sorts of gizmos and sensors in our smart devices. Some measure the ambient light, others measure elevation, rotation, and even magnetic fields. Inputs from each of these must be read, translated into usable data, then acted upon by the operating system. From there, the source or intensity of a sound may need to be adjusted and the corresponding signal sent to a set of virtual-surround speakers. The picture must pan or tilt, shadows must be cast, lighting adjusting, lense flare applied, and particles mapped. All of which must happen in an instant. In reality, it takes much, much longer.

The prototypes of the Oculus Rift used an off-the-shelf sensor. The company quickly realized it needed something better, something faster. Ultimately it developed its own sensor, the Oculus VR sensor. This marvel of engineering supports sampling rates “up to 1000hz” — that’s a thousand measurements every second. This reduces the time between a user’s head moving and the virtual reality environment being able to begin reacting down to roughly 2 milliseconds. That’s a huge improvement, but it’s only the sensing and detecting component. From there, the OS and app must apply logic, the CPU and GPU have to get to work, and that data must be translated and transmitted to the screens and speakers in the virtual reality headset.

Each step along the way stacks on additional time, until the magic sub-50ms bar is frighteningly close. Add to that data lag, and it’s game over. The illusion is lost, and instead of a believable world, now you’re just a lonely person wearing a ridiculous pair of glasses.

Smartphones

Maybe it's just the Matrix resetting itself.

Maybe it’s just the Matrix resetting itself.

Let’s bring this full-circle. Current virtual reality systems achieve their magic by using near-real-time operating systems and very customized hardware.

Regardless of what OS is inside your smartphone, or what SoC is powering it, you simply don’t have anywhere close to “real-time” processing.

When all is said and done, it looks like latency will most certainly doom smartphone-based virtual reality.

Advertisement

What's your reaction?
Love It
0%
Like It
0%
Want It
0%
Had It
0%
Hated It
0%
About The Author
Joe Levi
Joe graduated from Weber State University with two degrees in Information Systems and Technologies. He has carried mobile devices with him for more than a decade, including Apple's Newton, Microsoft's Handheld and Palm Sized PCs, and is Pocketnow's "Android Guy".By day you'll find Joe coding web pages, tweaking for SEO, and leveraging social media to spread the word. By night you'll probably find him writing technology and "prepping" articles, as well as shooting video.Read more about Joe Levi here.