Carmakers Have Another Screen in Their Sights: Your Windshield

Next-gen augmented reality is about to get weird.

Augmented reality in vehicles is nothing new. In fact, it's currently installed in millions of vehicles equipped with Head Up Display (HUD). But major car and tech companies think projection technology is about to become the next big thing as it expands beyond the driver's cockpit. After all, what else are you supposed to look at, stuck sitting in your new electric car while it gasses up at a charging station?

BMW, which has championed HUD as safety technology in vehicles for the past two decades, demonstrated at Consumer Electronics Show 2023 a "mixed-reality slider" that will extend augmented reality across the bottom length of the windscreen. In addition to taking up more visual real estate, the next-generation technology aims to "blend the real and virtual worlds." It'd potentially culminate in a full-screen display with the capability of turning the entire windshield into a big screen TV.

Chairman Oliver Zipse said (during the weirdest media event that no one is talking about) that the technology isn't just a concept–it's headed to series production in two years and will appear in the upcoming Neue Klasse in 2025. But just how far the German manufacturers can go with increasingly immersive AR effects in the cabin may ultimately be up to the United Nations.

The big picture

The BMW Vision DEE, which stands for Digital Emotional Experience, is an ambitious follow up to the iX Flow that debuted the previous year at CES. Rather than be painted in monochrome, the Dee is wrapped with hundreds of color-changing E Ink panels. It also features a Zuckerberg-esque avatar that greets you when you approach the vehicle, and uses "shy tech" controls woven into the fabric, hidden behind the seamless dash that illuminated and revealed the mixed-reality slider buttons when a finger passes over it. Suffice to say, there are a lot of details that need to be worked out before it makes it to the dealership, but full-length augmented reality is a sure thing in the next generation of BMWs, according to the company.

The CES presentation laid out four levels of mixed-reality AR that, despite the confusing naming convention, are not tied to the self-driving levels established by the Society of Automotive Engineers. The data demonstrated in BMW's Level 1 of the mixed-reality slider is pretty much what's already found in the instrument panel, but rather than toggling through different screens to see the speed, audio, vehicle settings and other details about the surroundings, the information is strung along across the bottom length of the windshield.

Selecting Level 2 adds vehicle telemetry and advanced driver assistance system functions, such as obstacle avoidance and navigation, which is already part of most HUDs. "Brand communication," and social media "visualization" are introduced with Level 3 of the mixed-reality slider. Level 4 is where things begin to get weird.

In this level, AR overtakes the windshield and goes "way beyond reality," BMW's CES presenter, Alice, explains during the keynote event, and hints at the vehicle transforming into a metaverse filled with friends, "the entire virtual world right at your fingertips everything you can imagine right here right now."

Details are nebulous–and it's not clear who, if anyone, is doing the driving at this point–but in the cabin becomes "an endless virtual world you can meet, play, talk, love, hate–you can even go sightseeing together right inside your car," according to BMW. In short, as the driver selects higher levels of AR, the mix of virtual overtakes the physical, and the windshield can become a digital environment that blocks out the harshness of the real world. And these levels of increasing windshield takeover are technically possible, but would it be allowed?

Pushing boundaries

At a separate event held adjacent to CES 2023, Harman Automotive demonstrated its next generation Ready Vision technology, which is a platform that handles data from multiple sensors, such as navigation, ADAS and microphone, and casts those graphics onto a more traditional layout on the windshield of up to 15-degree by five-degree field of view in front of the driver. Powering the AR visuals are artificial intelligence and machine learning systems that turn the forward view outside the windshield into a mapped canvas with directional arrows, contextual points of interest, street names, and traffic data overlaid.

But there was no full-windshield display. There would be no social media, no avatars, or videos cast on the window–even if the vehicle was parked. In fact, Rhita Boufelliga, senior director of Harman's digital cockpit product, explained that there are regulations that hadn't yet gone into effect, but would eventually limit the type of information that the company could display in front of the driver above a certain speed. For example, while turn-by-turn directions were allowed, Point of Interest information likely wouldn't be shown because it would be considered distracting.

But that was news to several policy experts, who were not aware of any specific rules governing HUDs.

"NHTSA has issued some visual-manual driver distraction guidelines that might apply to some of the augmented reality stuff as it pertains to the driver, but those are essentially recommendations with no force of law," explains Michael Brooks, Executive Director for the Center for Auto Safety. Experts at the Insurance Institute for Highway Safety also could not find anything pertaining to the technology.

NHTSA itself, when reached for comment, clarified that "[c]urrently there are no direct regulations on heads-up displays, however, the agency has released research and information regarding the subject: 1. Equipment Design Features (Impacts on Safety) and 2. Human Factors Design Guidance For Driver-Vehicle Interfaces." NHTSA does have guidelines for what "interferes with driver attention" and officially counts as distracting, the agency also noted.

While it would seem logical that there should be on-the-books regulations surrounding something that could be so visually distracting, there aren't any. That's because in the U.S., it's up to the automotive manufacturers to self-certify that their vehicles comply with the federal vehicle safety standards. However, Harman's demonstrator was likely referring to rules still in development by the United Nations Economic Commission for Europe and its World Forum (WF) for the Harmonization of Vehicle Regulations, or WP 29, which is part of the Inland Transport Commission.

Manufacturers that want to sell vehicles in the European Union and some other parts of the world have to get their vehicles or equipment type approved by these regulating bodies before it can be legally sold, explains law professor Bryant Walker Smith, who specializes in autonomous vehicles.

"In the U.S. we have self-certification that's developed a limited number of federal motor vehicle safety standards, and a manufacturer has to self-certify that it promises that their vehicles and equipment comply with those limited standards," says Walker Smith. "And if there's not a standard, they don't have to promise that it complies because there's nothing to comply with."

The National Highway for Traffic Safety Administration's distracted driving guidelines do not govern augmented reality technology, which was still considered novel when it was published in 2013. However, car makers also have to comply with state motor vehicle laws or contend with potentially violating other established regulations that might be affected by new technology, such as lighting standards.

"You can put a new technology in a vehicle not because of a law that says you can, but because there's no law that says you can't. There's also always uncertainty about how existing law awkwardly applies to these new things that that law did not contemplate," says Walker Smith.

That's a strategy that some automotive newcomers may attempt more frequently than conservative legacy carmakers. Tesla was the first car company to release video gaming integration using its embedded display late in 2021. Although the feature was initially marketed as a way to entertain vehicle occupants while parked at charging stations, there was nothing that stopped someone from using it while driving. It only took a couple of weeks before NHTSA began its investigatory process into the feature. Shortly after, Tesla released an over-the-air update that restricted the ability to play video games while the vehicle was in motion.

BMW is unlikely to make a similar mistake. Although the German luxury brand plans to introduce its mixed-reality slider first in a sedan and sport activity vehicle in the upcoming Neue Klasse, it's not likely to have the full-height capability demonstrated at CES 2023. At least, not yet.

Reclaiming the third space

As more owners switch from gasoline-powered cars to electric vehicles and Level 3 and 4 self-driving systems eventually take the wheel for them, drivers are going to have a lot more free time on their hands. Several automotive manufacturers signalled their intentions to fill that newfound freedom with concepts revealed at the CES 2023 that showcased innovative entertainment possibilities.

Mercedes-Benz Chief Technology Officer Markus Schäfer announced at the show that it was the first manufacturer to receive approval from the Nevada Department of Transportation for its conditionally autonomous Level 3 Drive Pilot system. The news made the perfect segue to introduce its partnership with Zync, a multi-sensory entertainment system. This won't be on a windshield but rather on its full-dash hyperscreen, which supports streaming video and gaming for the entire cabin.

Shortly after the close of CES 2023, Ferrari announced it will be leveraging Harman Automotive's Ready Upgrade platform, which could be used to develop and offer post-sale performance tuning improvements or other products that enhance the "digital cockpit." While these new products and technologies won't be available for a few years because of the long automotive development lifecycle, Polestar and Hyundai announced that Nvidia's made its GoForce cloud-based gaming platform accessible in some vehicles already on the roads.

And if there is a manufacturer best positioned to capitalize on the expansion of the original "third space" between home and work, Sony-Honda Mobility is it. The newly formed joint-venture between the electronics company and mobility company positions its Afeela electric mid-size sedan prototype as an entertainment system on wheels.

"The Sony Bravia Core, which is in the Afeela, is a streaming platform that allows us to bring Sony Pictures, both TV and movies, directly to our Sony customers using the Bravia core app," says Larry Harrison, product marketing manager for Sony-Honda Mobility. That means that not long after a blockbuster movie, such as Spiderman or the upcoming Gran Turismo is released in theaters, you’ll be able to stream it in the cabin in just a few clicks, he explains. Again, the Afeela program made no mention specifically of windshield projection, but it does represent a major step forward for immersive in-car entertainment.

BMW's Vision Dee, with its full-screen Level 4 augmented reality technology, seems primed for a similar in-cabin theater or metaverse experience in which to pass time if the vehicle is self-driving. However, the company says it's not tying the two technologies together.

"Level [4] on the mixed reality slider is meant to create an immersive virtual reality environment that may make a tedious drive more interesting," says BMW corporate communications manager Phil Diianni in an email. "In showcasing the BMW i Vision Dee, we were not showcasing any autonomous functionality. The BMW i Vision Dee is meant to demonstrate the future of digital interactions, user interface and experience."

But that doesn't mean it can't happen. After all, CES is a global conference, and laws governing self-driving vehicles are being developed faster in the E.U. than domestically. This means that it's possible these types of sensory-rich in-cabin video systems may launch first in E.U., where the company is headquartered, before being released in the U.S.

As self-driving technology takes more control of the vehicle and frees the driver's time and attention, screens are going to become a larger focus of in-vehicle technology, and so is the content that fills them. But what comes next may depend on how quickly manufacturers, regulators and the public embrace self-driving vehicles. However, when they do, the possibilities created by technology and innovation are limitless.

NASCAR Racer Looks Absurd, Goes Fast at Le Mans

The Rough Weekend of Indy's Fastest Woman

The Citroën Rally Tech So Advanced McLaren Uses It

Deconstructing the Engine That Dominated Le Mans

Move Heavy Batteries With Ease With This Tool

This 740-HP Toyota GR Yaris Is Incredible

Newgarden Wins Indianapolis 500 With Last-Lap Pass

How Modern TC and ESC Makes You Faster

Toyota Cannot Screw Up the Tacoma

The First Running of Le Mans Wasn't Even a Race.

BMW Might Build 50 New Z4 Clownshoes at $250,000

NASCAR's Most Iconic Ghost Track Is Back