Welcome to Lowpass, a newsletter about AR, VR, streaming and more. This week: A new Roku patent hints at plans to take ads beyond its own devices, and web-based VR lands on the Vision Pro.

This week’s Lowpass newsletter is free for all subscribers; next week’s lead story will only go out to paying members. Upgrade now to not miss it.

Your Roku TV may one day show ads when you pause video on your Apple TV

Roku is exploring ways to show consumers ads on its TVs even when they are not using its streaming platform: The company has been looking into injecting ads into the video feeds of third-party devices connected to its TVs, according to a recent patent filing. 

This way, when an owner of a Roku TV takes a short break from playing a game on their Xbox, or streaming something on an Apple TV device connected to the TV set, Roku would use that break to show ads. Roku engineers have even explored ways to figure out what the consumer is doing with their TV-connected device in order to display relevant advertising.

Taking a break can be lucrative for Roku. The streaming device and smart TV company figured out some time ago that those times when consumers don’t do anything represent an advertising opportunity. One of the ways Roku monetizes inactivity is its somewhat iconic Roku City screensaver.

• Roku began selling sponsorships for its screensaver last May; some of the companies that have paid to be featured in Roku City include Walmart and McDonalds.

• One indicator for how much Roku values these moments of inactivity is that the company doesn’t allow developers to integrate their own screensavers into their apps running on Roku devices.

• Developers are, however, able to build dedicated screensavers, which consumers can then download to replace the default screensaver.

HDMI is Roku’s blind spot. Roku’s ability to monetize moments when the TV is on but not actively being used goes away when consumers switch to an external device, be it a game console or an attached streaming adapter from a competing manufacturer. Effectively, HDMI inputs have been a bit of a black box for Roku.

The patent application aims to solve that by keeping tabs on the video signal that is being delivered over those HDMI ports. And it’s clear that Roku would have to keep very close attention to those video signals. Randomly interrupting a video game a consumer is playing on their Xbox would almost guarantee that no gamer would ever buy a Roku TV again.

• The patent application proposes a number of ways to identify actual pauses, including grabbing frames from a video feed for comparison’s sake. If nothing changes between a number of frames, a viewer has likely pressed pause.

• To make sure that it’s not just a freeze frame, Roku’s engineers propose to also monitor the audio feed for extended moments of silence.

• Depending on a consumer’s device configuration, a Roku TV may also be able to use HDMI CEC – a protocol that’s meant to help TV-connected devices work better together.

How to make that ad relevant. I imagine it could be pretty jarring to play a game on your Xbox, take a quick break, and find your TV playing ads for toilet paper. But what if the ad was somehow relevant to your gameplay? The patent application details ways to achieve just that for a variety of connected devices.

• One of those approaches is pretty straightforward: If a consumer watches something on their Apple TV and then presses the pause button, a Roku TV set could use either audio or video-based content recognition technologies (known in the industry as ACR) to identify what’s being watched, match the current scene to a database and extract relevant information to pair an ad with it.

• From the patent application: “The fingerprint and/or watermark may correspond to a certain movie title, famous actor, and movie genre. The metadata may correspond to a champagne bottle and mountain scenery. The relevant ads may include the famous actor, a type of champagne, vacation opportunities that include the mountain scenery recognized.”

• But what if someone plays a game, and there’s no database to match it to? Even knowing that a consumer is a gamer, and is using a certain game console, may be valuable to Roku.

• The patent application envisions using the type of control signals game consoles send to TVs to invoke low latency mode to differentiate gameplay from video viewing, and then for instance recommend different games, or even a newer game console.

There’s no guarantee Roku will ever do any of this. Companies patent things all the time that don’t make it into actual products; a Roku spokesperson declined to comment when contacted for this story.

However, the fact that Roku even explored this points to a major underlying issue: These days, TV makers hardly make any money with their physical products. Roku’s FY 2023 earnings report shows that the company lost $44 million on the sale of smart TVs, streaming players and other devices in 2023. What brings in the bacon are ads and services; Roku generated a gross profit of nearly $1.6 billion with this business segment.

Now, if someone goes and buys a Roku TV primarily to play on their Xbox, or watch something on their Apple TV, they’re effectively a lost cause for Roku – unless the company monetizes the use of such third-party devices as well, that is.

Web-based VR lands on the Vision Pro, with some caveats

Immersive app and game developers can now build web-based VR experiences for Apple’s Vision Pro headset, thanks to Apple supporting the WebXR standard on the device. This makes it possible to build web-based apps that run on a variety of devices ranging from mobile phones to Meta’s Quest VR headsets to the Vision Pro, as Niantic’s AR platforms product manager Tom Emrich pointed out in a press release this week.

“With a category of MR headsets now available for consumers, cross-device content creation has become critical for developers who want to engage as many users as possible,” Emrich was quoted in the release, adding that web-based AR and VR helped developers to “build once and deploy everywhere, including computers, smartphones, and headsets such as Meta Quest and now Apple Vision Pro."

Niantic’s 8th Wall unit issued the release to announce that the company’s metaverse deployment tools now extend to the Vision Pro, but the release also highlighted a notable shortcoming of Apple’s take on WebXR: Web-based apps running in Safari on the Vision Pro do not have access to the device’s camera feed or mapped spatial data. Without it, it’s impossible to build any web-based augmented reality apps for the device. 

(Another shortcoming that is likely going away over time: At least for now, end users still have to enable WebXR support in the browser’s settings.)

What developers can do is build fully immersive VR apps that end users can launch without downloading anything from the App Store. This focus on VR is a bit ironic, given that Apple has gone to great lengths branding the Vision Pro as anything but a VR headset, but the ability to launch VR apps this way should not be underestimated. 

While web-based VR may have initially been seen as a path primarily for lightweight apps or experiments, developers have long proven that web browsers can be used to deliver impressive immersion and complex gameplay. Case in point: The developers of Moon Rider managed to build a full-featured Beat Saber clone that runs entirely in the browser. Niantic, meanwhile, developed a basketball demo that’s now available in VR on the Vision Pro.

Apple did not immediately respond to a request for comment for this story.

What else

Interview with Paramount Streaming CEO Tom Ryan. Ten years ago, Ryan headed the team that launched Pluto. This week, he told Variety about Pluto’s path to 80 million monthly viewers.

Apple Vision Pro + Entertainment report. Speaking of Variety: I did a report about the Vision Pro and immersive entertainment for Variety VIP+.

Fast Travel Games sold more than one million VR games. The Swedish studio has released 12 VR titles to date.

YouTube TV could be bigger than Comcast and Charter by 2026. That’s according to MoffettNathanson analyst Michael Nathanson, who thinks that YouTube’s TV service could reach 12.4 million subscribers by then.

Apple told Jon Stewart not to invite Lina Khan. The company also discouraged bits about AI, according to Stewart.

A first look at Europe’s alternative iPhone app stores. This is pretty interesting, considering that Epic’s app store and others are going to be game- or entertainment-focused.

Spotify has a new CFO. Christian Luiga previously worked as an executive at a European defense and security company.

That’s it

Inherent Vice: I still vaguely remember seeing a paper copy of Vice magazine for the very first time in the 90s. If my memory doesn’t deceive me, the mag had somehow found its way across the Atlantic, and into a German coffee shop, where I was a bit dumbfounded about a story in which two writers described their quest of spending a big chunk of money ($10,000 perhaps?) in the shortest amount of time in an Asian metropole (Tokyo perhaps?), buying a bunch of crap and gloating about it.

Little did I know that setting money on fire was apparently also the M.O. of Vice, the online publishing empire that this weird little mag eventually evolved into. Granted, anyone who has seen Shane Smith’s antics over the years probably had a hunch that something was off. But reading Elizabeth Lopatto’s excellent investigation into Vice’s downfall was nonetheless an eye opener. Do take some time this weekend to give it a read, I promise you’ll be as dumbfounded as I was back in the 90s reading Vice for the very first time.

Thanks for reading, have a great weekend!