The new Apple Vision platform and 2023 Pre-InfoComm thoughts

I typically try to write on more evergreen topics than thoughts about a convention that will be in everyone’s rear view in a couple days. But my thoughts going into InfoComm this year are about topics that could potentially age well over the next few years. Let’s dive in.

Apple’s new spatial computing platform

What does this mean for my niche of the AV industry? Isn’t Apples new Vision headset just an overpriced, consumer focused media consumption device? Well… No it’s not. It’s a full featured M2 based computer with a radically new interface paradigm. I’m not going to be as bold as to say that Apple has solved VR headsets and we will all chuck our MacBooks or iPads for a $3,500 Apple Ski Mask computer. But Apple has taken away a slew of negatives that Meta hasn’t even considered relating to VR and AR Headsets. Apple has put massive R & D into how humans can easily interact the software inside the headset while still accessing the real world around them. And equally into how others that are not wearing the headset will interact with those that are. Anyone who says this isn’t a game changing leap toward wider adoption and acceptance of headset based computing isn’t paying attention in my opinion. Once again Apple moved everyone’s cheese to a place that most didn’t think anyone would want to go. And just like their over-priced phone, tablet, and watch, you will start to interact with others who will rave about the experience until you slink into an Apple store to try it out for yourself. You might be won over. Time will tell. But Apple has a pretty solid track record of innovation that changes the way that humans interface with technology. Now what’s the tie into the nations premier convention for installed and live audio-visual? Let’s explore this.

Spatial Audio

Apple calls this spatial computing for a reason. The Vision headset isn’t just an alternative display. It also includes spatial audio. Your stereo music or surround designed movies will not only image much closer to the designer’s intent. But the headset is touted to analyze your environment with LiDAR and model the overall audio experience as if you are listening to the media in the room you are in. Or, perhaps a space you would rather be in. This experience will highlight the differences between mono, stereo, surround, and a fully immersive spatial audio experience. This will be brand new to many people outside of the audiophile community. This will cause ripples up and down through the production chain as content creators who are not currently taking advantage will be pushed into new tools to create these experiences. And it will increase awareness and potentially demand in the Installed and live AV industry to try and replicate this experience in a group setting. I predict this will fall flat in most temporary implementations. But I’ve seen so many crazy requests over the years for something new or “fresh” when it comes to corporate meetings. These requests often flow from senior management who certainly have the income to afford this new Apple experience for themselves or at least for their kids. And these demands flow down through creative teams who are over weighted toward apple ecosystem users. These are Apple’s early adopter demographics and they are often our specifying client base. So don’t think that some client in the next year or two won’t ask if we can incorporate some sort of immersive audio element to their meeting. Just because of exposure to this tech. That’s just the media consumption component. I haven’t even touched on the imaging within the headset when you do a multi-way facetime call. I can certainly see this concept being requested in higher end teleconference rooms.

Aspect ratio independence

My role as a video switcher operator in the corporate meeting niche has increasingly become more and more aspect ratio agnostic. The widespread adoption of LED walls is the leading reason. Having nonstandard aspect LED displays has also influenced projection based design into nonstandard ratios and mapping. It has also pushed much more adoption of portrait monitor implementations for digital signage. Stepping back to the Apple Vision, the interface is fully immersive with virtual floating display panels that are optimized to the content being displayed. As a strong proponent of standards, this hurts my heart. But it makes too much sense in this new spatial environment. You don’t want to waste any real estate when you can see the world all around and sometimes through the data you are referencing or content you are manipulating. This will further push aspect ratio agnosticism in content creation. I see display manufacturers being pushed to create more and more fit for bespoke purpose displays. Strips for ticker type information, portrait, oval, or square aspects for video conference, superwide for gaming and film consumption, etc. This will ripple through all of AV. While I don’t see the 16:9 ratio going away or even losing it’s dominance all together, I think we should expect to see more aspect ratio flexibility in cameras, switching, signal distribution, and display device hardware.

Resolution Agnosticism

Once again, Apple has led the pack in electronics interfacing with humans. Apple quit chasing the high resolution race of all the other manufacturers several years ago. I believe that they realized it’s a wasteful pursuit with diminishing financial returns past a certain point. Hence, the retina display.

For decades, display technology has been incapable of creating an image that exceeds our eye’s capacity to resolve individual pixels. A few years back, Apple recognized that their phone displays had reached the point where human eyes could no longer resolve any higher resolution. So they wisely did two things. First, they coined the term “Retina Display” as a huge selling point of their new phone models. The second thing is they probably diverted further R&D away from creating higher and higher resolution displays. And into things that mattered. This pulling out of the resolution race has kept their graphics fluid and accelerated other innovations in user interface like the Vision headset. While Samsung is trying to make pixels the size of molecules to out spec their phone competition, Apple is focused on improving the experience in other areas. What Vision points out is that resolution past a certain point doesn’t matter. Once you achieve retina pixel density, stop. Stop and focus on other things.

What does this mean at InfoComm? It means we as AV professionals need to stop geeking out over higher and higher resolution display technology and focus on other things. Due to COVID and a slew of work since, I missed the last few InfoComms. I distinctly remember the last one I attended though. The booth everyone was talking about was the Sony LED wall booth. Right near one of the main entrances sat a massive ultra tight pitch LED wall. I saw so many attendees with their faces right up to it trying to see the pixels. It was crisp. However, that wall was delicate and extremely difficult to install, limiting it to permanent installation only. It required very specialized content and playback capabilities to source content that took advantage of all of those pixels. All that I saw when I looked at it was a one off publicity stunt with very little application in the live event space I work daily in.

So what has happened in these following years? Obviously COVID, supply chain and the chip crisis put a damper on development of some technologies. But when I survey the professional AV gear landscape pre InfoComm this year, I see an environment that mimics more the napster impact on the music industry versus the development of a path towards supporting that beautiful Sony display from three years ago. COVID brought virtual and that meant video over network infrastructure. Not so much pristine 4K video. But the equivalent of trashy MP3 rip files from the napster days. If the frame rate approached 15 fps and the picture looked somewhat legible, that was a win. That’s what I anticipate every video manufacturer to be chasing this week on the exhibit floor. Shoving as much video through a network switch as they can. While this is going to eventually reap benefits as public networks get faster, most common internet can barely play a highly compressed 4K stream or two. And 4K+ resolutions are a crap shoot to get to pass through copper wiring over six feet a lot of the time. So where are we at on super ultra high resolution?

My unabashed favorite switcher manufacturer For-A is tipping their toe into 8K, Like most, it starts with the low hanging fruit of signal routers and a signal processor box. But they recognize 4K still exceeds many if not most of their market’s requirements. They have a great and complete 4K product ecosystem. I believe the highlight of their booth this year will be the integration of NDI and NDI/HX into their mid market HD switcher line. They recognize that there is money to be made in the HD resolution space. The demand in the HoW space is almost completely HD. Small corporate, educational, Institutional, and many other markets are quite content with HD. It’s about the content and being able to move signals reliably and efficiently over copper, fiber, and now the network.

Blackmagic has a history of pushing the resolution limits. Often beyond ratified industry standards. But there 8K equipment hasn’t found it’s legs while it’s HD products are flying off the shelves. They too have some interesting network oriented innovations with the new HD8 switcher’s ability to source and control their cameras over the open internet. It makes some gears turn in my head to think how this could be implemented in my niche. I can certainly see this as a cool solution in the HoW and Educational markets

Barco is teasing a new switcher product with VoIP interfaces. I look forward to seeing where the international market Barco reaches has steered their product development? Will it be resolution independent, tied to a popular format like NDI, or standards based. The Barco Eventmaster line, Christie Spyder, and Analog Ways offerings are providing a critical service to my industry niche by bridging the gap of switching between standard resolution and non standard resolution sources. As I monitor the operator facebook pages, my guess would be that 9 out of 10 posts are about non standard aspect ratio interfacing, not since the middle of the pandemic has the majority of the posts reflected a desire for IP based signal integration.

Are Barco and For-A both looking to solve for the hopefully once in a lifetime demands of a shift to largely virtual production. Yes and no. While I believe we are unlikely to return to pandemic levels of virtual versus in-person production, IP based video transport is still super helpful and will become a valuable tool for all production moving forward.?Now Apple has moved the cheese. How does the Apple Vision impact the whole resolution discussion?

Where I believe all manufacturers need to be looking toward is resolution agnosticism. IP based video transport infrastructures are priming the pump for this. Powerful computing and AI are likely to result in better and better compression algorithms. Aspect and resolution are simply headers in the stream of bits that flow through networks. Right now, the popular IP based video standards are 16:9 ratio, HD or 4K resolution with hardware encode and decode. Standards breed interoperability. Standards are something I love because it makes my daily tasks of interfacing components from various manufacturers easy. However, interoperability is something Apple has never cared a lick about. The new Apple paradigm is to create flexible aspect and resolution experiences within tiny display panels that are retina resolution. Not fixed resolution media playback that fills the eyepiece displays. Apple is creating an immersive environment because they recognize your environment is a critical part of enjoying the content. And they utilize every visual cue to focus you on the content. The vision headset touts 4K displays which I don’t see becoming higher resolution any time soon. This means that watching a feature film streamed in 4K will be down scaled to something closer to HD. And that will be just fine. Because it’s about how the experience makes you feel that matters. Should we ignore resolution specs moving forward? Absolutely not. I see the first lesson to learn is that how compelling the content is vastly outweighs the resolution of your display. Second, that as long as your content is displayed roughly between 720p and “retina” resolution, it’s all about the same. It’s like driving a supercar or a pickup truck on the same road. Both will get you there around the same time because of the speed limits. But the experience of getting there is what matters. Visual acuity is like the speed limit in this metaphor, It doesn’t matter how fast the supercar can go, if the pickup can go the same speed limit, you will both get there at the same time. So I foresee a future where resolution doesn’t matter nearly as much as how it is utilized. Professional AV being the stepchild of consumer electronics are starting to realize 8K is not selling that well. We have hit the speed limit somewhere between 720p and 4k. We have got to realize too that resolution is in the eye of the beholder. Chasing ever increasing resolution displays and supporting infrastructure have for the most part, become a losing proposition. I believe the future winners in the post spatial computing world are going to be the manufacturers that create aspect and resolution agnostic products that interact effortlessly with the network. Time will tell?

The article diverges, sort of..

While I’m here, I’m going to share some additional thoughts about resolution that I feel like many in my specific industry niche need to hear. The rest of this article is more of a deep dive rant that pertains to my very narrow niche of the industry. If you are a little lost already. Feel free to stop now. You will be underwater quickly in the rest.

As a regular E2 operator in the corporate meeting space, I’m immersed in the world of nonstandard aspect displays and content created for them. One thing that keeps coming up is the availability of tighter and tighter pitch LED displays within my client’s budgets as LED panel adoption drives economy of scale. This is all good right? Well… It has its pros and cons. Obviously when the LED is the background of a camera shot, the tighter the pitch the better to avoid morie artifacts in the shots. But driving the high pixel counts sometimes means maxing out the destination canvas on a single switcher frame requiring chaining multiple frames together. Or, the push towards the Analog Way or Christie switchers that support more canvas within a single frame. That’s all fine because I get paid to solve issues like this. And these switchers interface with LED processors in standards based methods like HDMI 2.0 that can be extended over fiber. In general, all of these switchers do a good job of pushing pixels over multiple synced outputs of HDMI, 12G SDI, or DisplayPort.

The input side

As we have more pixels on the display side, the assumption is that we would want to display content that matches these nonstandard and extreme resolutions right? As I mentioned above, It seems like 9 out of 10 posts on the E2 support facebook groups involve challenges getting macs or PCs to display these resolutions into inputs on the switcher. In my particular niche, Macs running Millumen have become the go-to solution for this type of media playback because the other media servers are more complex, less accessible, and often much more expensive reducing ROI. Being based on Apple hardware means that the content folks feel really comfortable with them as well. This is what PlaybackPro rode the wave of for years. As a model, Millumen is the epitome of modern resolution and aspect agnostisism. Millumen will input, playback, and output any resolution or aspect imaginable. It does this because it is built with modern IP based video transport protocols built in. However, just because these files can be created, and Millumen can play them in software, it doesn’t automatically mean that the graphics hardware will easily spit these resolutions out of their standards based copper ports. If you are not in this niche running Millumen, you may be unaware that Apple has never been able to sync video playback perfectly across physical outputs. Some PC and Linux solutions with standard resolution outputs do this like the Coyote Media server. But right now Millumen is ruling the roost, so our super large displays have to flow out of a single insane resolution output. Or does it?

Back to the Apple Retina paradigm

In my typical daily setup, the closest attendee is sitting at least 25 feet away from the backdrop LED wall. 20/20 vision can roughly resolve a distinguishable line of 1.75mm at 20 feet according to Wikipedia. At 25 feet, the minimum distinguishable pixel line quickly approaches 2, then 3 then 4 mm. Where the bulk of the attendees are, we are certainly into the mid 3 to 4 mm pitch being “retina” resolution. So we are basically exceeding retina resolution for all but the closest rows in my large LED wall meetings. The 3-4mm tiles are so popular in Rentals and Staging inventories because they are road rugged and field serviceable products. But they are also satisfying the attendees. Attendees eyes can’t resolve any tighter pitches in most cases. Should we stop at 3-4mm. In many cases we can. However, not in the background of camera shots because the cameras are resolving these pixels much better when zoomed in and now magnifying everything making for a gritty, morie filled background.

What should we do?

The answer isn’t always tighter pitch walls. In the camera instance, yes. Tighter pitch walls are the best ?solution for LED behind camera shots. Now remembering the spatial computing paradigm? Don’t we want our content to be king? And the content on camera is the presenter right? So let’s take a page out of the film school textbook in bringing the subject out and directing the eye. We have all seen ultra shallow depth of field shots our whole lives in films. The DSLR popularized this to a fault in recent years because it works. It’s a trick movie makers use to steer our eyes and a convention utilized to keep attention where they want. So blur out the background in a cinematic way on IMAG. That will cover the content is king paradigm. The problem is that we generally keep the cameras in the back of the room requiring long sports lenses which optically have long depth of field. That’s just how lenses like that work. It’s physics.

Why wouldn’t we want to artificially soften the content behind IMAG shots? Just like in Teams and Zoom? Besides being cheesy and poorly implemented, it is still popular because it’s effective. Regardless of how ugly us video experts feel. But there isn’t a device that does this at low latency even if we wanted to. Hopefully no one invents this! Let’s do it better because we won’t apply it using an algorithm in post. I'm proposing we literally display a softer image behind the presenter in the camera shots.

Content creators could blur or digitally soften the background plate of the content if displayed on a tighter pitch wall. That would certainly be helpful. But that would beg the question. Why are we killing ourselves to play softened content on super resolution displays when we can simply scale up 4K content to fill. Just like the Apple Vision having 4K displays that meet retina criteria and everything else is scaled within it. We have already established that 3-4mm LED walls are approaching retina resolution for most attendees. Most video content is no greater than 4k if originally shot on camera, so super wide video rolls could still be displayed losslessly without being the resolution of the wall. ?All other content is scaled like app panels within the retina resolution display space. Similar as we do now with powerpoint slides scaled down into pips on large walls decimating their fine details. If the background content is slightly soft due to upscaling the background content, it achieves subtle but effective depth of field focusing the viewer on the primary content within the pips. And a softer scaled up image displayed at a retinal resolution image to the camera as the background of the IMAG shots actually steers viewer focus onto the presenter. So again begging the question why are we killing ourselves to match content pixel for pixel on the backgrounds of super resolution presentation background displays when a standards based 3840 or 4096 pixel background input scaled up minimally to 5-7000 pixels using hardware scaling will work better in nearly every case.

The challenge

I put this out as a challenge to my industry comrades. Explain to me the benefits of spending hours on site trying different mac dongles and sacrificing goats to get Macs to feed non standard input resolutions into scaler switchers that are designed to scale anyway. It only makes the background crisper than the scaled Pip content and contributes to more distracting backgrounds in IMAG shots. It’s the exact opposite of what Apple is doing with the Vision goggles. It also adds unnecessary strain on the media server often causing motion artifacts and increasing render times and file transfer times for content creators. We are focusing on achieving resolution specs for our own satisfaction versus focusing attendees attention subtly on the content they are actually there for.

So next time you have a 5-6,000 pixel 3:1 LED wall, how about telling the content creator that the background is 3840px by 1280px. You might get your content a day earlier because it doesn’t take as long to render, you will be ready for rehearsals an hour sooner because you can instantly plug and play a 4k input with nearly any aftermarket mac dongle, and no attendee will bat an eye.