How your PC’s camera can help you use less battery

Computer vision is a big area of focus for me right now.

We’re putting accelerators in things called FPGAs, field programmable gate arrays. For anyone unfamiliar, this is a hardware circuit that, just to give you one example, they used at NASA to land Perseverance on Mars. It’s an integrated circuit array that can be programmed and reprogrammed to carry out different logical operations without the hardware needing to be replaced.

For more clarity, an FPGA is different to your standard CPU chip. With a CPU, what you get is what you get. It gets programmed once and that’s it. The software on the chip can be deleted and replaced, or the chip itself can be replaced with a newer model. Those are your options.?

There’s no chip for an FPGA. It can be reprogrammed to carry out simple or complex functions, standalone or as a wider, multi-core experience.?

You can see how FPGAs would be useful for a Mars rover - bit difficult to replace a rover’s chips once you’ve launched it.

How does this relate to computer vision, Jerry?

A while back I wrote an article about the curse of the cat. I talked about how we’ve developed AI that can tell when you, the human, registered user, approaches your PC, and it can start booting up in anticipation for you (instead of your cat, or your colleague who shares your desk space).?

We’re using an FPGA to make that happen with the computer’s vision - the cameras. I only want my computer to come on if it’s Jerry the camera recognizes, and I only want it to boot up if it recognizes I’m here to sit down, not pick up my coffee cup and leave again. That’s the kind of logic we’re burning into the FPGA to develop the algorithm that it starts to recognize me and my intention.?

Here’s another situation where that can come in useful. A lot of people these days use multiple monitors. When your attention is focused on the secondary monitor, the brightness on your primary display doesn’t need to be at full. The computer vision system should be able to recognize that and dim appropriately, to save on battery and max performance on that second screen.?

Also, if you’re on a phone call, swiveling around in your chair, both monitors should be able to react appropriately. There goes Jerry, not paying us any attention. Let’s put everything on hold for a minute.?

These processes can save twenty minutes to an hour of battery time in a day. That can be the difference between making it to the next charging station and not.?

The future of FPGA-powered device experiences

The best part of using FPGAs for this sort of thing is that as we learn more - the manufacturers, the developers - we get smarter about how we do things. This usually results in the next generation of devices being better, right? Now you don’t have to wait for the next gen. With FPGAs, you get the benefit of everything we know at a point in time, because we can keep reprogramming the arrays, updating the algorithms, to the best of our ability.?

I wrote another article more recently about the concept of the XPU, which is about developing custom chips for custom processes (think a customized gaming chip, or a customized collaboration chip), and another one about how we can develop an AI layer working with the hardware to decide where processes run. I saw this article from Microsoft talking about NPUs - same idea.?

FPGAs are a factor in how you make customizable chips work. I can use an FPGA’s programmable logic to direct a system to focus on this set of factors for better collaboration tasks, or this set of factors for better gaming, or this set of factors for better performance for an animator’s workstation versus an AI developer’s workstation.?

These experiences can probably be brute-forced with CPUs, but you’d need a way bigger battery to start with, and a lot more fiddling around besides.?

The future of customer experience with devices

All of this technical talk boils down to one thing: we want to make our PCs smarter, but subtly.?

I want my PC to handle increasingly complex tasks without me, the user, having to worry about how it’s doing it.?

It’s not Jerry-the-user’s job to worry about how the PC is doing something, just that it’s getting done. It’s Jerry-the-developer’s job to worry about how the PC is carrying out its functions, and if he can’t be on call to come to every user’s house to reprogram or upgrade their internal systems, we’ve got to find other ways to keep him in the loop - but invisibly.?

That’s what we’re experimenting with now. Back when my team and I started delving into this a few years ago, it was my impression that FPGAs worked fine, they were flexible, but they weren’t power efficient. But that’s changed now. We’ve come a long way in how we use them, how we program them, and how we pair them with each other and with other parts of the system to do some really interesting things.?

The jury is still out on the final form of what this is going to look like when it comes to computer vision. I find that thrilling. Across my 29.5 years with IBM and Lenovo, I’ve always been excited by the idea that we’re not working towards an end-state. There’s always going to be someone asking “what’s next?”?

I doubt when we first put webcams into computers, we thought they’d be an interesting tool that could help the computer run smarter and use less battery, but here we are.?

Right now, this is what’s next. Six months from now, who knows? Stay tuned.?