Adding Gender to Lidar Journey Measurement

Lidar Retail Analytics with Camera Re-Match

One of lidar’s selling points is that it comes with a great privacy story. Lidar is more like a radar than a camera. It doesn’t capture any fine-grained information about the objects it’s tracking. It knows an object’s size and speed, but that’s about it. With lidar people-measurement you don’t capture ANY personally identifiable information (PII). And that’s great.

Except when it isn’t.

Because while very few of our clients want to delve into facial recognition, many – particularly in retail – find gender segmentation to be analytically useful and potentially important for in-store personalization. Fortunately, if you're interested in gender segmentation, it’s not too hard to get the best of both camera and lidar sensors in a single implementation. You can use a single camera sensor for gender classification and then do all of your full-journey flow analytics with lidar sensors. The key is a technique we call camera re-match – it allows you blend data from a camera sensor with data on lidar sensors. It’s inexpensive to deploy, quite accurate, and it provides all the advantages of lidar tracking (cost, efficiency, coverage, privacy) with the addition of gender classification.

?

Camera Re-match Systems from End-to-End

Camera re-match works by blending data from two different kinds of sensors into a single stream. The data from each stream is matched and then the meta-data from one stream is added to the journey data from the other stream. The meta-data then becomes available for downstream filtering and segmentation allowing the analyst to report on the prevalence and behavior of a specific gender.

It looks like this:

The rematch transfers meta-data from one system to another. It’s important to realize that this doesn’t create any new journey records. We aren’t (as we sometimes do), fusing two measurement systems together so that records are moving from camera coverage to lidar coverage and vice versa. Instead, the coverage areas overlap. Typically, lidar covers the entire space or location and camera just covers a single chokepoint. Because of this, camera records are just thrown away after the re-match process – the only part of them that lives on is the meta-data (usually gender) that they captured and that is transferred to the lidar journey record. This meta data is attached to the object type and then becomes available in our segmentation engine:

?This lets the analyst build dynamic segments that capture all the behavior from a specific gender (or all behavior). Segments can then be applied to any analytic view including full journey playback. Here’s a heatmap of the footfall for females in a sample location:

?The beauty of dynamic segmentation with gender classification means that you can get every analysis with or without gender baked in. You can compare the path of men and women in the store:

And you can use journey playback to visually track those different journeys:?

Finally, since object types are part of our real-time alerting capabilities, you can use gender to drive specific personalization strategies in the store. That might include tweaks to lighting and soundscapes, or, most commonly, changes to video content. And keep in mind that while I’ve treated camera re-match as a way to create gender classifications, it’s uses are broader. If, for example, you are using camera to identify a specific person, you can then use re-match to match that person and their journey at every subsequent touch-point. That gives you a way to potentially create highly personalized on-location experiences based on a combination of CRM data and journey data. You can even use the camera capabilities to create whole new kinds of meta data. You might, for example, use re-match to append the make and model of a car to a lidar journey record. This would allow you to segment and report on car makes – something that’s impossible with just lidar data.

?

The Technical Challenges

The basic idea for camera re-match is simple. A camera sensor is added to one or more key chokepoints at a location with lidar-based people measurement. The most common choice is to mount a camera at the entrance. Because the camera and lidar sensors overlap in coverage, you can align the measurement systems of each and then match records between the two systems. Ideally, if you see person X at 12:11.13 pm in a particular spot with the lidar sensors, you’ll see person Y at 12:11.13pm at that same spot with your camera sensor. That makes a match. You use the camera record to attach the gender classification to the lidar record, and you can now build reports and analyze behavior using those gender classifications.

Simple, right?

But this being people-measurement, it’s usually a bit more complex than that under the hood. Even if you can perfectly align the coordinate systems, lidar and camera tend to use somewhat different algorithms for assigning the location of an object. It might be the location of the head, the estimated location of the feet, or the centroid of the object. That means that object X,Y coordinates never match exactly. Complicating matters, camera and lidar will usually work at different frame rates and both systems often produce slightly gappy data. That means a lidar record might have data for 12:11.13.1, 12:11.13.3, 12:11.13.7, and 12:11.13.9 – or four frames in a second. The camera record might have 2 frames in that time and none of them might exactly match down to the 10th of a second. That shouldn’t make a difference if a single person enters a location, but the more people there are entering together, the more important those fine-grained decisions are going to be. If Person X follows Person Y through a door, their X,Y coordinates will be nearly identical but offset by a few 10ths of a second. Figuring out who is who in each system is critical to robust camera re-match and gender classification.

You can use any camera/ML system to do the gender classification and rematch. However, the easiest solution is to use a camera sensor (like Xovis) that is already doing both gender classification and x,y coordinate assignment.

There’s one final complication to camera re-match and it’s particularly telling when you’re doing rematch at an entry (though it is important everywhere). I’ve written before about start-stop asymmetries. Measurement systems typically require a second or two to acquire and identify an object. That means when an object is first detected by a system, it will take a few frames before that object is identified and classified. Because of this, lidar and camera systems will almost always create the same object at a different time. Depending on coverage areas, creation may even be at a dramatically different time. Re-match needs to handle this seamlessly – you’ll almost never see two records that start (or stop) at the same time when doing re-match. It’s a classic IT problem – different systems are going to generate different data.

Naturally, all this complexity gets handled out-of-the-box in our DM1 system. We use a variation of our track stitching logic to find the best possible match for each camera/lidar record. It finds the closest possible frame times, calculates the distance difference, and finds the match that minimizes the sum of those matched frames. It also let’s us set boundaries so that if that sum of differences gets too large, we reject the match. That boundary is like the reserve price at an auction. We don’t want to match two records just because they are the best match – they also need to be a good match!

Once we’ve got a good match, there’s at least one more challenge to solve. Camera systems attach a gender classification to every frame. That’s great, but it creates situations where the gender classification changes from frame to frame. By far the most common pattern is going from unknown to a specified gender. But in any system that does frame by frame classification, you will see every possible combination of classifications. A single id may get classified as unknown, male and female multiple times over a single track. So, what to do?

We tend to be very conservative in this situation. We expect records to go from unknown to male or female, but if a record is classified as both male and female, we usually set the classification to unknown. We do the same thing later in our cleaning process when we stitch records together. In theory, we can use the gender assignment to help with stitching, but we tend to prefer movement-based stitching. If it looks to our stitching engine like record X should be stitched to record Y and they have different gender classifications, we usually do the stitch and set the gender classification to unknown. That doesn’t happen often, of course, but it DOES happen.

Why it Matters

Although gender is getting harder to classify, it remains a uniquely valuable demographic variable, especially in retail. The ability to code for gender creates deeper analytic and reporting opportunities and can help illuminate WHY things happened. It can even help track the effectiveness of channel marketing strategies and how they are impacting the customer mix in your stores. And, as I pointed out above, this isn’t just about gender.

Camera re-match is a generic capability that can be used to append any kind of data you can derive from camera into a lidar journey stream. Not only is camera a richer mechanism for many kinds of object classification, there are a lot more camera-based ML models for doing that specialized object classification. If you need object identification capabilities that are either unsupported or simply impossible with lidar, camera rematch may give you the best of both worlds. You get the journey measurement benefits of lidar and the object classification benefits of video – all with the addition (in most cases) of a single, inexpensive camera sensor!