Amazon Go: the droid Amazon is looking for

Last Monday Amazon lifted the veil on its newest venture Amazon Go. Out of this realm nature of “the thing” caused shock and awe, inducing mixed feelings to retailers, anger and fear to cashiers and syndicates. Much to the dismay and cheering Amazon Go is one of the best tangible examples I have ever seen which increases wellbeing by applying AI into everyday life.

(If you haven’t seen Amazon Go in action, I advise you to watch its debut video before reading the rest)

Basically, the technology behind Amazon Go enables a new shopping experience, founded on deep learning and sensors. The experience seems so frictionless that shopping there will certainly create the uncanny feeling of stealing.

The most friction attenuating part is removing the cashiers out of equation, the second largest occupation (3.4 million) in the United States and grocery stores are the biggest employers (25%). According to a study an average grocery shopper spends 44% of the time navigating between aisles and waiting in checkout lines.

The technology; is Amazon Go “A T?r?k”?

The technology Amazon Go houses is a very good implementation of readily available technology to reduce that “44%”. Let’s demystify inner workings by dividing the whole experience into subprocesses.

1. Identifying the customer, 2. Tracking the customer, 3. Registering a purchase/return, 4. Extreme cases (theft, unprecedented situations/scenarios).

I am going to supplement my narration with visuals, please explore the links for a better and more enjoyable understanding

1. The moment a customer scans her phone, the system identifies and tags her with the associated Amazon account

2. Once she is tagged at the entrance, cameras will track her movements all around the store. The technology is readily available and already in use for more demanding areas like NBA games

3. An item has two states A: at the possession of a customer, B: not at the possession of a customer (on its original shelf, on another shelf, on the ground). Two states are very distinct and easy to classify for a well-trained machine vision.

Conveniently, brands are working as hard as they can to help you distinguish their products instantly. Distinct branding makes products recognizable to machine vision as well. As a matter of fact, machines identify images better than humans do, so identifying and counting products on any shelf or ground is doable with very high accuracy. However, it requires tedious work of teaching the system multiple times how each item looks like from different angles (I think Amazon has already done it for its warehouses long time ago) 

Until now system I’ve defined could identify who entered the store and where she is standing/walking. Next is tying the action “brand A milk has been taken off the shelf” with a payment account, in other words registering transition of an item from stationary state to possession state on a specific customer. System should be able to tell the difference between two customers which one is idly browsing and which one is picking up an item. Item pickup could be registered by detecting an arm (or any other limb) extending to the product which was crudely implementable even with an open source machine vision library nearly a decade ago, let alone up to date capabilities of Amazon. Putting an item on the shelf or any other place will do the reverse and deduct the item from the cart

4. There will of course be theft and extreme scenarios.

I don’t think theft will be a major issue. First, the moment a person steps into the store her/his personal identity will be revealed, a huge put off for theft (now you see, security guards will be pushed to fringe job descriptions). Second, system will track everything from multiple angles constantly. I think chances of a security guard catching a person stealing is smaller than an always on full attention system

An extreme case might be two customers walking into a grocery store wearing the same clothing, with similar body sizes bumping to each other in front of an aisle. Even if such a circumstance arises, thermal camera, gait recognition, or any other biometric marker will identify each person effortlessly.

However, there are shortcomings of computer vision such as adversarial noise. A kind of noise, imperceptible to human eye, that can fool a machine vision to misclassify an object or action by a huge margin (identifying a dog as an ostrich for example). Using images like this on a cardboard could compromise the system to an extent it allows theft and vandalism go unnoticed. Employing different algorithms and/or adding sensors (this is the first-time Amazon Go might need sensors other than cameras) may overcome the problem. It is very hard to tell at this early stage.

Despite malicious use cases, adversarial noise may prove beneficial on labels. Each product could be tracked with high accuracy from origin to consumer and enable deeper analytics and tracking options in case of outbreaks, such as of E. coli, and reduce recall costs and other risks.

I am confident to generalize that Amazon Go’s system performs just like a retail salesperson who closely watches a customer, nothing less, probably more. So, why did I go extra mile to dissect how the system might work? To show Amazon’s success will not boil down to technical ability to create a proof of concept, rather its unique capability to scale.

The strategy; Is Amazon in for the kill?

A proof of concept doesn’t necessarily have to scale. Each part could work optimal in a small experiment but resulting system might become suboptimal under pressure as it grows in size. That’s where Amazon’s real intentions begin to surface.

AWS’ offerings are nothing more than what a credible IT service provider is able to build in the premises of a customer. However, they are cheaper and scalable. They so cheaply scale up in a considerably shorter amount of time that companies like Netflix have been relying on AWS.

Two weeks ago, in Las Vegas Amazon introduced new capabilities of AWS and made it clear artificial intelligence (including digital assistant) is a very very very important field for Amazon. They were so serious that one week later they showed off what it is capable of in a retail store by giving us Amazon Go.

Amazon now puts the retail market in a choke hold (for now) and divides it into two. One group has the capital, assets and ambitions to research and create an Amazon Go all by itself (maybe Walmart), and will face complex cost and scale dilemmas. The other group has no capital (or interest in) to invest in research and will either buy or die. It is very hard to compete in such tight margins and forego even the faintest opportunity to create an advantage, so they like it or not every retailer is in.

I believe reserving those capabilities for itself and play a long game of retail dominance through physical store expansion and a cost based fistfight is not what Amazon is after. Why? Because selling it “as a Service” is very much aligned with Amazon’s strategies and more beneficial:

• Increasingly Amazon’s volume comes from third party sellers, currently standing at 50%. Amazon lets third party retailers tap into its supply chain, and help them deliver their products. Likewise, Amazon Go will empower lots of established and new retailers by giving them tools (discussed at the competition part) to operate and compete. The fragmentation already induced by Amazon in the US retail market will accelerate and wear off big chains’ competitive leverage by frittering their scale away
• AWS model is built upon scale, the bigger it gets the better margins it yields. And of course, the quicker it upsizes, the better cost advantage it gets with respect to its competitors. Waiting to scale with own store expansions is not going to cut it
• It might yield higher margins than operating a physical store. Currently AWS is the most profitable segment of Amazon, with 26.6% operating income margin (North America 1.4%, International -5.1%)
• Create an ecosystem to expand the service to other retail markets
• Amazon Go will accelerate offline to online migration substantially, so the more contracts Amazon signs, the wider reach and exposure it will get

Those upsides are all obvious first order benefits, but as always there are second order benefits. In this case, very big and lucrative ones (which need lengthy analysis of their own) hide in plain sight:

• Infusing its technology into other stores will yield immense amount of data about offline market dynamics such as shopping cart compositions, shopping behaviors, store optimization which will later turn into better tailored services and stores. Amazon will hold cross section view to the market practices, up to date offline and online prices, impact of campaigns and etc. and devise strategies no other firm can match (of course an antitrust issue may arise)
• Migrating offline purchase information to online will effortlessly create a digital inventory for every customer. Having an asset list automatically categorizes consumers as potential merchants and raises the possibility of frictionless second hand market (maybe with a dash of social aspect). Given the fact that 10% of households own a self-storage space and 1 in 3 people think it is easier to throw things out, an easier asset/inventory management seamlessly integrated to a marketplace has the potential to bring lots of idle stuff to market, and traffic to Amazon assets. On top, Amazon may add those storages into its logistics as last mile storage units (or maybe venture into a $27 bn self-storage business,)
• Amazon recently ramped up its clothing vertical efforts by launching thousands of products. The instore videos hold key to digitize the body shapes and sizes of people, which will in return optimize size and cut ranges of Amazon. Also, consumers shopping on Amazon would instantly see the best fitting size of a clothing item (a very good opportunity for second hand market as well as an average American throws away 37 kilograms of clothes in a year)
• Once stores enroll to Amazon Go and integrate their backend with Amazon’s (or migrate all together to AWS), they will be extensions to overall supply chain. There is a possibility that Amazon may absorb logistics of extensions into Fulfilled by Amazon (FBA) and beef up its transportation and warehouse businesses significantly
• Amazon may intensify its delivery efforts and gobble up weary instant delivery market. Market’s low to no margin and scale dependent nature is extremely nourishing for Amazon to thrive and crush competition. Not only Amazon is a logistics behemoth, hearing name of it will drive away investors of most delivery companies which depend on constant cash supply. Mechanical Turk, Amazon’s crowdsourcing platform, may evolve into a broader program to coordinate both instant delivery and in store crowdsourcing operations (let’s leave MT as a wild card)
• According to a survey, Amazon payments is the second most preferred alternative payment option to debit/credit cards. Integrating payment to the service might hold the key to the kingdom of alternative payment/wallet (which retailers failed with MCX initiative, and Apple, Google, Samsung haven’t gain much traction despite their colossal platforms). Scale in payment will bring the opportunity to sustain a closed economy inside Amazon ecosystem with herd to break loyalty (dystopic or utopic?)
• As in other parts of our lives, Amazon will position smartphone to the center of offline shopping experience. Smartphone will control the information flow between the product and customer. It will accelerate development of both customer and store facing ecosystems. One of the most important aspects of the ecosystem will be AI powered digital assistant. Currently, and arguably, Amazon leads the digital assistant race with Alexa. Spreading Amazon Go through retail stores will expose Alexa to inconceivable amount of consumer data to train with. Also, consumers will get exposed to it as an assistant on the shop floor (yes, the largest occupation in the US, retail salesperson will go away as well) and the more they use it in stores, the more they will let it in their daily lives, until to a point Alexa will be the default OS whatever device they use. Think about for a while how big the world gets for Amazon then (in my opinion this is by far the most important benefit Amazon could extract from Amazon Go opportunity)

The competition; too big for Amazon to keep it all, where are the others?

Of course, not all the riches will be Amazon’s to enjoy. Google, Microsoft, IBM, and other cloud providers will rush to the market as soon as possible. They may currently be trying to convince Walmart into a form of partnership to recreate Amazon Go. One thing is certain they will figure out competing offerings and challenge Amazon like they have been doing with the cloud business.

Will they catch up? Not so quickly. Amazon holds very decisive advantages. The platform offers great benefits to both consumer and store owners in single player mode. For consumers:

• Amazon has already established relations with consumers. It has payment information, purchase history and will be the single source of billing
• It will complement online purchasing with additional benefits like delivery pickup options, extend benefits of Prime membership to offline stores
• It will make in store comparison extremely easy through automatically bringing up reviews, price comparisons, promotional codes and loyalty program coverages about the product a shopper holds at hand. This is a very hard to replicate feature. If Amazon adds a social aspect to it, it will be a killer feature

On the store side potential is much bigger:

• Cost benefits are big. Total cost of cashiers to grocery stores last year was around $19 bn, %3.0 of grocery stores’ $617 bn revenue. 3.0% is more than double the 1.4% profit margin of listed grocery businesses in US. Also, instore tracking coupled with AWS’ IoT capabilities will reduce store operating costs such as energy
• Stores will enjoy Amazon grade online store, analytics/forecast, reporting and CRM, security tools
• They will tap into Amazon Prime customers which spend 2-3 times the non-Prime members. Loyalty difference is stark and worth enrolling into Amazon Go.
• Stores will have access to FBA, it will be a gamechanger for smaller stores and worthy benefit to bigger ones

Those will give head start to Amazon to build multiplayer mode benefits like developer ecosystem of Amazon Go on AWS, better margins due to size of the service, store-to-store (S2S) collaboration through Amazon back-end etc. (for stores); increased Prime geographic and store coverage, lower prices, cleverer digital assistant, more standardized offline experiences, seeing availability of an item in another store etc (for consumers). Once multiplayer kicks in Amazon’s previously established levers, like FBA, being the default destination for product search will evolve into unimaginable moats. I literally cannot stop thinking of new ways to utilize Amazon Go and how it will drastically alter the retail as we know it.

The elephant in the room; what will happen to all the cashiers, security guards and retail salespersons?

Inventing new things without proper countermeasures has always been one of the problems of science. Driving people out of jobs, in this case, is something we should really find a solution to. This is kind of a topic that I want to dig deep in another time, so let me briefly tell my view and leave you with it.

Although capabilities of machines on sensory based readings exceed humans (like classifying images, speech recognition, reading lips etc.), they are far behind on cognitive abilities. Employing people on tasks machines are better therefore is waste of their precious mental capabilities and time. Leaning on that argument, I think stores should shift their staff to more value-added operations, like empowering them to address the unbelievable food waste problem of the United States.

Now that Amazon opened the Pandora’s Box, the others will quickly roll out other use cases and intensify the competition, the kind of scenario that will fuel the fear of losing jobs to AI. Amazon Go, therefore, might be the kind of nudge market has been waiting for to start unloading much held back arsenals.

It looks like losing jobs to machines rather than immigrants will be a thing in Trump era, whether his propaganda machine is ready or not!