Exclusive: Amazon’s cameras can recognize products with computer vision, no barcodes needed. Eventually the system will support robots.
Robots may be the future, but robotic arms are apparently no good at using an old, steadfast form of technology: the barcode. Barcodes can be hard to find and can be affixed to oddly shaped products, something robots can’t troubleshoot very well.
Using pictures of items in Amazon warehouses and training a computer model, the e-commerce giant has developed a camera system that can monitor items flowing one-by-one down conveyor belts to make sure they match their images. Eventually, Amazon’s AI experts and roboticists want to combine the technology with robots that identify items while picking them up and turning them around.
“Solving this problem, so robots can pick up items and process them without needing to find and scan a barcode, is fundamental,” said Nontas Antonakos, an applied science manager in Amazon’s computer vision group in Berlin. “It will help us get packages to customers more quickly and accurately.”
The system, called multi-modal identification, isn’t going to fully replace barcodes quite yet. It’s currently in use in facilities in Barcelona, Spain, and Hamburg, Germany, according to Amazon. Still, the company says it’s already speeding up the time it takes to process packages there. The technology will be shared across Amazon’s businesses, so it’s possible you could one day see a version of it at a Whole Foods or another Amazon-owned chain with in-person stores.
Amazon has built computer vision into other products. You can ask an Echo Show smart display, “Alexa, what am I holding?” to get help recognizing objects around the house. The feature is called Show and Tell and was designed with vision impaired people in mind. Smart phone makers and social media companies have also included AI features in camera and photo apps, categorizing photos automatically, for example.
The problem that the system eliminates — incorrect items coming down the line to be sent to customers — doesn’t happen too often, Amazon says. But even infrequent mistakes add up to significant slowdowns when considering just how many items a single warehouse processes in one day.
Amazon’s AI experts had to start by building up a library of images of products, something the company hadn’t had a reason to create prior to this project. The images themselves as well as data about the products’ dimensions fed the earliest versions of the algorithm, and the cameras continually capture new images of items to train the model with.
The algorithm’s accuracy rate was between 75% and 80% when first used, which Amazon considered a promising start. The company says the accuracy is now at 99%. The system faced an initial hiccup when it failed to catch color differences. During a Prime Day promotion, the system couldn’t distinguish between two different colors of Echo Dots. The only difference between the packages was a small dot that was either blue or gray. With some retooling, the identification system can now assign confidence scores to its ratings that only flag items it’s very sure are incorrect.
Amazon’s AI team says it will be a challenge to fine-tune the multi-modal identification system to assess products that are being handled by people, which is why the ultimate goal is to have robots handle them instead.
