Abstract: A system that determines the items on a shelf by projecting camera images to a surface aligned with the tops of the items. Projecting the images to the top surface removes distortions due to camera projections and aligns multiple images to a common reference frame. Item tops may be visible without occlusion in one or more camera images, simplifying item identification. The projected images may be input into an item detector that is trained to recognize images of the tops of items. The item detector may process projected images from different cameras with parallel feature extractor subnetworks that generate feature maps; the feature maps may then be averaged across images and the averaged feature map may be input into an item detection subnetwork.

Abstract: A system that analyzes images of a shelf to determine the available shelf space. Camera images of the shelf from multiple viewpoints may be projected onto the shelf surface to remove distortions from camera projections and to align images to a common shelf reference frame. A mask may be calculated from each projected image that identifies regions that match the appearance of the shelf surface. The shelf surface may have a specific pattern to facilitate identification of these regions. A combined mask may be formed as a union of the masks from individual projected image masks. The available shelf space corresponds to the regions in the combined mask. Combining image masks from multiple viewpoints reduces the effect of occlusion of the shelf surface by items on the shelf.

Abstract: System that facilitates rapid onboarding of an autonomous (cashier-less) store by capturing images of the store's items from multiple angles, with varying background colors, and that builds a classifier training dataset from these images. Background surfaces may for example be coated with retroreflective tape or film, and variable-color incident light sources may generate the desired background colors. Embodiments may automatically rotate or otherwise reorient the item placed in the onboarding system, so that a relatively small number of cameras can capture views from multiple angles. When an item is placed in the system, a fully automated process may generate a sequence of item orientations and background colors, and may capture and process images from the cameras to create training images. Images of the item from multiple angles, under varying lighting conditions, may be captured without requiring an operator to move or reorient the item.

Abstract: System that analyzes images of an item from multiple viewpoints to construct a parameterized three-dimensional shape that models the item's shape. The system may search for parameter values that minimize a cost function that measures differences between the observed item images and those that would be expected with those parameter values. One illustrative cost function may measure differences between binary image masks associated with the images and projections of the parameterized shape onto each associated image reference frame. Another illustrative cost function may measure differences between colors from different images at points that are projected from the parameterized surface. These two cost functions may be used together to successively derive the parameterized shape. A byproduct of the shape estimation may include a texture map for the appearance of the item, which may be used for example to read and analyze data from an item label.

Abstract: System that analyzes data from a smart shelf that is monitored by weight sensors and cameras to identify items that are removed from the shelf and the locations of these items on the shelf. By using multiple shelf weight sensors, the location of items removed from or added to a shelf can be calculated from static equilibrium conditions. This weight-based location can be compared to regions of visual change in camera images to cross-check the location of events and to improve accuracy. The location of an item change may also be used in conjunction with a planogram to determine the item expected to be at this location; the expected item can be compared to the item identified using image analysis to further increase item identification accuracy. Weight changes can also be used to determine the quantity of items taken from a shelf.

Abstract: System that allows devices of an autonomous store, such as sensors in product display areas, to receive power and communicate data over conductive rails of store fixtures. By using fixtures to transmit data and power, the need for cabling to these devices, or for batteries to power the devices, is eliminated or greatly reduced, thereby dramatically simplifying installation and maintenance. Illustrative fixtures over which devices can communicate include slatwalls, pegboards, and rectangular support bars. Power and data may be multiplexed onto the same pair of conductive rails. Embodiments may use device hubs to coordinate communication with devices and to act as gateways between devices and centralized store servers. Devices may communicate their identities and locations to store servers to facilitate installation; for example, they may display their identities on electronic labels that are imaged by store cameras, so that the store server can learn the location of each device automatically.

Abstract: System that facilitates rapid onboarding of an autonomous (cashier-less) store by capturing images of the store's items from multiple angles, with varying backgrounds, and that builds a classifier training dataset from these images. The system may have cameras in different positions, and backgrounds that generate different background colors. It may have a platform for items that can be switched between a transparent and non-transparent state; cameras below the platform may therefore capture images of the bottom side of the item when the platform is transparent. When an item is placed in the imaging system, a fully automated process may generate a sequence of background colors and may capture and process images from all of the cameras to create training images. Images of the item from multiple angles, including views of the entire external surface of the item, may be captured without requiring an operator to move or reorient the item.

Abstract: A projected image item tracking system that analyzes projected camera images to determine items taken from, placed on, or moved on a shelf or other area in an autonomous store. The items and actions performed on them may then be attributed to a shopper near the area. Projected images may be combined to generate a 3D volume difference between the state of the area before and after shopper interaction. The volume difference may be calculated using plane-sweep stereo, or using convolutional neural networks. Because these methods may be computationally intensive, the system may first localize a change volume where items appear to have been displaced, and then generate a volume difference only within that change volume. This optimization results in significant savings in power consumption and in more rapid identification of items. The 3D volume difference may also indicate the quantity of items displaced, for example from a vertical stack.

Abstract: A case with an integrated or attached credential reader; when a user presents a valid credential (such as a biometric identity like a fingerprint, handprint, or face), the case is unlocked and the user can take items from the case. Sensors in the case such as cameras and weight sensors detect items taken by the user; these are automatically charged to the user or added to the user's shopping cart. The credential reader may be on a door handle so that as the user reaches for the handle the credential is automatically captured. The entire shopping experience may be quick and seamless since the user's credential may be captured automatically and the items the user takes may be accounted for automatically. Embodiments may have a door that opens automatically when the credential is accepted and closes and locks automatically when sensors detect that the user has retracted from the case.

Abstract: A system having a bar containing distance sensors, such as LIDARs, that may be installed behind a shelf in an automated store. Shelves may be divided into storage zones, such as bins or lanes, and a distance sensor may measure the item quantity in each zone. Quantity changes indicate that a shopper has taken or placed items in the zone. Quantity changes may trigger analysis of camera images of the shelf to identify the items taken or replaced. The distance sensor elements within the bar may be relocatable so they can be positioned correctly behind the corresponding storage zones. The bar may have mounting mechanisms on either side for attachment to shelving support structures. These mounting mechanisms may include security locks. The bar may rotate relative to the mounting mechanisms to provide access to the shelf from behind.

Abstract: An autonomous store that tracks shopper movements and actions, and performs store cleaning actions based on analysis of shopper activity. Cleaning actions may include disinfecting the store or regions within the store using radiation, fogging or spraying, or ventilation. Cleaning actions may be targeted; for example, zones where shoppers linger or congregate may be cleaned more frequently or intensively, or shelves or items that shoppers touch may be cleaned after these interactions. Shopper activity information may be used to limit the number of shoppers in a store at once, for example by denying entry when the store is at capacity. The density of shoppers in regions of the store may be communicated to shoppers so that they can limit their interactions with other shoppers. Shopper activity history may be used for contact tracing by identifying other shoppers that may have been exposed to an individual who was in the store.

Abstract: A projected image item tracking system that analyzes projected camera images to determine items taken from, placed on, or moved on a shelf or other area in an autonomous store. The items and actions performed on them may then be attributed to a shopper near the area. Projected images may be combined to generate a 3D volume difference between the state of the area before and after shopper interaction. The volume difference may be calculated using plane-sweep stereo, or using convolutional neural networks. Because these methods may be computationally intensive, the system may first localize a change volume where items appear to have been displaced, and then generate a volume difference only within that change volume. This optimization results in significant savings in power consumption and in more rapid identification of items. The 3D volume difference may also indicate the quantity of items displaced, for example from a vertical stack.

Abstract: System that facilitates rapid onboarding of an autonomous (cashier-less) store by capturing images of the store's items from multiple angles, with varying background colors, and that builds a classifier training dataset from these images. Background surfaces may for example be coated with retroreflective tape or film, and variable-color incident light sources may generate the desired background colors. Embodiments may automatically rotate or otherwise reorient the item placed in the onboarding system, so that a relatively small number of cameras can capture views from multiple angles. When an item is placed in the system, a fully automated process may generate a sequence of item orientations and background colors, and may capture and process images from the cameras to create training images. Images of the item from multiple angles, under varying lighting conditions, may be captured without requiring an operator to move or reorient the item.

Abstract: An automated store that calculates a confidence score for virtual shopping carts of shoppers, and selects carts for manual review based on these scores. Carts with low confidence scores may be more likely to contain errors, so prioritizing manual review of these carts is a cost-effective method of improving overall accuracy. A cart confidence score may be a function of factors such as confidence in the trajectory of the shopper generated by the store tracking system, confidence in the events (such as taking an item from a shelf) that affect the cart, and confidence that events are attributed to the correct shopper. Situations that make tracking, item identification, or attribution more complex may reduce confidence levels. For example, attribution confidence may be low when multiple shoppers are near an event, and item confidence may be low if the probabilistic classifier that identifies the item assigns nontrivial probabilities to multiple items.

Abstract: System that facilitates rapid onboarding of an autonomous (cashier-less) store by capturing images of the store's items from multiple angles, with varying backgrounds, and that builds a classifier training dataset from these images. The system may have cameras in different positions, and backgrounds that generate different background colors. It may have a platform for items that can be switched between a transparent and non-transparent state; cameras below the platform may therefore capture images of the bottom side of the item when the platform is transparent. When an item is placed in the imaging system, a fully automated process may generate a sequence of background colors and may capture and process images from all of the cameras to create training images. Images of the item from multiple angles, including views of the entire external surface of the item, may be captured without requiring an operator to move or reorient the item.

Abstract: System that allows devices of an autonomous store, such as sensors in product display areas, to receive power and communicate data over conductive rails of store fixtures. By using fixtures to transmit data and power, the need for cabling to these devices, or for batteries to power the devices, is eliminated or greatly reduced, thereby dramatically simplifying installation and maintenance. Illustrative fixtures over which devices can communicate include slatwalls, pegboards, and rectangular support bars. Power and data may be multiplexed onto the same pair of conductive rails. Embodiments may use device hubs to coordinate communication with devices and to act as gateways between devices and centralized store servers. Devices may communicate their identities and locations to store servers to facilitate installation; for example, they may display their identities on electronic labels that are imaged by store cameras, so that the store server can learn the location of each device automatically.

Abstract: An automated store attached to or integrated into a site where vehicles park, such as a gas station, charging station, or parking lot. The store may obtain the identity of the vehicle automatically, for example in a message sent over a charging cable, or by scanning a license plate. An authorization linked to the vehicle identity may be extended to passengers who exit the vehicle, so that these passengers may take items from the store and have them automatically charged to the vehicle's account. Locked cases containing products may be unlocked automatically when a shopper who exited an authorized vehicle arrives at the case. As passengers move around the site and obtain items from the store, messages may be transmitted back to the vehicle, or to a mobile device of the vehicle owner or driver, showing the items that have been taken.

Abstract: A sensor bar shelf monitor, for example that may be added to an existing shelving system to convert a store to autonomous operation. The sensor bar may contain distance sensor to detect shoppers reaching towards items on a shelf, and cameras to determine which items shoppers have taken. It may be installed into shelf supports such as gondola shelving uprights. The sensor bar may be located at the front edge of a shelf, and may monitor the shelf below. Placing the sensor bar along the front edge prevents damage to electronics from spills or shelf cleaning, and prevents heat from the sensor bar electronics from damaging items on the shelf. The sensor bar may have a local sensor bar processor that collects sensor data; images may be analyzed locally or transferred to more powerful store processors. A sensor bar may also have controllable lights and controllable electronic labels.

Abstract: A robot for capturing image frames of subjects in response to a request includes a base, a robot head, a camera, a vertical positioning mechanism, and a control system. The base has a transport mechanism for controllably positioning the robot along a lateral surface. The vertical positioning mechanism couples the robot head to the base and adjusts a vertical distance between the robot head and the support surface. The control system controls image capture of the camera, the transport mechanism, and the vertical positioning mechanism.

Abstract: An autonomous store that tracks shopper movements and actions, and performs store cleaning actions based on analysis of shopper activity. Cleaning actions may include disinfecting the store or regions within the store using radiation, fogging or spraying, or ventilation. Cleaning actions may be targeted; for example, zones where shoppers linger or congregate may be cleaned more frequently or intensively, or shelves or items that shoppers touch may be cleaned after these interactions. Shopper activity information may be used to limit the number of shoppers in a store at once, for example by denying entry when the store is at capacity. The density of shoppers in regions of the store may be communicated to shoppers so that they can limit their interactions with other shoppers. Shopper activity history may be used for contact tracing by identifying other shoppers that may have been exposed to an individual who was in the store.