Abstract: Sensors in a facility generate sensor data associated with a region of the facility, which can be used to determine a 3D location of an object in the facility. Some sensors may sense overlapping regions of the facility. For example, a first sensor may generate data associated with a first region of the facility, while a second sensor may generate data associated with a second region of the facility that partially overlaps the first region. Sensors may fail at times as determined from sensor output data or status data. In response to identifying a failed sensor, an undetected region corresponding to the failed sensor is identified, as well as a substitute sensor that partially senses the undetected region. Sensor data from the substitute sensor, such as 2D data, is acquired and used to estimate a 3D location of an object in the undetected region.

Abstract: This disclosure is directed to techniques in which a first user in an environment scans visual indicia associated with an item, such as a barcode, before handing the item to a second user. One or more computing devices may receive an indication of the scan, retrieve image data of the interaction from a camera within the environment, identify the user that received the item, and update a virtual cart associated with the second user to indicate addition of the item.

Abstract: Sensor data from load cells at a shelf is processed using a first time window to produce first event data describing coarse and sub-events. Location data is determined that indicates where on the shelf weight changes occurred at particular times. Hypotheses are generated using information about where items are stowed, weights of those of items, type of event, and the location data. If confidence values of these hypotheses are below a threshold value, second event data is determined by merging adjacent sub-events. This second event data is then used to determine second hypotheses which are then assessed. A hypothesis with a high confidence value is used to generate interaction data indicative of picks or places of particular quantities of particular types of items from the shelf.

Abstract: Sensors in a facility obtain sensor data about a user's interaction with a fixture, such as a shelf. The sensor data may include images such as obtained from overhead cameras, weight changes from weight sensors at the shelf, and so forth. Based on the sensor data, one or more hypotheses that indicate the items and quantity may be determined and assessed. The hypotheses may be based on information such as the location of a user and where their hands are, weight changes, physical layout data indicative of where items are stowed, cart data indicative of what items are in the possession of the user, and so forth. A hypothesis having a greatest confidence value may be deemed to be representative of the user's interaction, and interaction data indicative of the item and quantity may be stored.

Abstract: This disclosure describes systems and techniques for detecting certain activity in image data, such as frames of video data. For example, the systems and techniques may create and utilize an activity classifier for detecting and classifying certain human activity in video data of a facility. In some instances, the classifier may be trained to identify, from the video data, certain predefined activity such as a user picking an item from a shelf, a user returning an item to a shelf, a first user passing an item to a second user, or the like. In some instances, the techniques enable activity detection using only video data, rather than in addition to data acquired by other sensors.

Abstract: In a materials handling facility, events may be associated with users based on imaging data captured from multiple fields of view. When an event is detected at a location within the fields of view of multiple cameras, two or more of the cameras may be identified as having captured images of the location at a time of the event. Users within the materials handling facility may be identified from images captured prior to, during or after the event, and visual representations of the respective actors may be generated from the images. The event may be associated with one of the users based on distances between the users' hands and the location of the event, as determined from the visual representations, or based on imaging data captured from the users' hands, which may be processed to determine which, if any, of such hands includes an item associated with the event.

Abstract: Load cells measure the weight of items on a shelf. Weight changes occur as items are picked from or placed to the fixture. Information about these weight changes is used to determine an estimated location on the shelf of a weight change. Hypotheses are generated using information about where particular types of items are stowed, the weights of those particular types of items, information about the weight changes, and the estimated locations of the weight changes. A model is used to produce confidence values in the hypotheses based on a change in weight measured at a first side and a change in weight measured at a second side of the shelf. A hypothesis with a confidence value that exceeds the threshold may be selected and used to determine interaction data indicative of a quantity picked or placed, type of item, and location on the shelf.

Abstract: Described are systems and techniques to generate data for display to present visualizations of data acquired from sensors in a facility. The data visualizations may be used to develop, configure, administer, or otherwise support operation of the facility. In one implementation, the visualization may include a view incorporating aggregated images acquired from multiple cameras, depth data, tracking information about objects in the facility, and so forth. An analyst may use the data visualization to determine occurrence of an action in the facility such as a pick of an item, place of an item, what item was involved with an action, what user was involved with the action, and so forth. Based on the information presented by the data visualization, changes may be made to data processing parameters.

Abstract: In one or more implementations, groups of users enter a facility within a short time. Automated systems attempt to predict a grouping for various users, but the predicted grouping may not exhibit a confidence level above a certain threshold level. In such cases, a user interface is generated and presented to an associate to confirm the grouping of users. The user interface includes an image gallery and interface elements that allow for the selection of users into a group. The image gallery includes previously stored images of users, such as images obtained during a registration process or during a previous visit to the facility. In addition, the user interface also includes image data obtained from a camera at the facility depicting visitors at various times or locations. Once a grouping has been selected or confirmed by an associate, group data may be generated and stored for the group of users.

Abstract: This disclosure describes techniques for updating planogram data associated with a facility. The planogram may indicate, for different shelves and other inventory locations within the facility, which items are on which shelves. For example, the planogram data may indicate that a particular item is located on a particular shelf. Therefore, when a system identifies that a user has taken an item from that shelf, the system may update a virtual cart of that user to indicate addition of the particular item. In some instances, however, a new item may be stocked on the example shelf instead of a previous item. The techniques described herein may use sensor data generated in the facility to identify this change and update the planogram data to indicate an association between the shelf and the new item.

Abstract: This disclosure describes, in part, systems for enabling physical retail stores and other facilities to implement automated-checkout techniques for the purchase of customizable and/or made-to-items items by customers of the stores and/or facilities. For example, the described systems may enable a retail store to implement technology where users are able to order customizable items from an ordering location, an inventory location, or an associate of the facility, and exit the store without performing manual checkout of the items. The systems described herein thus enable customized retail facilities, as opposed to a retail facility that allows automated-checkout only for prepackaged-type or otherwise non-customizable merchandise.

Abstract: In a materials handling facility, events may be associated with users based on imaging data captured from multiple fields of view. When an event is detected at a location within the fields of view of multiple cameras, two or more of the cameras may be identified as having captured images of the location at a time of the event. Users within the materials handling facility may be identified from images captured prior to, during or after the event, and visual representations of the respective actors may be generated from the images. The event may be associated with one of the users based on distances between the users' hands and the location of the event, as determined from the visual representations, or based on imaging data captured from the users' hands, which may be processed to determine which, if any, of such hands includes an item associated with the event.

Abstract: Described is a multiple-camera system for use in capturing images of users within a materials handling facility and processing those images to monitor the movement of users. For large materials handling facilities, a large number of cameras may be required to monitor the facility. Processing of the data generated from a large number of cameras becomes difficult. The implementations described herein include a hierarchy that allows image data from any number of cameras within a materials handling facility to be processed without substantially increasing the processing time needed or sacrificing processing capabilities.

Abstract: Described is a multiple-camera system and process for determining an item involved in an event. For example, when a user picks an item or places an item at an inventory location, image information for the item may be obtained and processed to identify the item involved in the event and associate that item with the user.

Abstract: During operation, a facility may utilize many sensors, such as cameras, to generate sensor data. The sensor data may be processed by an inventory management system to track objects, determine the occurrence of events at the facility, and so forth. At any given time, some of these sensors may fail to provide timely data, may fail to provide any data, may generate inaccurate data, and so forth. Described are techniques to determine failure of sensors and adjust operation of the inventory management system to maintain operability during sensor failure.

Abstract: Described is a multiple-camera system and process for disambiguating between multiple users and identifying which of the multiple users performed an event. For example, when an event is detected, user patterns near the location of the event are determined, along with touch points at the location of the event. User pattern orientation and/or arm trajectories between the event location and the user patterns may be determined and processed to disambiguate between multiple users and determine which user pattern is involved in the event.

Abstract: Described is a multiple-camera system and process for identifying a user that performed an event and associating that user with the event. For example, when an event is detected, user patterns near the location of the event are determined, along with touch points at the location of the event. User pattern orientation and/or arm trajectories between the event location and the user pattern may be determined and processed to link the user pattern to the event, thereby confirming the association between the event and the user.

Abstract: Users may enter a facility singly or in groups. A user interface may be presented to an associate that includes information obtained from sensors in the facility. The associate may utilize the information presented to determine if a user is alone, or to associate that user with a particular group of users. Input from the associate obtained from this user interface may then be used by a facility management system during operation to attribute the actions of the users to a particular account.

Abstract: Described is a multiple-camera system and process for determining an item involved in an event. For example, when a user picks an item or places an item at an inventory location, image information for the item may be obtained and processed to identify the item involved in the event and associate that item with the user.

Abstract: Users may enter a facility singly or in groups. Users may present entry credentials to a scanner. The entry credentials assist in identification of the user. After entry, users are tracked within the facility. If determinations about one or more of identification or tracking by an automated system fall below a threshold value, an employee associated with the facility is presented with a user interface. The user interface may include a gallery of images including those of the user and others in the facility. User input to the user interface is then used to reestablish one or more of identification or tracking.