Abstract: Systems and techniques are provided for calibrating cameras in a real space for tracking puts and takes of items by subjects. The method includes first processing one or more selected images captured by an extrinsic calibration tool to extract from the images, a three-dimensional (3D) point cloud of points corresponding to displays or structures that remain substantially immobile in the real space, and a second processing of a second set of images captured by a camera installation in the real space to match points in the second set of images to the point cloud. Differences in position of matching points can be used to determine transformation information therebetween, which can be used to calibrate the installed cameras.

Abstract: Image analysis from one or more cameras are used for monitoring items on a display. A count of items on the display is received. Images from the one or more cameras are used to count a number of items removed from the display. A report is generated containing a number of items on the display, after counting the number of items removed from the display using the images from the one or more cameras.

Abstract: Systems and techniques are provided for linking subjects in an area of real space with user accounts. The user accounts are linked with client applications executable on mobile computing devices. A plurality of cameras are disposed above the area. The cameras in the plurality of cameras produce respective sequences of images in corresponding fields of view in the real space. A processing system is coupled to the plurality of cameras. The processing system includes logic to determine locations of subjects represented in the images. The processing system further includes logic to match the identified subjects with user accounts by identifying locations of the mobile computing devices executing client applications in the area of real space and matching locations of the mobile computing devices with locations of the subjects.

Abstract: Techniques for managing coverage constraints are provided for determining an improved camera coverage plan including a number, a placement, and a pose of cameras that are arranged to track puts and takes of items by subjects in a three-dimensional real space. The method includes receiving an initial camera coverage plan including a three-dimensional map of a real space, an initial number and initial pose of a plurality of cameras and a camera model including characteristics of the cameras. The method can iteratively apply a machine learning process to an objective function of number and poses of cameras, and subject to a set of constraints, obtain an improved camera coverage plan. The improved camera coverage plan is provided to an installer to arrange cameras to track puts and takes of items by subjects in the three-dimensional real space.

Abstract: The technology disclosed teaches systems and methods for self-service installation of monitoring operations in an area of real space, the method including scanning the area of real space to generate a 3D representation of the area of real space, placing a camera at an initial location and orientation for monitoring a zone within the area of real space, configuring a computing device to be connected to a cloud network hosting an image processing service and couplable to the camera, coupling the camera to the computing device via a local connection using a unique identifier associated with the camera, and finetuning the camera placement to a calibrated location and orientation, wherein the finetuning is assisted by information received from a cloud-based application associated with the image processing service.

Abstract: A method for identifying gestures in an area of real space is provided. The method including using a plurality of sensors to produce respective sequences of frames of corresponding fields of view in the area of real space, detecting subjects in the area of real space, using a sequence of frames produced by a corresponding sensor in the plurality of sensors in a first inference engine to identify inventory items carried by the detected subjects in the sequence of frames, using outputs of the first inference engine over a period of time in a second inference engine to identify gestures of the detected subjects, and storing the identified gestures in a database.

Abstract: Automated techniques provide for recalibrating cameras configured for monitoring an area of real space. The method includes first processing one or more selected images selected from a plurality of sequences of images received from a plurality of cameras calibrated using a set of calibration images that were used to calibrate the cameras previously. Transformation information between the selected images and the set of calibration images is obtained based on one or more features extracted from the selected images using a trained neural network classifier corresponding to points located at relatively immobile structures. The features extracted from the selected images match features in the set of calibration images. Camera calibrations can be updated when transform information between features matched meets or exceeds a threshold.

Abstract: The technology disclosed relates to a method for tracking inventory items including dividing an area of real space into a plurality of cells and identifying inventory items and their locations in the area of real space using sequences of images produced by at least two sensors. The identifying of inventory items further includes comparing a score for the particular inventory item having a location matching a particular cell and based on one or more counts of inventory events, and a second threshold, and accepting or rejecting the particular inventory item as the predicted inventory item in dependence on whether the score is above the second threshold. In response to the identification of the inventory items, the method also includes determining, for a particular inventory item matched with a particular cell, whether a count of the particular inventory item is below a third threshold for re-stocking the particular inventory item.

Abstract: A method for identifying and tracking an inventory item is provided. The method includes using two or more sensors to produce two or more outputs associated with an inventory item, including at least one of a possible identifier for the inventory item and a possible event type indicating whether the inventory item has been taken from a shelf or placed onto a shelf. The method further includes generating confidence levels for the inventory item, wherein each respective confidence level is based on a subset of one or more outputs, and identifying, based on the confidence levels, a prediction for the inventory item including at least one of a predicted identifier for the inventory item and a predicted event type for the inventory item. The method further includes modifying a planogram of a shopping store based on the prediction for the inventory item.

Abstract: Automated techniques provide for recalibrating cameras in a real space in which puts and takes of items are tracked. The method includes first processing one or more selected images selected from a plurality of sequences of images received from a plurality of cameras calibrated using a set of calibration images that were used to calibrate the cameras previously. The first processing includes a process step to match one or more features from the selected images with features extracted from the set of calibration images using a trained neural network classifier. The features correspond to points located at displays or structures that remain substantially immobile. Camera calibrations can be updated when transform information between features matched meets or exceeds a threshold.

Abstract: A method for tracking inventory items is provided. The method includes using sensors to produce respective sequences of images of corresponding fields of view, the field of view of each sensor overlapping with the field of view of at least one other sensor. The method includes dividing the area of real space into cells having volume and coordinates in a real space, and identifying inventory items and their locations in three dimensions using the sequences of images produced by at least two sensors having overlapping fields of view. The method includes, in response to the identification of the inventory items: (i) matching locations in three dimensions of identified inventory items with cells in the plurality of cells, and (ii) determining, for a particular inventory item matched with a particular cell, whether a count of the particular inventory item is below a threshold for re-stocking the particular inventory item.

Abstract: A method for identifying an inventory item is provided. The method includes using first one or more sensors to produce first one or more outputs indicative of an inventory item, using second one or more sensors to produce second one or more outputs indicative of the inventory item, generating, based on the first one or more outputs and independent of the second one or more outputs, a first confidence level of the inventory item being a first inventory item, generating, based on the second one or more outputs and independent of the first one or more outputs, a second confidence level of the inventory item being a second inventory item, and identifying, based on the first confidence level and the second confidence level, the inventory item as one of the first inventory item or the second inventory item.

Abstract: The technology disclosed teaches a system and methods for tracking subjects in an area of real space, as well as tracking subjects in a plurality of areas of real space. The technology disclosed can also track subjects in two or more separate tracking spaces that are within a previously designated region but have separate sets of sensors and cameras. The disclosed system and methods can track groups of subjects in two or more separate tracking spaces that are within a previously designated region. For the purposes of tracking subjects, a subject persistence processing engine compares the newly located (or newly identified) subjects in the current identification interval with preceding identification intervals. This can be applied to an autonomous shopping environment, enabling shoppers to seamlessly make shopping transactions across multiple stores or locations within the previously designated region and allowing for sharing of shopping data across the multiple stores or locations.

Abstract: A method of tracking subjects in an area. The method includes processing by a plurality of trained inference engines that process respective sequences of images in the plurality of sequences of images to locate features of subjects in corresponding fields of view of the respective sequences and receiving information that includes combined located features from more than one of the trained inference engines which process respective sequences of images having overlapping fields of view to generate data locating subjects in three dimensions in the area of real space during identification intervals, and matching located subjects from a plurality of identification intervals to identify tracked subjects, including comparing located subjects with tracked subjects.

Abstract: The technology disclosed relates to systems and methods for predicting a path of a subject in an area of real space in a shopping store including a cashier-less checkout system. The system comprises a plurality of sensors producing respective sequences of frames of corresponding fields of view in the real space. The system comprises an identification device comprising logic to identify, for a particular subject, a determined path in the area of real space over a period of time using the sequences of frames produced by sensors in the plurality of sensors. The system comprises a path prediction device to predict path of new subjects in the area of real space. The system comprises a layout generation device comprising logic to change a preferred placement of a particular item, in dependence on the predicted path.

Abstract: The technology disclosed relates to detecting events and identifying items in detected events in an area of real space in a shopping store including a cashier-less checkout system. The system comprises an image sensor assembly comprising at least one narrow field of view (NFOV) image sensor and at least one wide field of view (WFOV) image sensor. The at least one NFOV image sensor produces raw image data of first-resolution frames of a corresponding field of view and the at least one WFOV image sensor produces raw image data of second-resolution frames of a corresponding field of view. The system comprises logic to provide at least a portion of a sequence of second-resolution frames produced by the WFOV image sensor to an event detection device configured to detect (i) a particular event and (ii) a location of the particular event in the area of real space.

Abstract: Systems and techniques are provided for linking subjects in an area of real space with user accounts. The user accounts are linked with client applications executable on mobile computing devices. A plurality of cameras are disposed above the area. The cameras in the plurality of cameras produce respective sequences of images in corresponding fields of view in the real space. A processing system is coupled to the plurality of cameras. The processing system includes logic to determine locations of subjects represented in the images. The processing system further includes logic to match the identified subjects with user accounts by identifying locations of the mobile computing devices executing client applications in the area of real space and matching locations of the mobile computing devices with locations of the subjects.

Abstract: Techniques for managing coverage constraints are provided for determining an improved camera coverage plan including a number, a placement, and a pose of cameras that are arranged to track puts and takes of items by subjects in a three-dimensional real space. The method includes receiving an initial camera coverage plan including a three-dimensional map of a real space, an initial number and initial pose of a plurality of cameras and a camera model including characteristics of the cameras. The method can iteratively apply a machine learning process to an objective function of number and poses of cameras, and subject to a set of constraints, obtain an improved camera coverage plan. The improved camera coverage plan is provided to an installer to arrange cameras to track puts and takes of items by subjects in the three-dimensional real space.

Abstract: An apparatus for performing image analysis to identify human actions represented in an image, comprising: a joint-determination module configured to analyse an image depicting one or more people using a first computational neural network to determine a set of joint candidates for the one or more people depicted in the image; a pose estimation module configured to derive pose estimates from the set of joint candidates that estimate a body configuration for the one or more people depicted in the image; and an action-identification module configured to analyse a region of interest within the image identified from the derived pose estimates using a second computational neural network to identify an action performed by a person depicted in the image.

Abstract: A mechanized store uses a mobile device to authenticate the user. Items removed from one or more displays of the mechanized store by the user are tracked and a list of items removed by the user is updated. The list of items removed is linked with an account of the user.