Abstract: Systems and techniques are provided for linking subjects in an area of real space with user accounts. The user accounts are linked with client applications executable on mobile computing devices. A plurality of cameras are disposed above the area. The cameras in the plurality of cameras produce respective sequences of images in corresponding fields of view in the real space. A processing system is coupled to the plurality of cameras. The processing system includes logic to determine locations of subjects represented in the images. The processing system further includes logic to match the identified subjects with user accounts by identifying locations of the mobile computing devices executing client applications in the area of real space and matching locations of the mobile computing devices with locations of the subjects.

Abstract: Techniques for managing coverage constraints are provided for determining an improved camera coverage plan including a number, a placement, and a pose of cameras that are arranged to track puts and takes of items by subjects in a three-dimensional real space. The method includes receiving an initial camera coverage plan including a three-dimensional map of a real space, an initial number and initial pose of a plurality of cameras and a camera model including characteristics of the cameras. The method can iteratively apply a machine learning process to an objective function of number and poses of cameras, and subject to a set of constraints, obtain an improved camera coverage plan. The improved camera coverage plan is provided to an installer to arrange cameras to track puts and takes of items by subjects in the three-dimensional real space.

Abstract: An apparatus for performing image analysis to identify human actions represented in an image, comprising: a joint-determination module configured to analyse an image depicting one or more people using a first computational neural network to determine a set of joint candidates for the one or more people depicted in the image; a pose estimation module configured to derive pose estimates from the set of joint candidates that estimate a body configuration for the one or more people depicted in the image; and an action-identification module configured to analyse a region of interest within the image identified from the derived pose estimates using a second computational neural network to identify an action performed by a person depicted in the image.

Abstract: A mechanized store uses a mobile device to authenticate the user. Items removed from one or more displays of the mechanized store by the user are tracked and a list of items removed by the user is updated. The list of items removed is linked with an account of the user.

Abstract: Systems and techniques are provided for determining an improved camera coverage plan including a number, a placement, and a pose of cameras that are arranged to track puts and takes of items by subjects in a three-dimensional real space. The method includes receiving an initial camera coverage plan including a three-dimensional map of a real space, an initial number and initial pose of a plurality of cameras and a camera model including characteristics of the cameras. The method can iteratively apply a machine learning process to an objective function of number and poses of cameras, and subject to a set of constraints, obtain an improved camera coverage plan. The improved camera coverage plan is provided to an installer to arrange cameras to track puts and takes of items by subjects in the three-dimensional real space.

Abstract: Systems and techniques are provided for linking subjects in an area of real space with user accounts. The user accounts are linked with client applications executable on mobile computing devices. A plurality of cameras are disposed above the area. The cameras in the plurality of cameras produce respective sequences of images in corresponding fields of view in the real space. A processing system is coupled to the plurality of cameras. The processing system includes logic to determine locations of subjects represented in the images. The processing system further includes logic to match the identified subjects with user accounts by identifying locations of the mobile computing devices executing client applications in the area of real space and matching locations of the mobile computing devices with locations of the subjects.

Abstract: An apparatus for performing image analysis to identify human actions represented in an image, comprising: a joint-determination module configured to analyse an image depicting one or more people using a first computational neural network to determine a set of joint candidates for the one or more people depicted in the image; a pose-estimation module configured to derive pose estimates from the set of joint candidates that estimate a body configuration for the one or more people depicted in the image; and an action-identification module configured to analyse a region of interest within the image identified from the derived pose estimates using a second computational neural network to identify an action performed by a person depicted in the image.

Abstract: Systems and techniques are provided for detecting gaze direction of subjects in an area of real space. The system receives a plurality of sequences of frames of corresponding fields of view in the real space. The system uses sequences of frames in a plurality of sequences of frames to identify locations of an identified subject and gaze directions of the subject in the area of real space over time. The system includes logic having access to a database identifying locations of items in the area of real space. The system identifies items in the area of real space matching the identified gaze directions of the identified subject.

Abstract: Systems and techniques are provided for linking subjects in an area of real space with user accounts. The user accounts are linked with client applications executable on mobile computing devices. A plurality of cameras are disposed above the area. The cameras in the plurality of cameras produce respective sequences of images in corresponding fields of view in the real space. A processing system is coupled to the plurality of cameras. The processing system includes logic to determine locations of subjects represented in the images. The processing system further includes logic to match the identified subjects with user accounts by identifying locations of the mobile computing devices executing client applications in the area of real space and matching locations of the mobile computing devices with locations of the subjects.

Abstract: Systems and methods for tracking and re-identifying tracked subjects in an area of real space are disclosed. The method includes generating first and second reidentification feature vectors of a first subject identified from a first time interval from the respective first and second sequences of images as obtained from the first time interval. The method includes generating third and fourth reidentification feature vectors of a second subject identified from a second time interval from the respective first and second sequences of images. The method includes matching a second subject identified from a second time interval with the first subject identified from the first time interval when at least one of a first similarity score between the first and the third reidentification feature vectors and a second similarity score between the second and the fourth reidentification feature vectors is above a pre-defined threshold.

Abstract: Systems and methods for detecting a size of an inventory item in an area of real space are disclosed. The method includes receiving respective sequences of images of corresponding fields of view in the area of real space from a plurality of sensors. The method includes determining a two-dimensional plane in the area of real space by matching at least three points on a first image obtained by a first sensor and a second image obtained by a second sensor. The method includes selecting an inventory item, from the first image, for which a size is to be determined. The method includes warping the first image onto the two-dimensional plane. The method includes cropping out the selected inventory item from the warped image. The method includes, inputting, the cropped out image of the inventory item and another image of an inventory item to a trained size determination model.

Abstract: Systems and methods for tracking inventory items in an area of real space are disclosed. The method includes receiving a signal generated in dependence on sensors. The signal indicates a change to a portion of an image of an area of real space. The method includes, in response to receiving the signal, implementing a trained location detection model to determine, based on inputs, whether an inventory item identified in the portion of the image has changed a position in the area of real space. The method includes implementing a trained item classification model to determine a classification of the inventory item. The method includes updating an inventory database with inventory item data determined in dependence on the classification of the inventory item to provide an updated map of the area of real space as a result of the received signal indicating the change to the portion of the image.

Abstract: A system for training a computational neural network to recognise objects and/or actions from images, the system comprising: a training unit, comprising: an input interface configured to receive: a plurality of images captured from one or more cameras, each image having an associated timestamp indicating the time the image was captured; a data stream containing a plurality of timestamps each associated with an object and/or action; the data stream being generated by a system in an operative field of view of the one or more cameras; an image identification unit configured to identify from the plurality of images a set of images that each have a timestamp that correlates to a timestamp associated with an object and/or action from the data stream; a data-labelling unit configured to determine, for each image of the set of images, an image label that indicates the probability the image depicts: (i) an object of each of a set of one or more specified object classes; and/or (ii) a specified human action in dependen

Abstract: Systems and techniques are provided for detecting gaze direction of subjects in an area of real space. The system receives a plurality of sequences of frames of corresponding fields of view in the real space. The system uses sequences of frames in a plurality of sequences of frames to identify locations of an identified subject and gaze directions of the subject in the area of real space over time. The system includes logic having access to a database identifying locations of items in the area of real space. The system identifies items in the area of real space matching the identified gaze directions of the identified subject.

Abstract: Systems and techniques are provided for linking subjects in an area of real space with user accounts. The user accounts are linked with client applications executable on mobile computing devices. A plurality of cameras are disposed above the area. The cameras in the plurality of cameras produce respective sequences of images in corresponding fields of view in the real space. A processing system is coupled to the plurality of cameras. The processing system includes logic to determine locations of subjects represented in the images. The processing system further includes logic to match the identified subjects with user accounts by identifying locations of the mobile computing devices executing client applications in the area of real space and matching locations of the mobile computing devices with locations of the subjects.

Abstract: A mechanized store uses a mobile device to authenticate the user. Items removed from one or more displays of the mechanized store by the user are tracked and a list of items removed by the user is updated. The list of items removed is linked with an account of the user.

Abstract: Systems and techniques are provided for determining an improved camera coverage plan including a number, a placement, and a pose of cameras that are arranged to track puts and takes of items by subjects in a three-dimensional real space. The method includes receiving an initial camera coverage plan including a three-dimensional map of a real space, an initial number and initial pose of a plurality of cameras and a camera model including characteristics of the cameras. The method can iteratively apply a machine learning process to an objective function of number and poses of cameras, and subject to a set of constraints, obtain an improved camera coverage plan. The improved camera coverage plan is provided to an installer to arrange cameras to track puts and takes of items by subjects in the three-dimensional real space.

Abstract: Automated techniques provide for recalibrating cameras in a real space in which puts and takes of items are tracked. The method includes first processing one or more selected images selected from a plurality of sequences of images received from a plurality of cameras calibrated using a set of calibration images that were used to calibrate the cameras previously. The first processing includes a process step to match one or more features from the selected images with features extracted from the set of calibration images using a trained neural network classifier. The features correspond to points located at displays or structures that remain substantially immobile. Camera calibrations can be updated when transform information between features matched meets or exceeds a threshold.

Abstract: Systems and techniques are provided for calibrating cameras in a real space for tracking puts and takes of items by subjects. The method includes first processing one or more selected images captured by an extrinsic calibration tool to extract from the images, a three-dimensional (3D) point cloud of points corresponding to displays or structures that remain substantially immobile in the real space, and a second processing of a second set of images captured by a camera installation in the real space to match points in the second set of images to the point cloud. Differences in position of matching points can be used to determine transformation information therebetween, which can be used to calibrate the installed cameras.

Abstract: A method for tracking inventory items is provided. The method includes using sensors to produce respective sequences of images of corresponding fields of view, the field of view of each sensor overlapping with the field of view of at least one other sensor. The method includes dividing the area of real space into cells having volume and coordinates in a real space, and identifying inventory items and their locations in three dimensions using the sequences of images produced by at least two sensors having overlapping fields of view. The method includes, in response to the identification of the inventory items: (i) matching locations in three dimensions of identified inventory items with cells in the plurality of cells, and (ii) determining, for a particular inventory item matched with a particular cell, whether a count of the particular inventory item is below a threshold for re-stocking the particular inventory item.