Abstract: Described is a multiple-camera system and process for detecting, tracking, and re-verifying agents within a materials handling facility. In one implementation, a plurality of feature vectors may be generated for an agent and maintained as an agent model representative of the agent. When the object being tracked as the agent is to be re-verified, feature vectors representative of the object are generated and stored as a probe agent model. Feature vectors of the probe agent model are compared with corresponding feature vectors of candidate agent models for agents located in the materials handling facility. Based on the similarity scores, the agent may be re-verified, it may be determined that identifiers used for objects tracked as representative of the agents have been flipped, and/or to determine that tracking of the object representing the agent has been dropped.

Abstract: Commercial interactions with non-discretized items such as liquids in carafes or other dispensers are detected and associated with actors using images captured by one or more digital cameras including the carafes or dispensers within their fields of view. The images are processed to detect body parts of actors and other aspects therein, and to not only determine that a commercial interaction has occurred but also identify an actor that performed the commercial interaction. Based on information or data determined from such images, movements of body parts associated with raising, lowering or rotating one or more carafes or other dispensers may be detected, and a commercial interaction involving such carafes or dispensers may be detected and associated with a specific actor accordingly.

Abstract: Where an event is determined to have occurred at a location within a vicinity of a plurality of actors, imaging data captured using cameras having the location is processed using one or more machine learning systems or techniques operating on the cameras to determine which of the actors is most likely associated with the event. For each relevant pixel of each image captured by a camera, the camera returns a set of vectors extending to pixels of body parts of actors who are most likely to have been involved with an event occurring at the relevant pixel, along with a measure of confidence in the respective vectors. A server receives the vectors from the cameras, determines which of the images depicted the event in a favorable view, based at least in part on the quality of such images, and selects one of the actors as associated with the event accordingly.

Abstract: An identification system includes one or more infrared light sources and a camera that acquires images of a user's palm. For example, at a first time, one or more first images may be acquired by the camera using infrared light with a first polarization that represent external characteristics of the user's palm. At a second time, one or more second images may be acquired using infrared light with a second polarization that represent internal characteristics of the user's palm. These images are processed to determine a first set of feature vectors and a second set of feature vectors. A current signature may be determined using the first set of feature vectors and the second set of feature vectors. In addition, a user may be identified based on a comparison of the current signature and previously stored reference signatures that are associated with candidate user identifiers.

Abstract: Described is a multiple-camera system and process for re-identifying an agent located in a materials handling facility based on anterior views of agents. An anterior view of a newly detected agent may be partitioned and color signatures generated for each partition. Likewise, stored anterior views of agents (candidate agents) that may potentially be the newly detected agent are partitioned and color signatures generated for each partition. Based on the color signatures, a similarity between the anterior view of the newly detected agent and the candidate agents is determined. The similarity may be used to either determine that the newly detected agent is one of the candidate agents or reduce the set of candidate agents that are considered during a manual review.

Abstract: A neural network is trained to process input data and generate a classification value that characterizes the input with respect to an ordered continuum of classes. For example, the input data may comprise an image and the classification value may be indicative of a quality of the image. The ordered continuum of classes may represent classes of quality of the image ranging from “worst”, “bad”, “normal”, “good”, to “best”. During training, loss values are determined using an ordered classification loss function. The ordered classification loss function maintains monotonicity in the loss values that corresponds to placement in the continuum. For example, the classification value for a “bad” image will be less than the classification value indicative of a “best” image. The classification value may be used for subsequent processing. For example, biometric input data may be required to have a minimum classification value for further processing.

Abstract: Actors may be detected and tracked within a scene using multiple imaging devices provided in a network that are aligned with fields of view that overlap at least in part. Processors operating on the imaging devices may evaluate the images using one or more classifiers to recognize body parts within the images, and to associate the body parts with a common actor within the scene. Each of the imaging devices may generate records of the positions of the body parts and provide such records to a central server, that may correlate body parts appearing within images captured by two or more of the imaging devices and generate a three-dimensional model of an actor based on positions of the body parts. Motion of the body parts may be tracked in subsequent images, and the model of the actor may be updated based on the motion.

Abstract: A person may attempt to gain access to a facility via transaction data, such as images of a hand of the person or other identifying information as acquired by an input device. Possible fraud may be detected by comparing the transaction data with previously stored exclusion data. The exclusion data may include known bad data or synthetic trained data for detecting possible fraud. If the biometric input matches or is similar to the exclusion data, possible fraud is detected and the person is prompted for additional data. The reply data acquired from the person is compared with the exclusion data to determine if possible fraud is still detected. If so, additional prompts are presented to the person until the reply data provides enough confidence of no fraud or until the transaction is terminated.

Abstract: An event at a shelf, such as a user interacting with items on the shelf, may be detected using sensors associated with the shelf. Hypotheses to describe the event are generated based on the collected sensor data. However, hypotheses determined based on only one type of sensor data may not have a confidence value that is sufficiently reliable. Hypotheses derived from different sensor types may be combined to generate new hypotheses that have higher confidence values. For example, hypotheses based on image data may be combined with hypotheses based on weight data to produce hypotheses with higher confidence values that are sufficiently reliable. A hypothesis, from the set of combined hypotheses, having the highest confidence value may then be used to describe the interaction at the shelf, such as identifying the item and quantity of item the user interacted with during the event.

Abstract: Biometric input, such as an image of a hand, may be processed to determine embedding vector data that may be used to identify users. Accuracy of the identification is improved by using high resolution inputs to a deep convolutional neural network (DCNN) that is trained to generate the embedding vector data that is representative of features in the input. Training data sets are expensive to develop and thus may be relatively small. During training of the DCNN, confidence loss values corresponding to the entire input as well as particular patches or portions of the input are determined. These patch-wise confidence loss values mitigate potential overfitting during training of the DCNN and improve overall performance of the trained DCNN to determine embedding vector data suitable for identification.

Abstract: This disclosure describes techniques for identifying users that are enrolled for use of a user-recognition system and updating identification data of these users over time. To enroll in the user-recognition system, the user may initially scan his or her palm. The resulting image data may later be used when the user requests to be identified by the system by again scanning his or her palm. However, because the characteristics of user palms may change over the time, the user-recognition system may periodically perform processes for updating the identification data stored in association with the user in order to maintain or increase an accuracy of the user-recognition system.

Abstract: This disclosure describes techniques for updating planogram data associated with a facility. The planogram may indicate inventory locations within the facility for various types of items supported by product fixtures. In particular an image of a product fixture is analyzed to identify image segments corresponding to product groups, where each product group consists of instances of the same product and each image segment corresponds to a group of image points. Image data is further analyzed to determine coordinates of the points of each image segment. A product space corresponding to the product group is then defined based on the coordinates of the points of the product group. In some cases, for example, a product space may be defined in terms of the coordinates of the corners of a rectangular bounding box or volume.

Abstract: A pair of input images acquired using a first modality and a second modality is processed using a multi-classifier trained to determine classification data indicative of whether the pair is normal or abnormal. A pair may be deemed abnormal if one or both input images are obscured or inconsistent with one another. Training data comprising normal and abnormal images are used to train the multi-classifier. During training, the multi-classifier uses an objective function that includes cross entropy loss, distance loss, and discrepancy loss to process the training data. During use, the trained multi-classifier processes a pair of input images. If the resulting classification data indicates the pair of input images are normal, the pair of input images may be processed to assert an identity.

Abstract: Techniques for an identity-verification system to analyze image data representing palms of users using a segmented, characteristic-based approach. The system may compare palm-feature data representing characteristics of a palm of a user (or “query palm”) with stored palm-feature data of palms for user profiles (or “stored palms”). For instance, the system may identify characteristics of the query palm having salient or discriminative features, and compare palm-feature data for those discriminative characteristics to palm-feature data representing corresponding characteristics of stored palms of enrolled users. Additionally, the system may compare characteristics of the query palm with corresponding characteristics of stored palms until the system is confident that the query palm corresponds to a stored palm of a user profile.

Abstract: Images of a hand are obtained by a camera. These images may depict the fingers and palm of the user. A pose of the hand relative to the camera may vary due to rotation, translation, articulation of joints in the hand, and so forth. One or more canonical images are generated by mapping the images to a canonical model. A first embedding model is used to determine a first embedding vector representative of the palm as depicted in the canonical images. A second embedding model is used to determine a set of second embedding vectors, each representative of individual fingers as depicted in the canonical images. Embedding distances in the embedding space from the embedding vectors to a closest match of previously stored embedding vectors are multiplied together to determine an overall distance. If the overall distance is less than a threshold value, an identity of a user is asserted.

Abstract: A machine learning system to determine an identity of a user is trained using triplets of ad hoc synthetic data and actual data. The data may comprise multimodal images of a hand. Each triplet comprises an anchor, a positive, and a negative image. Synthetic triplets for different synthesized identities are generated on an ad hoc basis and provided as input during training of the machine learning system. The machine learning system uses a pairwise label-based loss function, such as a triplet loss function during training. Synthetic triplets may be generated to provide more challenging training data, to provide training data for categories that are underrepresented in the actual data, and so forth. The system uses substantially less memory during training, and the synthetic triplets need not be retained further reducing memory use. Ongoing training is supported as new actual triplets become available, and may be supplemented by additional synthetic triplets.

Abstract: A biometric identification system uses inputs acquired using different modalities. A model having an intersection branch and an XOR branch is trained to determine an embedding using features present in all modalities (an intersection of modalities), and features that are distinctive to each modality (an XOR of that modality relative to the other modality(s)). During training, a first loss function is used to determine a first loss value with respect to the branches. Probability distributions are determined for the output from the branches, corresponding to the intersection and XORs of each modality. A second loss function uses these probability distributions to determine a second loss value. A total loss function for training the model may be a sum of the first loss and the second loss. Once trained, the model may process query inputs to determine embedding data for comparison with embedding data of a previously enrolled user.

Abstract: A scanner acquires a set of images of a hand of a user to facilitate identification. These images may vary, due to changes in relative position, pose, lighting, obscuring objects such as a sleeve, and so forth. A first neural network determines output data comprising a spatial mask and a feature map for individual images in the set. The output data for two or more images is combined to provide aggregate data that is representative of the two or more images. The aggregate data may then be processed using a second neural network, such as convolutional neural network, to determine an embedding vector. The embedding vector may be stored and associated with a user account. At a later time, images acquired from the scanner may be processed to produce an embedding vector that is compared to the stored embedding vector to identify a user at the scanner.

Abstract: This disclosure describes a user-recognition system that may perform one or more verification methods upon identifying a previous image that matches a current image of a palm of a user. For instance, the user-recognition system may perform the verification method(s) as part of the recognition method (e.g., after recognizing a matching image), in response to an audit process, in response to a request to re-analyze the image data (e.g., because a user indicates that he or she was not associated with a particular purchase or shopping session), and/or the like.

Abstract: Described are systems and methods for training machine learning models of an ensemble of models that are de-correlated. For example, two or more machine learning models may be concurrently trained (e.g., co-trained) while adding a decorrelation component to one or both models that decreases the pairwise correlation between the outputs of the models. Unlike traditional approaches, in accordance with the disclosed implementations, only the negative results need to be decorrelated.