Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using dual descriptor data. One of the methods includes: detecting, using a first set of descriptor features included in dual descriptor data, a first representation within first image data collected by a camera; determining a change to an imaging modality of the camera; detecting, using a second set of features included in the dual descriptor data, a second representation within second image data collected by the camera; classifying the first representation and the second representation as associated with a same object using the dual descriptor data; and in response to classifying the first representation and the second representation as associated with the same object using the dual descriptor data, transmitting operational instructions to one or more appliances connected to the system.

Abstract: Disclosed are methods, systems, and apparatus for object localization in video. A method includes obtaining a reference image of an object; generating, from the reference image, homographic adapted images showing the object at various locations with various orientations; determining interest points from the homographic adapted images; determining locations of an object center in the homographic adapted images relative to the interest points; obtaining a sample image of the object; identifying matched pairs of interest points, each matched pair including an interest point from the homographic adapted images and a matching interest point in the sample image; and determining a location of the object in the sample image based on the locations of the object center in the homographic adapted images relative to the matched pairs. The method includes generating a homography matrix; and projecting the reference image of the object to the sample image using the homography matrix.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for monitoring events using a Virtual Inductive Loop system. In some implementations, image data is obtained from cameras. A region depicted in the obtained image data is identified, the region comprising lines spaced by a distance that satisfies a distance threshold. For each line included in the region: an object depicted crossing the line is determined whether to satisfy a height criteria indicating that the line is activated. In response to determining that an object depicted crossing the line satisfies the height criteria, an event is determined to have likely occurred using data indicating (i) which lines of the lines were activated and (ii) an order in which each of the lines were activated. In response to determining that an event likely occurred, actions are performed using at least some of the data.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for reducing camera false detections. One of the methods includes providing, to a neural network of an image classifier that is trained to detect objects of two or more classification types, a feature vector for a respective training image; receiving, from the neural network, an output vector that indicates, for each of the two or more classification types, a likelihood that the respective training image depicts an object of the corresponding classification type; accessing, from two or more ground truth vectors each for one of the two or more classification types, a ground truth vector for the classification type of an object depicted in the training image; and adjusting one or more weights in the neural network using the output vector and the ground truth vector; and storing, in a memory, the image classifier.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for retroactive event detection. The methods, systems, and apparatus include actions of obtaining an image captured by a camera at a current time, determining that the image depicts a change in a region from a previous image captured by the camera at a previous time, determining, based on determining that the image depicts the change in the region, whether the change depicted in the image is of a known object type, determining, based on the determination that the change depicted in the image is of a known object type, whether the change does not correspond to a previously detected event, and determining, based on the determination that the change does not correspond to a previously detected event, whether the images captured by the camera between the current time and the previous time depict an event.

Abstract: Disclosed are methods, systems, and apparatus for object localization in video. A method includes obtaining a reference image of an object; generating, from the reference image, homographic adapted images showing the object at various locations with various orientations; determining interest points from the homographic adapted images; determining locations of an object center in the homographic adapted images relative to the interest points; obtaining a sample image of the object; identifying matched pairs of interest points, each matched pair including an interest point from the homographic adapted images and a matching interest point in the sample image; and determining a location of the object in the sample image based on the locations of the object center in the homographic adapted images relative to the matched pairs. The method includes generating a homography matrix; and projecting the reference image of the object to the sample image using the homography matrix.

Abstract: Methods and systems, including computer programs encoded on a storage medium, are described for implementing item monitoring using a doorbell camera. A system generates an input video stream that has image frames corresponding to detection of activity at a property. Timing information is generated for the video stream and includes a timestamp for each image frame of the stream. Using the timing information, the system processes a pre-event image frame that precedes detection of the activity and a post-event image frame that coincides with detection of the activity. An image score is computed with respect to placement of a candidate item at the property in response to processing the pre-event and post-event image frames. The image score is used to determine that a first item was delivered to the property or that a second item was removed after being delivered to the property.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training an event detector. The methods, systems, and apparatus include actions of identifying a portion of a first interframe difference image that represents motion of an OI, determining that a second interframe difference image represents motion by a non-OI, combining the portion of the first interframe difference image and the second interframe difference image as a third interframe difference image labeled as motion of both an OI and a non-OI, and training an event detector with the third interframe difference image.

Abstract: Methods, systems, an apparatus, including computer programs encoded on a storage device, for training an image classifier. A method includes receiving an image that includes a depiction of an object; generating a set of poorly localized bounding boxes; and generating a set of accurately localized bounding boxes. The method includes training, at a first learning rate and using the poorly localized bounding boxes, an object classifier to classify the object; and training, at a second learning rate that is lower than the first learning rate, and using the accurately localized bounding boxes, the object classifier to classify the object. The method includes receiving a second image that includes a depiction of an object; and providing, to the trained object classifier, the second image. The method includes receiving an indication that the object classifier classified the object in the second image; and performing one or more actions.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using feature descriptors to track objects depicted in images. One of the methods includes receiving hue, saturation, value data for an image and data that indicates an object detected in the image, generating a feature descriptor that includes hue data and saturation data, determining, for each of two or more tracked objects that each have a historical feature descriptor that includes historical hue data and historical saturation data, a distance between (i) the respective historical feature descriptor and (ii) the feature descriptor, associating the feature descriptor for the object with a tracked object from the two or more tracked objects, and tracking the tracked object in one or more images from a video sequence using the feature descriptor and the historical feature descriptor.

Abstract: Methods, systems, and apparatus for motion-based human video detection are disclosed. A method includes generating a representation of a difference between two frames of a video; providing, to an object detector, a particular frame of the two frames and the representation of the difference between two frames of the video; receiving an indication that the object detector detected an object in the particular frame; determining that detection of the object in the particular frame was a false positive detection; determining an amount of motion energy where the object was detected in the particular frame; and training the object detector based on penalization of the false positive detection in accordance with the amount of motion energy where the object was detected in the particular frame.

Abstract: Methods, systems, an apparatus, including computer programs encoded on a storage device, for tracking human movement in video images. A method includes obtaining a first image of a scene captured by a camera; identifying a bounding box around a human detected in the first image; determining a scale amount that corresponds to a size of the bounding box; obtaining a second image of the scene captured by the camera after the first image was captured; and detecting the human in the second image based on both the first image scaled by the scale amount and the second image scaled by the scale amount. Detecting the human in the second image can include identifying a second scaled bounding box around the human detected in the second image scaled by the scale amount.

Abstract: Methods and systems, including computer programs encoded on a storage medium, are described for implementing item monitoring using a doorbell camera. A system generates an input video stream that has image frames corresponding to detection of activity at a property. Timing information is generated for the video stream and includes a timestamp for each image frame of the stream. Using the timing information, the system processes a pre-event image frame that precedes detection of the activity and a post-event image frame that coincides with detection of the activity. An image score is computed with respect to placement of a candidate item at the property in response to processing the pre-event and post-event image frames. The image score is used to determine that a first item was delivered to the property or that a second item was removed after being delivered to the property.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for tracking objects of interest using distance-based thresholding. One of the methods includes detecting an object depicted in an image captured by a camera, determining a predicted physical distance between the object and the camera, selecting, from a plurality of predetermined confidence thresholds, a confidence threshold for the predicted physical distance, each confidence threshold in the plurality of predetermined confidence thresholds for a different physical distance range, the confidence threshold having a physical distance range that includes the predicted physical distance, and determining, using the confidence threshold and a confidence score that indicates a likelihood that the object is an object of interest, that the object is likely an object of interest.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using dual descriptor data. One of the methods includes: detecting, using a first set of descriptor features included in dual descriptor data, a first representation within first image data collected by a camera; determining a change to an imaging modality of the camera; detecting, using a second set of features included in the dual descriptor data, a second representation within second image data collected by the camera; classifying the first representation and the second representation as associated with a same object using the dual descriptor data; and in response to classifying the first representation and the second representation as associated with the same object using the dual descriptor data, transmitting operational instructions to one or more appliances connected to the system.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for tracking moving objects depicted in multiple images. One of the methods includes determining, for an image captured by a camera, a first bounding box that represents a first moving object depicted in the image, determining that the first bounding box and a second bounding box overlap in an overlap area, determining that the first moving object represented by the first bounding box was farther from the camera that captured the image than a second moving object represented by the second bounding box, generating a mask for the first bounding box based on the overlap area, and determining, using data from the image that is associated with the mask, that the first moving object matches an appearance of another moving object depicted in another image captured by the camera.

Abstract: Methods, systems, an apparatus, including computer programs encoded on a storage device, for training an image classifier. A method includes receiving an image that includes a depiction of an object; generating a set of poorly localized bounding boxes; and generating a set of accurately localized bounding boxes. The method includes training, at a first learning rate and using the poorly localized bounding boxes, an object classifier to classify the object; and training, at a second learning rate that is lower than the first learning rate, and using the accurately localized bounding boxes, the object classifier to classify the object. The method includes receiving a second image that includes a depiction of an object; and providing, to the trained object classifier, the second image. The method includes receiving an indication that the object classifier classified the object in the second image; and performing one or more actions.

Abstract: Methods, systems, and apparatus for motion-based human video detection are disclosed. A method includes generating a representation of a difference between two frames of a video; providing, to an object detector, a particular frame of the two frames and the representation of the difference between two frames of the video; receiving an indication that the object detector detected an object in the particular frame; determining that detection of the object in the particular frame was a false positive detection; determining an amount of motion energy where the object was detected in the particular frame; and training the object detector based on penalization of the false positive detection in accordance with the amount of motion energy where the object was detected in the particular frame.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for monitoring events using a Virtual Inductive Loop system. In some implementations, image data is obtained from cameras. A region depicted in the obtained image data is identified, the region comprising lines spaced by a distance that satisfies a distance threshold. For each line included in the region: an object depicted crossing the line is determined whether to satisfy a height criteria indicating that the line is activated. In response to determining that an object depicted crossing the line satisfies the height criteria, an event is determined to have likely occurred using data indicating (i) which lines of the lines were activated and (ii) an order in which each of the lines were activated. In response to determining that an event likely occurred, actions are performed using at least some of the data.

Abstract: Methods, systems, an apparatus, including computer programs encoded on a storage device, for training an image classifier. A method includes receiving an image that includes a depiction of an object; generating a set of poorly localized bounding boxes; and generating a set of accurately localized bounding boxes. The method includes training, at a first learning rate and using the poorly localized bounding boxes, an object classifier to classify the object; and training, at a second learning rate that is lower than the first learning rate, and using the accurately localized bounding boxes, the object classifier to classify the object. The method includes receiving a second image that includes a depiction of an object; and providing, to the trained object classifier, the second image. The method includes receiving an indication that the object classifier classified the object in the second image; and performing one or more actions.