Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing visual tracking. In one aspect, a method comprises receiving: (i) one or more reference video frames, (ii) respective reference labels for each of a plurality of reference pixels in the reference video frames, and (iii) a target video frame. The reference video frames and the target video frame are processed using a colorization machine learning model to generate respective pixel similarity measures between each of (i) a plurality of target pixels in the target video frame, and (ii) the reference pixels in the reference video frames. A respective target label is determined for each target pixel in the target video frame, comprising: combining (i) the reference labels for the reference pixels in the reference video frames, and (ii) the pixel similarity measures.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing visual tracking. In one aspect, a method comprises receiving: (i) one or more reference video frames, (ii) respective reference labels for each of a plurality of reference pixels in the reference video frames, and (iii) a target video frame. The reference video frames and the target video frame are processed using a colorization machine learning model to generate respective pixel similarity measures between each of (i) a plurality of target pixels in the target video frame, and (ii) the reference pixels in the reference video frames. A respective target label is determined for each target pixel in the target video frame, comprising: combining (i) the reference labels for the reference pixels in the reference video frames, and (ii) the pixel similarity measures.

Abstract: A system and method for providing a plurality of frames from a video, the video having been taken from a first person's perspective, identifying patterns of attention depicted by living beings appearing in the plurality of frames, identifying social interactions associated with the plurality of frames using the identified patterns of attention over a period of time and using the identified social interactions to affect the subsequent use, storage, presentation, or processing of the plurality of frames.