Abstract: Systems, methods, and computer-readable storage devices are disclosed for improved real-time audio processing. One method including: constructing a deep neural network model, including a plurality of at least one-bit neurons, configured to output a predicted label of audio data, the plurality of at least one-bit neurons arranged in a plurality of layers, including at least one hidden layer, and being connected by a plurality of connections, each connection having at least a one-bit weight, wherein one or both of the plurality of at least one-bit neurons and the plurality of connections have a reduced bit precision; receiving a training data set, the training data set including audio data; training the deep neural network model using the training data set; and outputting a trained deep neural network model configured to output a predicted label of real-time audio data.

Abstract: A method can include classifying, using a compressed and specialized convolutional neural network (CNN), an object of a video frame into classes, clustering the object based on a distance of a feature vector of the object to a feature vector of a centroid object of the cluster, storing top-k classes, a centroid identification, and a cluster identification, in response to receiving a query for objects of class X from a specific video stream, retrieving image data for each centroid of each cluster that includes the class X as one of the top-k classes, classifying, using a ground truth CNN (GT-CNN), the retrieved image data for each centroid, and for each centroid determined to be classified as a member of the class X providing image data for each object in each cluster associated with the centroid.

Abstract: Described herein is a mechanism for creating a dynamic video highlight from a plurality of video segments. A metadata collection agent collects metadata comprising attributes about a video, segments within the video where one or more events occur, attributes about the creator of the video and so forth. The metadata is collected and used to create highlight video definitions comprising a set of metadata attribute-value pairs. The highlight video definitions can be created in an interactive fashion by presenting a user interface allowing selection of a combination of attribute-value pairs to include/exclude segments from the definition and/or manual selection of custom segments. The highlight video definitions can be stored and/or shared among users. The video highlight definitions are utilized to instantiate one or more video players to play the video segments in an identified order without assembling the video segments into a separate video.

Abstract: An imaging sensor system includes a control system that helps save energy, computing, and communication resources by capturing evaluation images with a gating image sensor and processing the evaluation images to determine whether there is a region of interest (ROI) present that warrants capturing a subsequent image with a gated image sensor for additional image processing. That is, the gating image sensor and preliminary processing operates as a gate for turning on the gated sensor for the additional image capturing and processing. The additional image captured by the gated sensor may include only the ROI, which is provided to an image processing algorithm for further image processing. In addition, a gating sensor may include infrared sensing to help determine whether the ROI includes thermal data indicative of human features.

Abstract: Described herein is a mechanism for creating a dynamic video highlight from a plurality of video segments. A metadata collection agent collects metadata comprising attributes about a video, segments within the video where one or more events occur, attributes about the creator of the video and so forth. The metadata is collected and used to create highlight video definitions comprising a set of metadata attribute-value pairs. The highlight video definitions can be created in an interactive fashion by presenting a user interface allowing selection of a combination of attribute-value pairs to include/exclude segments from the definition and/or manual selection of custom segments. The highlight video definitions can be stored and/or shared among users. The video highlight definitions are utilized to instantiate one or more video players to play the video segments in an identified order without assembling the video segments into a separate video.

Abstract: A method can include classifying, using a compressed and specialized convolutional neural network (CNN), an object of a video frame into classes, clustering the object based on a distance of a feature vector of the object to a feature vector of a centroid object of the cluster, storing top-k classes, a centroid identification, and a cluster identification, in response to receiving a query for objects of class X from a specific video stream, retrieving image data for each centroid of each cluster that includes the class X as one of the top-k classes, classifying, using a ground truth CNN (GT-CNN), the retrieved image data for each centroid, and for each centroid determined to be classified as a member of the class X providing image data for each object in each cluster associated with the centroid.

Abstract: A method can include classifying, using a compressed and specialized convolutional neural network (CNN), an object of a video frame into classes, clustering the object based on a distance of a feature vector of the object to a feature vector of a centroid object of the cluster, storing top-k classes, a centroid identification, and a cluster identification, in response to receiving a query for objects of class X from a specific video stream, retrieving image data for each centroid of each cluster that includes the class X as one of the top-k classes, classifying, using a ground truth CNN (GT-CNN), the retrieved image data for each centroid, and for each centroid determined to be classified as a member of the class X providing image data for each object in each cluster associated with the centroid.

Abstract: A method to record video with a video camera while respecting bystander privacy includes acquiring sensory data separate from the video, parsing the sensory data for evidence of a human being in a field of view of the video camera, and recording video with the video camera if no human being is detected in the field of view, based upon the sensory data.

Abstract: A classification system classifies different aspects of content of an input image stream, such as faces, landmarks, events, and so forth. The classification system includes a general classifier and at least one specialized classifier template. The general classifier is trained to classify a large number of different aspects of content, and a specialized classifier can be trained based on a specialized classifier template during operation of the classification system to classify a particular subset of the multiple different aspects of content. The classification system determines when to use the general classifier and when to use a specialized classifier based on class skew, which refers to the temporal locality of a subset of aspects of content in the image stream.

Abstract: Systems, methods, and computer-readable storage devices are disclosed for improved real-time audio processing. One method including: constructing a deep neural network model, including a plurality of at least one-bit neurons, configured to output a predicted label of audio data, the plurality of at least one-bit neurons arranged in a plurality of layers, including at least one hidden layer, and being connected by a plurality of connections, each connection having at least a one-bit weight, wherein one or both of the plurality of at least one-bit neurons and the plurality of connections have a reduced bit precision; receiving a training data set, the training data set including audio data; training the deep neural network model using the training data set; and outputting a trained deep neural network model configured to output a predicted label of real-time audio data.

Abstract: Wearable audio accessories for computing devices are described. In one embodiment the wearable audio accessory provides a speech based interface between the user and a nearby computing device for the performance of user-initiated or computing device initiated microtasks. Information is provided to the user via a loudspeaker and the user can provide input via a microphone. An audio sensing channel within the accessory continuously monitors the audio signal as detected by the microphone and in various embodiments will trigger more complex audio processing based on this monitoring. A wireless communication link is provided between the accessory and the nearby computing device. To mitigate any delay caused by the switching between audio processing techniques, the audio accessory may include a rolling buffer which continuously stores the audio signal and outputs a delayed audio signal to the audio processing engines.

Abstract: A method can include classifying, using a compressed and specialized convolutional neural network (CNN), an object of a video frame into classes, clustering the object based on a distance of a feature vector of the object to a feature vector of a centroid object of the cluster, storing top-k classes, a centroid identification, and a cluster identification, in response to receiving a query for objects of class X from a specific video stream, retrieving image data for each centroid of each cluster that includes the class X as one of the top-k classes, classifying, using a ground truth CNN (GT-CNN), the retrieved image data for each centroid, and for each centroid determined to be classified as a member of the class X providing image data for each object in each cluster associated with the centroid.

Abstract: Wearable audio accessories for computing devices are described. In one embodiment the wearable audio accessory provides a speech based interface between the user and a nearby computing device for the performance of user-initiated or computing device initiated microtasks. Information is provided to the user via a loudspeaker and the user can provide input via a microphone. An audio sensing channel within the accessory continuously monitors the audio signal as detected by the microphone and in various embodiments will trigger more complex audio processing based on this monitoring. A wireless communication link is provided between the accessory and the nearby computing device. To mitigate any delay caused by the switching between audio processing techniques, the audio accessory may include a rolling buffer which continuously stores the audio signal and outputs a delayed audio signal to the audio processing engines.

Abstract: A global manager communicates with various local managers to receive and process video queries. The video queries identify components that process live video streams, placement options for where the components of the video query may be executed, and various video query plans. The video query plans include options such as framerate and video quality. As the global manager processes the video queries, the global manager determines an initial set of video query configurations that identify a video query plan and placement option for each component of a given video query. Using the initial set of video query configurations, the global manager then determines an optimal set of video query configurations for the received set of video queries. The global manager communications instructions to the local managers to execute the components of the video queries using the video query plans and placement options from the optimal set of video query configurations.

Abstract: Various technologies described herein pertain to performing video analytics. The approaches set forth herein support live video analytics at scale with approximate and delay-tolerant processing. Video streams can be captured by multiple cameras and continuously streamed to a video analytics computing system; the video streams can be received at the video analytics computing system. Multiple video analytics queries can be executed on the video streams. The multiple video analytics queries can be concurrently executed by the video analytics computing system on the video streams as the video streams are continuously streamed to the video analytics computing system. The multiple video analytics queries can be executed utilizing resources of the video analytics computing system allocated between the multiple video analytics queries. Execution of the multiple video analytics queries can return respective results for the multiple video analytics queries.

Abstract: A global manager communicates with various local managers to receive and process video queries. The video queries identify components that process live video streams, placement options for where the components of the video query may be executed, and various video query plans. The video query plans include options such as framerate and video quality. As the global manager processes the video queries, the global manager determines an initial set of video query configurations that identify a video query plan and placement option for each component of a given video query. Using the initial set of video query configurations, the global manager then determines an optimal set of video query configurations for the received set of video queries. The global manager communications instructions to the local managers to execute the components of the video queries using the video query plans and placement options from the optimal set of video query configurations.

Abstract: A classification system classifies different aspects of content of an input image stream, such as faces, landmarks, events, and so forth. The classification system includes a general classifier and at least one specialized classifier template. The general classifier is trained to classify a large number of different aspects of content, and a specialized classifier can be trained based on a specialized classifier template during operation of the classification system to classify a particular subset of the multiple different aspects of content. The classification system determines when to use the general classifier and when to use a specialized classifier based on class skew, which refers to the temporal locality of a subset of aspects of content in the image stream.

Abstract: Wearable audio accessories for computing devices are described. In one embodiment the wearable audio accessory provides a speech based interface between the user and a nearby computing device for the performance of user-initiated or computing device initiated microtasks. Information is provided to the user via a loudspeaker and the user can provide input via a microphone. An audio sensing channel within the accessory continuously monitors the audio signal as detected by the microphone and in various embodiments will trigger more complex audio processing based on this monitoring. A wireless communication link is provided between the accessory and the nearby computing device. To mitigate any delay caused by the switching between audio processing techniques, the audio accessory may include a rolling buffer which continuously stores the audio signal and outputs a delayed audio signal to the audio processing engines.

Abstract: A classification system classifies different aspects of content of an input image stream, such as faces, landmarks, events, and so forth. The classification system includes a general classifier and at least one specialized classifier template. The general classifier is trained to classify a large number of different aspects of content, and a specialized classifier can be trained based on a specialized classifier template during operation of the classification system to classify a particular subset of the multiple different aspects of content. The classification system determines when to use the general classifier and when to use a specialized classifier based on class skew, which refers to the temporal locality of a subset of aspects of content in the image stream.

Abstract: Various technologies described herein pertain to performing video analytics. The approaches set forth herein support live video analytics at scale with approximate and delay-tolerant processing. Video streams can be captured by multiple cameras and continuously streamed to a video analytics computing system; the video streams can be received at the video analytics computing system. Multiple video analytics queries can be executed on the video streams. The multiple video analytics queries can be concurrently executed by the video analytics computing system on the video streams as the video streams are continuously streamed to the video analytics computing system. The multiple video analytics queries can be executed utilizing resources of the video analytics computing system allocated between the multiple video analytics queries. Execution of the multiple video analytics queries can return respective results for the multiple video analytics queries.