Abstract: Apparatus and methods are disclosed for using machine learning models with private and public domains. Operations can be applied to transform input to a machine learning model in a private domain that is kept secret or otherwise made unavailable to third parties. In one example of the disclosed technology, a method includes applying a private transform to produce transformed input, providing the transformed input to a machine learning model that was trained using a training set modified by the private transform, and generating inferences with the machine learning model using the transformed input. Examples of suitable transforms that can be employed include matrix multiplication, time or spatial domain to frequency domains, and partitioning a neural network model such that an input and at least one hidden layer form part of the private domain, while the remaining layers form part of the public domain.

Abstract: Systems and methods may be used for access control. These systems and methods may include using a data processing system to access a video stream, the video stream including an image including a virtual background, segmenting the image into a foreground portion and a background portion to determine whether the foreground portion or the background portion of the image meets a threshold requirement, and outputting an alert in response to determining that the foreground portion or the background portion of the image fails to meet the threshold requirement.

Abstract: Systems and methods may be used for access control. These systems and methods may include using a data processing system to identify a video stream, determine an identification of a source type of the video stream, determine whether the video stream is live or pre-recorded based on a determined type of the source type, and output an alert to a device in response to determining that the source type indicates the video stream is pre-recorded.

Abstract: Apparatus and methods are disclosed for using machine learning models with private and public domains. Operations can be applied to transform input to a machine learning model in a private domain that is kept secret or otherwise made unavailable to third parties. In one example of the disclosed technology, a method includes applying a private transform to produce transformed input, providing the transformed input to a machine learning model that was trained using a training set modified by the private transform, and generating inferences with the machine learning model using the transformed input. Examples of suitable transforms that can be employed include matrix multiplication, time or spatial domain to frequency domains, and partitioning a neural network model such that an input and at least one hidden layer form part of the private domain, while the remaining layers form part of the public domain.

Abstract: An agent interfaces with a sending computing device and a receiving computing device to automatically adjust one-way or two-way real-time audio and real-time video transmission parameters responsive to changing network conditions and/or application requirements. The agent incorporates a reinforcement learning model that adjusts transmission parameters to maximize an expected value of a sum of future rewards; the expected value of the sum of future rewards is based on a current state of the sending computing, a current action (e.g. a current set of transmission parameters) at the sending computing device and a reward provided by the receiving computing device. The reward is representative of a user-perceived quality of experience at the receiving computing device.

Abstract: Techniques are described for efficiently embedding frame masks in a video stream. In some solutions, a computer implemented method includes operations for encoding a frame of video data comprising an array of pixels to generate an encoded video frame, and transmitting the encoded video frame to a video decoder. The array of pixels include foreground pixels and background pixels. The foreground pixels have respective first chroma component values which are bounded within a first chroma component range. Encoding the frame of video data can include converting the first chroma component values of the foreground pixels to second chroma component values by applying a mapping function. The second chroma component values are bounded within a second chroma component range. The second chroma component range is shifted or compressed from the first chroma component range.

Abstract: Techniques are described for efficiently embedding frame masks in a video stream. In some solutions, a computer implemented method includes operations for encoding a frame of video data comprising an array of pixels to generate an encoded video frame and transmitting the encoded video frame. The array of pixels can include foreground pixels and background pixels. The foreground pixels can have respective original luma values which are bounded within a first luma range. In certain examples, encoding the frame of video data can include converting the original luma values of the foreground pixels to updated luma values which are bounded within a second luma range. The second luma range can be shifted and/or compressed from the first luma range.

Abstract: A computer implemented method can decode a frame of video data comprising an array of pixels to obtain decoded luma values and decoded chroma values corresponding to the array of pixels, and extract a frame mask based on the decoded luma values. The frame mask can include an array of mask values respectively corresponding to the array of pixels. A mask value indicates whether a corresponding pixel is in foreground or background of the frame. The method can perform a morphological operation to the frame mask to change one or more mask values to indicate their corresponding pixels are removed from the foreground and added to the background of the frame. The method can also identify foreground pixels after performing the morphological operation to the frame mask, and render a foreground image for display based on the decoded luma values and decoded chroma values of the foreground pixels.

Abstract: A computer implemented method can decode a frame of video data comprising an array of pixels to obtain decoded luma values and decoded chroma values corresponding to the array of pixels, and extract a frame mask based on the decoded luma values. The frame mask can include an array of mask values respectively corresponding to the array of pixels. A mask value indicates whether a corresponding pixel is in foreground or background of the frame. The method can perform a morphological operation to the frame mask to change one or more mask values to indicate their corresponding pixels are removed from the foreground and added to the background of the frame. The method can also identify foreground pixels after performing the morphological operation to the frame mask, and render a foreground image for display based on the decoded luma values and decoded chroma values of the foreground pixels.

Abstract: Techniques are described for efficiently embedding frame masks in a video stream. In some solutions, a computer implemented method includes operations for encoding a frame of video data comprising an array of pixels to generate an encoded video frame and transmitting the encoded video frame. The array of pixels can include foreground pixels and background pixels. The foreground pixels can have respective original luma values which are bounded within a first luma range. In certain examples, encoding the frame of video data can include converting the original luma values of the foreground pixels to updated luma values which are bounded within a second luma range. The second luma range can be shifted and/or compressed from the first luma range.

Abstract: An agent interfaces with a sending computing device and a receiving computing device to automatically adjust one-way or two-way real-time audio and real-time video transmission parameters responsive to changing network conditions and/or application requirements. The agent incorporates a reinforcement learning model that adjusts transmission parameters to maximize an expected value of a sum of future rewards; the expected value of the sum of future rewards is based on a current state of the sending computing, a current action (e.g. a current set of transmission parameters) at the sending computing device and a reward provided by the receiving computing device. The reward is representative of a user-perceived quality of experience at the receiving computing device.

Abstract: An agent interfaces with a sending computing device and a receiving computing device to automatically adjust one-way or two-way real-time audio and real-time video transmission parameters responsive to changing network conditions and/or application requirements. The agent incorporates a reinforcement learning model that adjusts transmission parameters to maximize an expected value of a sum of future rewards; the expected value of the sum of future rewards is based on a current state of the sending computing, a current action (e.g. a current set of transmission parameters) at the sending computing device and a reward provided by the receiving computing device. The reward is representative of a user-perceived quality of experience at the receiving computing device.

Abstract: Apparatus and methods are disclosed for using machine learning models with private and public domains. Operations can be applied to transform input to a machine learning model in a private domain that is kept secret or otherwise made unavailable to third parties. In one example of the disclosed technology, a method includes applying a private transform to produce transformed input, providing the transformed input to a machine learning model that was trained using a training set modified by the private transform, and generating inferences with the machine learning model using the transformed input. Examples of suitable transforms that can be employed include matrix multiplication, time or spatial domain to frequency domains, and partitioning a neural network model such that an input and at least one hidden layer form part of the private domain, while the remaining layers form part of the public domain.

Abstract: An agent interfaces with a sending computing device and a receiving computing device to automatically adjust one-way or two-way real-time audio and real-time video transmission parameters responsive to changing network conditions and/or application requirements. The agent incorporates a reinforcement learning model that adjusts transmission parameters to maximize an expected value of a sum of future rewards; the expected value of the sum of future rewards is based on a current state of the sending computing, a current action (e.g. a current set of transmission parameters) at the sending computing device and a reward provided by the receiving computing device. The reward is representative of a user-perceived quality of experience at the receiving computing device.

Abstract: Multiple data streams are transmitted from a transmitting device via an end-to-end or process-to-process channel to a receiving device. At the transmitting device a media data stream is generated. A maximum packet size for another data stream is determined based on a determined end-to-end or process-to-process bandwidth and using playout information pertaining to the media data, which conveys a desired playout rate of the media data. At the transmitting device, based on the determined maximum packet size, a plurality of data packets of the other data stream is generated, each having a packet size no more than the determined maximum. The media and other data streams are multiplexed onto the channel at the transmitting device, thereby transmitting both of the data streams from the transmitting device to the receiving device via the same channel.

Abstract: A method and system for providing an enhanced video stream includes receiving a video stream captured by a camera positioned to include a target in a camera field of view and entering a detection mode to detect the target in the received video stream, before detecting the target in the video stream. Upon detecting the target in the video stream, the method automatically switches from the detection mode to an enhancement mode configured to process the video stream to obtain an enhanced video stream of the detected target. The enhanced video stream is then presented for display on a display device, while a position of at least one of the camera or the target are monitored while presenting the enhanced video stream for display. The method and system then detects a change in the position of at least one of the camera or the target, and upon detecting the change in the position, automatically generates a signal to switch back to the detection mode and detect a position of the target in the video stream.

Abstract: According to a disclosed example, a first video stream is captured via a first camera associated with a first communication device engaged in a multi-party video conference. The first video stream includes a plurality of two-dimensional image frames. A subset of pixels corresponding to a first human subject is identified within each image frame of the first video stream. A second video stream is captured via a second camera associated with a second communication device engaged in the multi-party video conference. A composite video stream formed by at least a portion of the second video stream and the subset of pixels of the first video stream is rendered, and the composite video stream is output for display at one or more of the first and/or second communication devices. The composite video stream may provide the appearance of remotely located participants being physically present within the same visual scene.

Abstract: A transmitting device for generating a plurality of encoded portions of a video to be transmitted to a receiving device over a network configured to: receive an error message over a feedback channel from the receiving device indicating at least one of said plurality of encoded portions that has been lost at the receiving device; encode a recovery portion responsive to said receiving said error message; and transmit said recovery portion to the receiving device over said network; wherein said error message includes information pertaining to a decoded portion successfully decoded at the receiving device and said recovery portion is encoded relative to said decoded portion.

Abstract: Method, device and computer program product for stabilizing a video signal. A plurality of frames of the video signal are captured using a camera. A motion sensor associated with the camera is used to generate a plurality of samples representing motion of the camera. The samples are used to determine a displacement of the camera between a first time and a second time, wherein the first time corresponds to an exposure time midpoint of a first frame of the video signal and the second time corresponds to an exposure time midpoint of a second frame of the video signal. The determined displacement is used to compensate for motion in the video signal between the first and second frames caused by the motion of the camera, to thereby stabilize the video signal.

Abstract: A transmitting device for generating a plurality of encoded portions of a video to be transmitted to a receiving device over a network configured to: receive an error message over a feedback channel from the receiving device indicating at least one of said plurality of encoded portions that has been lost at the receiving device; encode a recovery portion responsive to said receiving said error message; and transmit said recovery portion to the receiving device over said network; wherein said error message includes information pertaining to a decoded portion successfully decoded at the receiving device and said recovery portion is encoded relative to said decoded portion.