Abstract: A computer implemented method includes obtaining video frames to transmit. The video frames are encoded and transmitted via a network connection. A probability that a last video frame will be or was received is determined. A next frame is encoded and transmitted as a P-frame that identifies changes from a previously transmitted frame. The previously transmitted frame is selected as a function of the probability that the last transmitted video frame was received.

Abstract: A gaze detection assembly comprises a plurality of infrared (IR) light emitters configured to emit IR light toward a user eye and an IR camera configured to sequentially capture IR images of the user eye. A controller is configured to, during a reference frame, control the plurality of IR light emitters to emit IR light toward the user eye with a reference intensity distribution. During the reference frame, an IR image is captured depicting a first glint distribution on the user eye. Based at least in part on the first glint distribution, during a subsequent frame, the plurality of IR light emitters are controlled to emit IR light toward the user eye with a subsequent intensity distribution, different than the reference intensity distribution. During the subsequent frame, a second IR image is captured depicting a second glint distribution on the user eye.

Abstract: Examples are disclosed that relate to generating expressive avatars using multi-modal three-dimensional face modeling and tracking. One example includes a computer system comprising a processor coupled to a storage system that stores instructions. Upon execution by the processor, the instructions cause the processor to receive initialization data describing an initial state of a facial model. The instructions further cause the processor to receive a plurality of multi-modal data signals. The instructions further cause the processor to perform a fitting process using the initialization data and the plurality of multi-modal data signals. The instructions further cause the processor to determine a set of parameters based on the fitting process, wherein the determined set of parameters describes an updated state of the facial model.

Abstract: A deep neural network (“DNN”) module compresses and decompresses neuron-generated activation data to reduce the utilization of memory bus bandwidth. The compression unit receives an uncompressed chunk of data generated by a neuron in the DNN module. The compression unit generates a mask portion and a data portion of a compressed output chunk. The mask portion encodes the presence and location of the zero and non-zero bytes in the uncompressed chunk of data. The data portion stores truncated non-zero bytes from the uncompressed chunk of data. A decompression unit receives a compressed chunk of data from memory in the DNN processor or memory of an application host. The decompression unit decompresses the compressed chunk of data using the mask portion and the data portion.

Abstract: A mechanism for adjusting the data rate of data to be transmitted over a data channel based on anticipated changes in the communication channel, rather than based on the current state of the communication channel. This is done by accessing real time environment context data obtained from sensor data generated by one or more sensors of a sensor device, and then predicting a future capacity of a communication channel with a subject head-mounted device based on the accessed real time environment context data. The appropriate data rate is then determined based on this predicted future channel capacity rather than the current channel capacity. The data rate of data that is initiated towards the communication channel is then adjusted based on the determined data rate in anticipation of the predicted future capacity of the communication channel.

Abstract: Eye and hand tracking systems in head-mounted display (HMD) devices are arranged with lensless camera systems using optical masks as encoding elements that apply convolutions to optical images of body parts (e.g., eyes or hands) of HMD device users. The convolved body images are scrambled or coded representations that are captured by a sensor in the system, but are not human-recognizable. A machine learning system such as a neural network is configured to extract body features directly from the coded representation without performance of deconvolutions conventionally utilized to reconstruct the original body images in human-recognizable form. The extracted body features are utilized by the respective eye or hand tracking systems to output relevant tracking data for the user's eyes or hands which may be utilized by the HMD device to support various applications and user experiences. The lensless camera and machine learning system are jointly optimizable on an end-to-end basis.

Abstract: Examples are disclosed that relate to using an array of hot mirrors in an eye-imaging system. One example provides a head-mounted display system, comprising a frame, an eye-imaging camera supported on the frame, a switchable hot mirror array comprising a plurality of switchable hot mirrors configured to direct light reflecting from an eye toward the eye-imaging camera, and a controller configured to control switching of a reflectivity of each of the plurality of switchable hot mirrors.

Abstract: Eye and hand tracking systems in head-mounted display (HMD) devices are arranged with lensless camera systems using optical masks as encoding elements that apply convolutions to optical images of body parts (e.g., eyes or hands) of HMD device users. The convolved body images are scrambled or coded representations that are captured by a sensor in the system, but are not human-recognizable. A machine learning system such as a neural network is configured to extract body features directly from the coded representation without performance of deconvolutions conventionally utilized to reconstruct the original body images in human-recognizable form. The extracted body features are utilized by the respective eye or hand tracking systems to output relevant tracking data for the user's eyes or hands which may be utilized by the HMD device to support various applications and user experiences. The lensless camera and machine learning system are jointly optimizable on an end-to-end basis.

Abstract: Examples are disclosed that relate to using an array of hot mirrors in an eye-imaging system. One example provides a head-mounted display system, comprising a frame, an eye-imaging camera supported on the frame, a switchable hot mirror array comprising a plurality of switchable hot mirrors configured to direct light reflecting from an eye toward the eye-imaging camera, and a controller configured to control switching of a reflectivity of each of the plurality of switchable hot mirrors.

Abstract: A mechanism for repeatedly adjusting communication with a subject head-mounted device based on a changing real time environment of the subject head-mounted device. By utilizing information about the environment context in which the head-mounted device exists, the optimal parameters may be more quickly determined and with less power. The environment context may be generated from sensors on the head-mounted device itself, or from a proximate sensor device. Thus, the communication properties (such as which protocol to use and what parameters) may be quickly determined in time to be useful to maintain a good connection despite movement of the head-mounted device, and despite the connection being dropped and reestablished. Furthermore, limited battery power is more judiciously utilized.

Abstract: A mechanism for adjusting the data rate of data to be transmitted over a data channel based on anticipated changes in the communication channel, rather than based on the current state of the communication channel. This is done by accessing real time environment context data obtained from sensor data generated by one or more sensors of a sensor device, and then predicting a future capacity of a communication channel with a subject head-mounted device based on the accessed real time environment context data. The appropriate data rate is then determined based on this predicted future channel capacity rather than the current channel capacity. The data rate of data that is initiated towards the communication channel is then adjusted based on the determined data rate in anticipation of the predicted future capacity of the communication channel.

Abstract: One disclosed example provides a near-eye display device. The near-eye display device comprises an eye tracking system configured to determine a position of an eye of a user relative to the near-eye display device, and a waveguide including at least an input coupler and an output coupler, the output coupler including a plurality of zones, each zone activatable via a dynamically controllable output coupling element of the zone. The near-eye display device further comprises an image source configured to output image light to the input coupler, and a controller configured to selectively activate one or more zones of the output coupler based at least on the position of the eye.

Abstract: A deep neural network (“DNN”) module can compress and decompress neuron-generated activation data to reduce the utilization of memory bus bandwidth. The compression unit can receive an uncompressed chunk of data generated by a neuron in the DNN module. The compression unit generates a mask portion and a data portion of a compressed output chunk. The mask portion encodes the presence and location of the zero and non-zero bytes in the uncompressed chunk of data. The data portion stores truncated non-zero bytes from the uncompressed chunk of data. A decompression unit can receive a compressed chunk of data from memory in the DNN processor or memory of an application host. The decompression unit decompresses the compressed chunk of data using the mask portion and the data portion. This can reduce memory bus utilization, allow a DNN module to complete processing operations more quickly, and reduce power consumption.

Abstract: A gaze detection assembly comprises a plurality of infrared (IR) light emitters configured to emit IR light toward a user eye and an IR camera configured to sequentially capture IR images of the user eye. A controller is configured to, during a reference frame, control the plurality of IR light emitters to emit IR light toward the user eye with a reference intensity distribution. During the reference frame, an IR image is captured depicting a first glint distribution on the user eye. Based at least in part on the first glint distribution, during a subsequent frame, the plurality of IR light emitters are controlled to emit IR light toward the user eye with a subsequent intensity distribution, different than the reference intensity distribution. During the subsequent frame, a second IR image is captured depicting a second glint distribution on the user eye.

Abstract: A deep neural network (“DNN”) module compresses and decompresses neuron-generated activation data to reduce the utilization of memory bus bandwidth. The compression unit receives an uncompressed chunk of data generated by a neuron in the DNN module. The compression unit generates a mask portion and a data portion of a compressed output chunk. The mask portion encodes the presence and location of the zero and non-zero bytes in the uncompressed chunk of data. The data portion stores truncated non-zero bytes from the uncompressed chunk of data. A decompression unit receives a compressed chunk of data from memory in the DNN processor or memory of an application host. The decompression unit decompresses the compressed chunk of data using the mask portion and the data portion.

Abstract: A deep neural network (“DNN”) module can compress and decompress neuron-generated activation data to reduce the utilization of memory bus bandwidth. The compression unit can receive an uncompressed chunk of data generated by a neuron in the DNN module. The compression unit generates a mask portion and a data portion of a compressed output chunk. The mask portion encodes the presence and location of the zero and non-zero bytes in the uncompressed chunk of data. The data portion stores truncated non-zero bytes from the uncompressed chunk of data. A decompression unit can receive a compressed chunk of data from memory in the DNN processor or memory of an application host. The decompression unit decompresses the compressed chunk of data using the mask portion and the data portion. This can reduce memory bus utilization, allow a DNN module to complete processing operations more quickly, and reduce power consumption.