Abstract: Embodiments of the present invention enable rich control input data to control video games that are remotely executed. Rich control input includes three-dimensional image data, color video, audio, device orientation data, and touch input. A remotely-executed video game is one executed on a server or other computing device that is networked to a client device receiving the rich control input. Rich control input includes more data than can be uploaded to a game server without degrading game performance. Embodiments of the present invention preprocess the rich control data on the client and into data that may be uploaded to the game server. The rich input stream may be processed in a general way or in a game-specific way.

Abstract: An initial candidate foreground region is identified within an infrared image that includes pixels exhibiting infrared intensity values within a pre-defined range. A depth of surfaces within the initial candidate foreground region is estimated based on infrared intensity values the pixels of the initial candidate foreground region. The initial candidate foreground region is expanded to an expanded candidate foreground region based on a body-model estimate. The body model estimate is seeded with one or more of the initial candidate foreground region, the depth of surfaces, and/or a face of a human subject identified by facial recognition. Each pixel of the infrared image is identified as either a foreground pixel or a background pixel based on a distance of that pixel relative to the expanded candidate foreground region. Pixels identified as background pixels may be modified within a corresponding visible light image.

Abstract: Techniques for automatically selecting from amongst multiple camera views available in a multi-camera system. In an aspect, one or more metrics are calculated for multiple camera views. The metrics may be used to identify one or more of the camera views as fulfilling a condition, in response to which a particular one of the camera views may be selected for display. For example, in a “select-better-view” strategy, the camera view associated with a more optimal image (e.g., better lighting, or absence of occlusion) is selected for display. In a “flag-diagnostic-view” strategy, the camera view associated with a less optimal image is displayed, e.g., to indicate that a particular camera may require user attention. Specific techniques for calculating metrics are further disclosed.

Abstract: Embodiments of the present invention split game processing and rendering between a client and a game server. A rendered video game image is received from a game server and combined with a rendered image generated by the game client to form a single video game image that is presented to a user. Game play may be controlled using a rich sensory input, such as three-dimensional image data and audio data. The three-dimensional image data describes the shape, size and orientation of objects present in a play space. The rich sensory input is communicated to a game server, potentially with some preprocessing, and is also consumed locally on the client, at least in part. In one embodiment, latency sensitive features are the only features processed on the client and rendered on the client.

Abstract: A signal encoding an infrared (IR) image including a plurality of IR pixels is received from an IR camera. Each IR pixel specifies one or more IR parameters of that IR pixel. IR-skin pixels that image a human hand are identified in the IR image. For each IR-skin pixel, a depth of a human hand portion imaged by that IR-skin pixel is estimated based on the IR parameters of that IR-skin pixel. A skeletal hand model including a plurality of hand joints is derived. Each hand joint is defined with three independent position coordinates inferred from the estimated depths of each human hand portion.

Abstract: A scene is illuminated with modulated illumination light that reflects from surfaces in the scene as modulated reflection light. Each of a plurality of pixels of a depth camera receive the modulated reflection light and observe a phase difference between the modulated illumination light and the modulated reflection light. For each of the plurality of pixels, an edginess of that pixel is recognized, and the phase difference of that pixel is smoothed as a function of the edginess of that pixel.

Abstract: Embodiments of the present invention enable rich control input data to control video games that are remotely executed. Rich control input includes three-dimensional image data, color video, audio, device orientation data, and touch input. A remotely-executed video game is one executed on a server or other computing device that is networked to a client device receiving the rich control input. Rich control input includes more data than can be uploaded to a game server without degrading game performance. Embodiments of the present invention preprocess the rich control data on the client and into data that may be uploaded to the game server. The rich input stream may be processed in a general way or in a game-specific way.

Abstract: Embodiments of the present invention enable rich control input data to control video games that are remotely executed. Rich control input includes three-dimensional image data, color video, audio, device orientation data, and touch input. A remotely-executed video game is one executed on a server or other computing device that is networked to a client device receiving the rich control input. Rich control input includes more data than can be uploaded to a game server without degrading game performance. Embodiments of the present invention preprocess the rich control data on the client and into data that may be uploaded to the game server. The rich input stream may be processed in a general way or in a game-specific way.

Abstract: A signal encoding an infrared (IR) image including a plurality of IR pixels is received from an IR camera. Each IR pixel specifies one or more IR parameters of that IR pixel. IR-skin pixels that image a human hand are identified in the IR image. For each IR-skin pixel, a depth of a human hand portion imaged by that IR-skin pixel is estimated based on the IR parameters of that IR-skin pixel. A skeletal hand model including a plurality of hand joints is derived. Each hand joint is defined with three independent position coordinates inferred from the estimated depths of each human hand portion.

Abstract: An initial candidate foreground region is identified within an infrared image that includes pixels exhibiting infrared intensity values within a pre-defined range. A depth of surfaces within the initial candidate foreground region is estimated based on infrared intensity values the pixels of the initial candidate foreground region. The initial candidate foreground region is expanded to an expanded candidate foreground region based on a body-model estimate. The body model estimate is seeded with one or more of the initial candidate foreground region, the depth of surfaces, and/or a face of a human subject identified by facial recognition. Each pixel of the infrared image is identified as either a foreground pixel or a background pixel based on a distance of that pixel relative to the expanded candidate foreground region. Pixels identified as background pixels may be modified within a corresponding visible light image.

Abstract: Techniques for automatically selecting from amongst a plurality of camera views available in a multi-camera system. In an aspect, one or more metrics are calculated for a plurality of camera views. The metrics may be used to identify one or more of the camera views as fulfilling a condition, in response to which a particular one of the camera views may be selected for display. For example, in a “select-better-view” strategy, the camera view associated with a more optimal image (e.g., better lighting, or absence of occlusion) is selected for display. In a “flag-diagnostic-view” strategy, the camera view associated with a less optimal image is displayed, e.g., to indicate that a particular camera may require user attention. Specific techniques for calculating metrics are further disclosed.

Abstract: A scene is illuminated with modulated illumination light that reflects from surfaces in the scene as modulated reflection light. Each of a plurality of pixels of a depth camera receive the modulated reflection light and observe a phase difference between the modulated illumination light and the modulated reflection light. For each of the plurality of pixels, an edginess of that pixel is recognized, and the phase difference of that pixel is smoothed as a function of the edginess of that pixel.

Abstract: Embodiments of the present invention split game processing and rendering between a client and a game server. A rendered video game image is received from a game server and combined with a rendered image generated by the game client to form a single video game image that is presented to a user. Game play may be controlled using a rich sensory input, such as three-dimensional image data and audio data. The three-dimensional image data describes the shape, size and orientation of objects present in a play space. The rich sensory input is communicated to a game server, potentially with some preprocessing, and is also consumed locally on the client, at least in part. In one embodiment, latency sensitive features are the only features processed on the client and rendered on the client.

Abstract: Embodiments of the present invention enable rich control input data to control video games that are remotely executed. Rich control input includes three-dimensional image data, color video, audio, device orientation data, and touch input. A remotely-executed video game is one executed on a server or other computing device that is networked to a client device receiving the rich control input. Rich control input includes more data than can be uploaded to a game server without degrading game performance. Embodiments of the present invention preprocess the rich control data on the client and into data that may be uploaded to the game server. The rich input stream may be processed in a general way or in a game-specific way.

Abstract: A method for estimating a metabolic equivalent of task for use with a computing device is provided herein. The method includes receiving input from a capture device of a user; and tracking a position of each of the plurality of joints of the user. The method further includes determining a distance traveled for each of the plurality of joints between a first frame and a second frame; and calculating a horizontal velocity and a vertical velocity for each of the plurality of joints based on the distance traveled and an elapsed time between the first and second frames. The method further includes estimating a value for the metabolic equivalent of task using a metabolic equation including a component for the horizontal velocity and a component for the vertical velocity for each of the plurality of joints; and outputting the value for display.