Abstract: Relightable free-viewpoint rendering allows a novel view of a scene to be rendered and relit based on multiple views of the scene from multiple camera viewpoints. An initial texture can be segmented into materials and an initial coarse color estimate is determined for each material. Scene geometry is estimated from the captured views of the scene and is used to scale the initial coarse color estimates relative to each other such that the different materials appear to be lit with a similar irradiance. In this way, a global irradiance function is estimated describing the scene illumination. This provides a starting point for a color estimate and shading estimate extraction. The shading estimate can be used to fit surface normals to the global irradiance function. The set of surface normals and the color estimate are stored for subsequent use to allow relighting of the scene.

Abstract: An image processing system and method for determining an intrinsic color component of one or more objects present in a sequence of frames, for use in rendering the object(s), is described. At least some of the frames of the sequence are to be used as lighting keyframes. A lighting estimate for a lighting keyframe A of the sequence of frames is obtained. An initial lighting estimate for a lighting keyframe B of the sequence of frames is determined. A refined lighting estimate for the lighting keyframe B is determined based on: (i) the initial lighting estimate for the lighting keyframe B, and (ii) the lighting estimate for the lighting keyframe A. The refined lighting estimate for the lighting keyframe B is used to separate image values representing the object(s) in the lighting keyframe B into an intrinsic color component and a shading component, for use in rendering the object(s).

Abstract: An image processing system and method for determining an intrinsic color component of one or more objects for use in rendering the object(s) is described herein. One or more input images are received, each representing a view of the object(s), wherein values of each of the input image(s) are separable into intrinsic color estimates and corresponding shading estimates. A depth image represents depths of the object(s). Coarse intrinsic color estimates are determined using the input image(s). The intrinsic color component is determined by applying bilateral filtering to the coarse intrinsic color estimates using bilateral filtering guidance terms based on depth values derived from the depth image.

Abstract: An image processing system and method for determining an intrinsic color component of one or more objects present in a sequence of frames, for use in rendering the object(s), are described. Some of the frames of the sequence are to be used as lighting keyframes. A lighting estimate for a lighting keyframe A of the sequence of frames is determined. A lighting estimate for a lighting keyframe B of the sequence of frames is determined. A lighting estimate for an intermediate frame positioned between the lighting keyframes A and B in the sequence is determined by interpolating between the lighting estimates determined for the lighting keyframes A and B of the sequence. The determined lighting estimate for the intermediate frame is used to separate image values representing the object(s) in the intermediate frame into an intrinsic color component and a shading component, for use in rendering the object(s).

Abstract: An image processing system and method for determining a set of surface normals of one or more objects for use in rendering the object(s) is described. One or more input images are received, each representing a view of the object(s). A depth image is received representing depth values of the object(s). Coarse surface normal estimates are determined based on depth values of the received depth image. The set of surface normals are determined by applying bilateral filtering to the coarse surface normal estimates. The bilateral filtering uses bilateral filtering guidance terms based on luminance values derived from the input image, wherein the determined set of surface normals are for use in rendering the object(s).

Abstract: An image processing system and method for determining an intrinsic colour component of one or more objects for use in rendering the object(s) is described. One or more input images are received, each representing a view of the object(s), wherein values of the input image(s) are separable into intrinsic colour estimates and corresponding shading estimates. A set of surface normals for the object(s) of the input image(s) is determined. In accordance with the values of the input image(s) and the determined set of surface normals, a global lighting estimate is determined which provides consistent corresponding intrinsic colour estimates for a plurality of regions of the object(s) from the input image(s). The intrinsic colour component is determined in accordance with the values of the input image(s) and the determined global lighting estimate. The determined intrinsic colour component of the object(s) and the determined set of surface normals for the object(s) are stored for use in rendering the object(s).

Abstract: Relightable free-viewpoint rendering allows a novel view of a scene to be rendered and relit based on multiple views of the scene from multiple camera viewpoints. Image values from the multiple camera viewpoints can be separated into diffuse image components and specular image components, such that an intrinsic color component of a relightable texture can be determined for a specular scene, by using the separated diffuse image components. Furthermore, surface normals of geometry in the scene can be refined by constructing a height map based on a conservative component of an initial surface normal field and then determining the refined surface normals based on the constructed height map.

Abstract: Relightable free-viewpoint rendering allows a novel view of a scene to be rendered and relit based on multiple views of the scene from multiple camera viewpoints. Image values from the multiple camera viewpoints can be separated into diffuse image components and specular image components, such that an intrinsic color component of a relightable texture can be determined for a specular scene, by using the separated diffuse image components. Furthermore, surface normals of geometry in the scene can be refined by constructing a height map based on a conservative component of an initial surface normal field and then determining the refined surface normals based on the constructed height map.

Abstract: Relightable free-viewpoint rendering allows a novel view of a scene to be rendered and relit based on multiple views of the scene from multiple camera viewpoints. An initial texture can be segmented into materials and an initial coarse colour estimate is determined for each material. Scene geometry is estimated from the captured views of the scene and is used to scale the initial coarse colour estimates relative to each other such that the different materials appear to be lit with a similar irradiance. In this way, a global irradiance function is estimated describing the scene illumination. This provides a starting point for a colour estimate and shading estimate extraction. The shading estimate can be used to fit surface normals to the global irradiance function. The set of surface normals and the colour estimate are stored for subsequent use to allow relighting of the scene.

Abstract: An image processing system and method for determining a set of surface normals of one or more objects for use in rendering the object(s) is described. One or more input images are received, each representing a view of the object(s). A depth image is received representing depth values of the object(s). Coarse surface normal estimates are determined based on depth values of the received depth image. The set of surface normals are determined by applying bilateral filtering to the coarse surface normal estimates. The bilateral filtering uses bilateral filtering guidance terms based on luminance values derived from the input image, wherein the determined set of surface normals are for use in rendering the object(s).

Abstract: An image processing system and method for determining an intrinsic colour component of one or more objects present in a sequence of frames, for use in rendering the object(s), are described. Some of the frames of the sequence are to be used as lighting keyframes. A lighting estimate for a lighting keyframe A of the sequence of frames is determined. A lighting estimate for a lighting keyframe B of the sequence of frames is determined. A lighting estimate for an intermediate frame positioned between the lighting keyframes A and B in the sequence is determined by interpolating between the lighting estimates determined for the lighting keyframes A and B of the sequence. The determined lighting estimate for the intermediate frame is used to separate image values representing the object(s) in the intermediate frame into an intrinsic colour component and a shading component, for use in rendering the object(s).

Abstract: An image processing system and method for determining an intrinsic colour component of one or more objects for use in rendering the object(s) is described. One or more input images are received, each representing a view of the object(s), wherein values of the input image(s) are separable into intrinsic colour estimates and corresponding shading estimates. A set of surface normals for the object(s) of the input image(s) is determined. In accordance with the values of the input image(s) and the determined set of surface normals, a global lighting estimate is determined which provides consistent corresponding intrinsic colour estimates for a plurality of regions of the object(s) from the input image(s). The intrinsic colour component is determined in accordance with the values of the input image(s) and the determined global lighting estimate. The determined intrinsic colour component of the object(s) and the determined set of surface normals for the object(s) are stored for use in rendering the object(s).

Abstract: An image processing system and method for determining an intrinsic colour component of one or more objects for use in rendering the object(s) is described herein. One or more input images are received, each representing a view of the object(s), wherein values of each of the input image(s) are separable into intrinsic colour estimates and corresponding shading estimates. A depth image represents depths of the object(s). Coarse intrinsic colour estimates are determined using the input image(s). The intrinsic colour component is determined by applying bilateral filtering to the coarse intrinsic colour estimates using bilateral filtering guidance terms based on depth values derived from the depth image.

Abstract: An image processing system and method for determining an intrinsic colour component of one or more objects present in a sequence of frames, for use in rendering the object(s), is described. At least some of the frames of the sequence are to be used as lighting keyframes. A lighting estimate for a lighting keyframe A of the sequence of frames is obtained. An initial lighting estimate for a lighting keyframe B of the sequence of frames is determined. A refined lighting estimate for the lighting keyframe B is determined based on: (i) the initial lighting estimate for the lighting keyframe B, and (ii) the lighting estimate for the lighting keyframe A. The refined lighting estimate for the lighting keyframe B is used to separate image values representing the object(s) in the lighting keyframe B into an intrinsic colour component and a shading component, for use in rendering the object(s).

Abstract: Relightable free-viewpoint rendering allows a novel view of a scene to be rendered and relit based on multiple views of the scene from multiple camera viewpoints. Cast shadows in the scene are handled to allow local occlusion to be taken into account when determining irradiance estimates for the sample positions in the scene. This is achieved by determining an initial estimate of irradiance by ignoring local occlusion, then using that initial irradiance estimate with knowledge of the scene geometry to determine an angle-dependent radiance estimate for the scene. This radiance estimate can then be used with the knowledge of the scene geometry to determine local irradiance estimates at the sample positions within the scene.

Abstract: Relightable free-viewpoint rendering allows a novel view of a scene to be rendered and relit based on multiple views of the scene from multiple camera viewpoints. An initial texture can be segmented into materials and an initial coarse color estimate is determined for each material. Scene geometry is estimated from the captured views of the scene and is used to scale the initial coarse color estimates relative to each other such that the different materials appear to be lit with a similar irradiance. In this way, a global irradiance function is estimated describing the scene illumination. This provides a starting point for a color estimate and shading estimate extraction. The shading estimate can be used to fit surface normals to the global irradiance function. The set of surface normals and the color estimate are stored for subsequent use to allow relighting of the scene.

Abstract: A model of a scene of an image (e.g. a frame of a video sequence) is generated from one or more views of the scene captured from one or more different camera viewpoints. An initial texture for applying to the model is derived from the one or more views of the scene. The initial texture is separated into a lighting estimate and a color estimate, which may be orthogonal and which may be processed independently. The lighting estimate is filtered with a high-pass filter to thereby determine shadow regions of the scene which are regions of detailed shadow which are likely to be caused by ambient occlusion in the scene and which are therefore retained when the texture is relit for rendering the image. A shadow-detail estimate (or “dark map”) is provided which indicates one or more shadow regions of the texture which are to remain in shadow when the image is rendered.

Abstract: Relightable free-viewpoint rendering allows a novel view of a scene to be rendered and relit based on multiple views of the scene from multiple camera viewpoints. Cast shadows in the scene are handled to allow local occlusion to be taken into account when determining irradiance estimates for the sample positions in the scene. This is achieved by determining an initial estimate of irradiance by ignoring local occlusion, then using that initial irradiance estimate with knowledge of the scene geometry to determine an angle-dependent radiance estimate for the scene. This radiance estimate can then be used with the knowledge of the scene geometry to determine local irradiance estimates at the sample positions within the scene.

Abstract: Relightable free-viewpoint rendering allows a novel view of a scene to be rendered and relit based on multiple views of the scene from multiple camera viewpoints. Image values from the multiple camera viewpoints can be separated into diffuse image components and specular image components, such that an intrinsic colour component of a relightable texture can be determined for a specular scene, by using the separated diffuse image components. Furthermore, surface normals of geometry in the scene can be refined by constructing a height map based on a conservative component of an initial surface normal field and then determining the refined surface normals based on the constructed height map.

Abstract: Relightable free-viewpoint rendering allows a novel view of a scene to be rendered and relit based on multiple views of the scene from multiple camera viewpoints. Image values from the multiple camera viewpoints can be separated into diffuse image components and specular image components, such that an intrinsic colour component of a relightable texture can be determined for a specular scene, by using the separated diffuse image components. Furthermore, surface normals of geometry in the scene can be refined by constructing a height map based on a conservative component of an initial surface normal field and then determining the refined surface normals based on the constructed height map.