Abstract: In holographic calling, it is difficult to capture the eyes of a caller due to lighting effects on an artificial reality (XR) headset. However, it can be important to capture the eyes when rendering the caller as they can show emotion, gaze, physical characteristics, etc., that aid in natural communication. Thus, implementations can capture the eyes of the caller using an external image capture device by briefly turning off the lighting effects on the XR headset. Some implementations can trigger the image capture device to capture an image of the eyes by temporal multiplexing in which timers on both the image capture device and the XR headset are synchronized. In other implementations, the image capture device can be an event-based camera that is automatically triggered to capture an image of the eyes based on a detected pixel change caused by deactivation of the lighting effects on the XR headset.

Abstract: Proposed are a deep learning method and apparatus for the automatic upright rectification of VR content. The deep learning method for the automatic upright rectification of VR content according to an embodiment may include inputting a VR image, to a neural network and outputting orientation information of the VR image through a trained neural network.

Abstract: Disclosed herein is a camera calibration method based on deep learning including acquiring an image captured by a camera and predicting an intrinsic parameter of the camera by applying, to the acquired image, a neural network module trained to predict the intrinsic parameter.

Abstract: Described herein are apparatus, systems and methods for synchronizing a reference video with an input video. A method comprises extracting first motion data from the input video having a first set of frames, extracting second motion data from the reference video having a second set of frames, computing motion descriptors for each frame in the first set of frames and the second set of frames based on the first and second motion data, respectively, and non-linearly mapping the first set of frames to the second set of frames based on the motion descriptors, respectively.

Abstract: There is provided a video processing system for use with a video having frames including a first frame and neighboring frames of the first frame. The system includes a memory storing a video processing application, and a processor. The processor is configured to execute the video processing application to sample scene points corresponding to an output pixel of the first frame of the frames of the video, the scene points including alternate observations of a same scene point from the neighboring frames of the first frame of the video, and filter the scene points corresponding to the output pixel to determine a color of the output pixel by calculating a weighted combination of the scene points corresponding to the output pixel.

Abstract: Proposed are a deep learning method and apparatus for the automatic upright rectification of VR content. The deep learning method for the automatic upright rectification of VR content according to an embodiment may include inputting a VR image, to a neural network and outputting orientation information of the VR image through a trained neural network.

Abstract: The present disclosure relates to an apparatus, system and method for processing transmedia content data. More specifically, the disclosure provides for identifying and inserting one item of media content within another item of media content, e.g. inserting a video within a video, such that the first item of media content appears as part of the second item. The invention involves analysing a first visual media item to identify one or more spatial locations to insert the second visual media item within the image data of the first visual media item, detecting characteristics of the one or more identified spatial locations, transforming the second visual media item according to the detected characteristics and combining the first visual media item and second visual media item by inserting the transformed second visual media item into the first visual media item at the one or more identified spatial locations.

Abstract: Disclosed herein is a camera calibration method based on deep learning including acquiring an image captured by a camera and predicting an intrinsic parameter of the camera by applying, to the acquired image, a neural network module trained to predict the intrinsic parameter.

Abstract: Systems and methods to generate omnistereoscopic panoramic videos are presented herein. Depth information, flow fields, and/or other information may be used to determine interpolated frame images between adjacent frame images. An omnistereoscopic panoramic video may be used in a real-world VR application.

Abstract: The present disclosure relates to an apparatus, system and method for processing transmedia content data. More specifically, the disclosure provides for identifying and inserting one item of media content within another item of media content, e.g. inserting a video within a video, such that the first item of media content appears as part of the second item. The invention involves analysing a first visual media item to identify one or more spatial locations to insert the second visual media item within the image data of the first visual media item, detecting characteristics of the one or more identified spatial locations, transforming the second visual media item according to the detected characteristics and combining the first visual media item and second visual media item by inserting the transformed second visual media item into the first visual media item at the one or more identified spatial locations.

Abstract: Systems and methods to generate omnistereoscopic panoramic videos are presented herein. Depth information, flow fields, and/or other information may be used to determine interpolated frame images between adjacent frame images. An omnistereoscopic panoramic video may be used in a real-world VR application.

Abstract: Described herein are apparatus, systems and methods for synchronizing a reference video with an input video. A method comprises extracting first motion data from the input video having a first set of frames, extracting second motion data from the reference video having a second set of frames, computing motion descriptors for each frame in the first set of frames and the second set of frames based on the first and second motion data, respectively, and non-linearly mapping the first set of frames to the second set of frames based on the motion descriptors, respectively.

Abstract: A method for image processing in video conferencing, for correcting the gaze of an interlocutor in an image or a sequence of images captured by at least one real camera, comprises the steps of the at least one real camera acquiring an original image of the interlocutor; synthesizing a corrected view of the interlocutor's face as seen by a virtual camera, the virtual camera being located on the interlocutor's line of sight and oriented towards the interlocutor; transferring the corrected view of the interlocutor's face from the synthesized view into the original image, thereby generating a final image; at least one of displaying the final image and transmitting the final image.

Abstract: There is provided a video processing system for use with a video having frames including a first frame and neighboring frames of the first frame. The system includes a memory storing a video processing application, and a processor. The processor is configured to execute the video processing application to sample scene points corresponding to an output pixel of the first frame of the frames of the video, the scene points including alternate observations of a same scene point from the neighboring frames of the first frame of the video, and filter the scene points corresponding to the output pixel to determine a color of the output pixel by calculating a weighted combination of the scene points corresponding to the output pixel.

Abstract: A method for image processing in video conferencing, for correcting the gaze of an interlocutor in an image or a sequence of images captured by at least one real camera, comprises the steps of the at least one real camera acquiring an original image of the interlocutor; synthesizing a corrected view of the interlocutor's face as seen by a virtual camera, the virtual camera being located on the interlocutor's line of sight and oriented towards the interlocutor; transferring the corrected view of the interlocutor's face from the synthesized view into the original image, thereby generating a final image; at least one of displaying the final image and transmitting the final image.