Abstract: Apparati, methods, and computer readable media for inserting identity information from a source image (static image or video) (301) into a destination video (302), while mimicking motion of the destination video (302). In an apparatus embodiment, an identity encoder (304) is configured to encode identity information of the source image (301). When source image (301) is a multi-frame static image or a video, an identity code aggregator (307) is positioned at an output of the identity encoder (304), and produces an identity vector (314). A driver encoder (313) is coupled to the destination (driver) video (302), and has two components: a pose encoder (305) configured to encode pose information of the destination video (302), and a motion encoder (315) configured to separately encode motion information of the destination video (302). The driver encoder (313) produces two vectors: a pose vector (308) and a motion vector (316).

Abstract: Apparati, methods, and computer readable media for inserting identity information from a source image (1) into a destination image (2), while mimicking illumination of the destination image (2). In an apparatus embodiment, an identity encoder (4) is configured to encode just identity information of the source image (1) and to produce an identity vector (7), where the identity encoder (4) does not encode any pose information or illumination information of the source image (1). A driver encoder (12) has two components: a pose encoder (5) configured to encode pose information of the destination image (2) and an illumination encoder (6) configured to separately encode illumination information of the destination image (2), and to produce two vectors: a pose vector (8) and an illumination vector (9). A neural network generator (10) is coupled to the identity encoder (4) and to the driver encoder (12), and has three inputs: the identity vector (7), the pose vector (8), and the illumination vector (9).

Abstract: Methods and apparati for inserting face and hair information from a source video (401) into a destination (driver) video (402) while mimicking pose, illumination, and hair motion of the destination video (402). An apparatus embodiment comprises an identity encoder (404) configured to encode face and hair information of the source video (401) and to produce as an output an identity vector; a pose encoder (405) configured to encode pose information of the destination video (402) and to produce as an output a pose vector; an illumination encoder (406) configured to encode head and hair illumination of the destination video (402) and to produce as an output an illumination vector; and a hair motion encoder (414) configured to encode hair motion of the destination video (402) and to produce as an output a hair motion vector. The identity vector, pose vector, illumination vector, and hair motion vector are fed as inputs to a neural network generator (410).

Abstract: Methods and apparati for inserting face and hair information from a source video (401) into a destination (driver) video (402) while mimicking pose, illumination, and hair motion of the destination video (402). An apparatus embodiment comprises an identity encoder (404) configured to encode face and hair information of the source video (401) and to produce as an output an identity vector; a pose encoder (405) configured to encode pose information of the destination video (402) and to produce as an output a pose vector; an illumination encoder (406) configured to encode head and hair illumination of the destination video (402) and to produce as an output an illumination vector; and a hair motion encoder (414) configured to encode hair motion of the destination video (402) and to produce as an output a hair motion vector. The identity vector, pose vector, illumination vector, and hair motion vector are fed as inputs to a neural network generator (410).

Abstract: Apparati, methods, and computer readable media for inserting identity information from a source image (static image or video) (301) into a destination video (302), while mimicking motion of the destination video (302). In an apparatus embodiment, an identity encoder (304) is configured to encode identity information of the source image (301). When source image (301) is a multi-frame static image or a video, an identity code aggregator (307) is positioned at an output of the identity encoder (304), and produces an identity vector (314). A driver encoder (313) is coupled to the destination (driver) video (302), and has two components: a pose encoder (305) configured to encode pose information of the destination video (302), and a motion encoder (315) configured to separately encode motion information of the destination video (302). The driver encoder (313) produces two vectors: a pose vector (308) and a motion vector (316).

Abstract: Apparati, methods, and computer readable media for inserting identity information from a source image (1) into a destination image (2), while mimicking illumination of the destination image (2). In an apparatus embodiment, an identity encoder (4) is configured to encode just identity information of the source image (1) and to produce an identity vector (7), where the identity encoder (4) does not encode any pose information or illumination information of the source image (1). A driver encoder (12) has two components: a pose encoder (5) configured to encode pose information of the destination image (2) and an illumination encoder (6) configured to separately encode illumination information of the destination image (2), and to produce two vectors: a pose vector (8) and an illumination vector (9). A neural network generator (10) is coupled to the identity encoder (4) and to the driver encoder (12), and has three inputs: the identity vector (7), the pose vector (8), and the illumination vector (9).