Abstract: Techniques for manipulating video livestream background images are disclosed. A short-form video, such as a livestream video or livestream replay video, can be analyzed for context. Computer-implemented techniques may be used for performing entity detection, and can also detect a change in subject based on speech and/or actions of a host individual. The subject can include a particular product. The detecting a change in subject can include detecting a foreground object and identifying the foreground object as a product. The identification of the foreground object as a product can include performing optical character recognition on text imprinted on a foreground object. The identification of the foreground object as a product can include image recognition techniques. The identification of the foreground object as a product can include scanning of an optical code such as a barcode that is imprinted on the product.

Abstract: Techniques for video generation based on machine learned video template usage are disclosed. A plurality of videos is obtained, and video scene analysis on each video is performed. Video cuts for each video are detected, and objects within each video are identified. The identifying includes detecting a person, face, building, or vehicle. Metadata is categorized for each of the videos based on the scene analysis, the video cuts, and the objects within the videos. Template information is stored, including the categorized metadata, on each of the videos. Each video is stored as a template video along with the template information. The template information on a subset of videos is ranked. A basis video is selected based on the template information. A further video is generated based on the basis video. The further video is stored as a further template video along with the template videos.

Abstract: Disclosed embodiments describe text-driven AI-assisted short-form video creation. Text from a website is extracted to generate extracted text. Possible summary sentences are formed from the extracted text. The forming is based on natural language processing. The summary sentences are ranked according to an engagement metric. Summary sentences from the possible summary sentences are picked based on a threshold engagement metric value. A list of video scenes is generated based on the summary sentences. Each video scene is associated with a summary sentence. A media asset from a media asset library is chosen for each video scene within the list of video scenes. The choosing is accomplished by machine learning. The list of video scenes, including the media asset that was chosen for each video scene, is compiled into a short-form video. Media is extracted from the website included into the short-form video. The compiling includes a dynamically generated image.

Abstract: Disclosed embodiments provide techniques for dynamic synthetic video chat agent replacement. A human host receives a request for a video chat initiated by a user. An image for a synthetic host including a representation of an individual is retrieved. The image of the synthetic host is selected based on information about the user. Aspects of the individual included in the image are extracted using one or more processors. A video performance by the human host responding to the statement or query by the user is captured. A synthetic host performance is created in which the video performance of the human host is dynamically replaced by the individual that was extracted so that the synthetic host performance responds to the user statement or query. The synthetic host performance is rendered to the user and supplemented with additional synthetic host performances as the video chat continues.

Abstract: Disclosed embodiments provide techniques for synthesized realistic metahuman short-form videos. A photorealistic representation of an individual is accessed from media sources including videos, photographs, livestreams, or a 360-degree recording of a human host. The individual can be selected based on information on the viewer of the short-form video, such as purchase history, viewing history, and metadata. The photorealistic representation is isolated using machine learning and is used to create a three-dimensional (3D) model of the individual, based on a game engine. A realistic synthetic performance is created by combining the 3D model of the individual with animation generated by a game engine. The synthesized performance can include the voice of the human host. The synthesized performance is inserted into a metaverse environment and rendered to a viewer as a short-form video, including an ecommerce window with an on-screen product card and virtual purchase cart.

Abstract: Disclosed embodiments provide techniques for iterative AI prompt optimization for video generation. A first text template to be read by a large language model (LLM) neural network is accessed. The template includes control parameters that are populated from within a website. The populated template is submitted as a request to the LLM neural network, which generates a first video script. The first video script is used to create a first short-form video. The first short-form video is evaluated based on one or more performance metrics. The text template, short-form video, website information, and evaluation are used to train a machine learning model that is used to create a second text template. The second text template can be used to generate a second short-form video. The evaluation of iterative text templates and resulting short-form videos continues until a usable video is produced.

Abstract: Disclosed embodiments provide techniques for livestream with synthetic scene insertion. A prerecorded livestream featuring a host is rendered to one or more viewers. An operator accesses a video segment related to the prerecorded livestream. The related video segment includes a performance by an individual. The operator retrieves an image of the host of the prerecorded livestream and creates from the related video segment a synthesized video segment that includes the performance of the individual accomplished by the host. One or more insertion points are determined within the prerecorded livestream for the insertion of the synthesized video segment. The operator inserts the synthesized video segment into the prerecorded livestream at the one or more determined insertion points. The insertion is accomplished dynamically and appears seamless to the viewer. The remainder of the prerecorded video is rendered to the one or more viewers after the insertion point.

Abstract: Disclosed embodiments provide techniques for augmented performance replacement in a short-form video. A short-form video is accessed, including a performance by a first individual. Using one or more processors, the performance of the first individual is isolated. Specific elements of the performance including gestures, clothing, expressions, and accessories are included in the isolation process. An image of a second individual is retrieved and information on the second individual is extracted from the image. A second short-form video is created by replacing the performance of the first individual with the second individual. The second short-form video is augmented based on viewer interaction. The augmenting of the second short-form video occurs dynamically. The augmenting includes additional audio content based on comments, responses to live polls or surveys, or questions and answers from viewers. The augmenting includes switching audio content in the second short-form video with additional audio content.

Abstract: Disclosed embodiments describe text-driven AI-assisted short-form video creation. Text from a website is extracted to generate extracted text. Possible summary sentences are formed from the extracted text. The forming is based on natural language processing. The summary sentences are ranked according to an engagement metric. Summary sentences from the possible summary sentences are picked based on a threshold engagement metric value. A list of video scenes is generated based on the summary sentences. Each video scene is associated with a summary sentence. A media asset from a media asset library is chosen for each video scene within the list of video scenes. The choosing is accomplished by machine learning. The list of video scenes, including the media asset that was chosen for each video scene, is compiled into a short-form video. Media is extracted from the website included into the short-form video. The compiling includes a dynamically generated image.

Abstract: Techniques for video manipulation based on machine learned video curating are disclosed. Web page content is loaded, where the content includes a frame for short-form videos. The content of the web page is analyzed for textual information. A short-form video server is accessed. Short-form videos are selected from the short-form video server, where the selecting includes automatically curating the short-form videos. Adaptive learning is used for the selecting, based on a user's web page behavior. The adaptive learning includes collecting the user's web page behavior before the selecting. Automatic curating includes selecting, by a neural network, a subset of short-form videos appropriate for the web page. The web page frame is populated with the short-form videos obtained from the video server. Representations of the short-form videos are displayed within the frame on the web page. The short-form videos are auto played within the frame.

Abstract: Disclosed embodiments provide techniques for search using generative model synthesized images. An image, or set of images, is generated based on a text query, and the generated image or images are used as input to an image-based search query. In some instances, this technique provides more comprehensive and effective search results. In cases where a user wishes to search for something for which no known image currently exists, disclosed embodiments can provide more effective results than a text-based query. Disclosed embodiments generate an input image based on the text query using generative model synthesized image techniques. The input image is used to perform an image-based search, and sorted results are returned. Disclosed embodiments are particularly useful when searching for visual art in the form of images and/or videos.

Abstract: Techniques for manipulating video livestream background images are disclosed. A short-form video, such as a livestream video or livestream replay video, can be analyzed for context. Computer-implemented techniques may be used for performing entity detection, and can also detect a change in subject based on speech and/or actions of a host individual. The subject can include a particular product. The detecting a change in subject can include detecting a foreground object and identifying the foreground object as a product. The identification of the foreground object as a product can include performing optical character recognition on text imprinted on a foreground object. The identification of the foreground object as a product can include image recognition techniques. The identification of the foreground object as a product can include scanning of an optical code such as a barcode that is imprinted on the product.

Abstract: Techniques for tokenizing a manipulated short-form video are disclosed. A short-form video is obtained from a short-form environment that includes a short-form video server. The short-form video includes news, weather, traffic information, music, sports highlights, vlog entries, product information, how-to videos, livestream replays, and/or other content. Highlight segments within short-form videos are identified, and a new video is created, based on one or more identified highlight segments. The new video may also be a short-form video. The highlight segments may be from the original short-form video, or may come from a variety of other sources. The new video is used to enhance entertainment value. The new video is tokenized. The tokenization can be used to support ownership identification. The tokenization can be used to create a non-fungible token.

Abstract: Disclosed techniques include video performance rendering modification based on device rotation metric. A media stream is received from a server. The media stream is targeted for display on a handheld media stream playback device. An orientation of the handheld device is determined during playback of the media stream on the handheld device. A change in the orientation of the handheld device is detected. The change occurs during playback of the media stream. Based on the detecting the change of orientation, a rotation angle of the handheld device is calculated. Based on the rotation angle, the playback speed of the media stream on the handheld device is changed. The changing the playback speed changes direction of playback rendering based on the rotation angle of the handheld device. The changing direction provides forward playback by rotating the handheld device in a clockwise direction from an initial orientation.

Abstract: Technologies relating to a system and method of real-time video overlaying and sharing from multiple cameras are disclosed. An example method of real-time video overlaying and sharing includes the steps of: capturing video output from a first camera on a mobile device, recognizing a portion of an individual in the captured video output and determining a user body contour, generating a binary mask, enabling real-time video processing, including the separate user body contour from a background of the video from the first camera, smoothing one or more edges of the binary mask, merging the binary mask with the video from the first camera, and creating a composite video, merged from the first camera video output onto a second camera video output. The second camera can be included on the same mobile device as the first camera, facing in the opposite direction, or on a second mobile device.

Abstract: Techniques for display using a video stream interface based on third-party webpage information are disclosed. Information on one or more products is obtained from a website by a third party. The one or more products comprise a product, a service, a coupon, or a membership. A video about the one or more products is rendered by a video user interface to a user. The video user interface comprises a video player in combination with a user interface. The video user interface and the third party are interfaced between each other based on a user interaction with the video user interface. Product information is fetched from a server based on the interfacing between the video user interface and the third party. Further product information comprises a reaction, a chat, a transaction, a promotion, and an interaction. The video is augmented with further information based on the fetching.

Abstract: Techniques for display using frame and child frame technology for video and webpage rendering are disclosed. A video stream is received. The video stream is displayed as a video in a window on a device display along with a link. The link includes a webpage uniform resource locator (URL). A selection of the link made by a user is received. The selection is made in response to a call to action. The video is rendered along with results of the link that was selected. Rendering of the results of the link includes a webpage associated with the webpage URL. The user can interact with the webpage while the video is being rendered. The video is reduced to render the video as a reduced video on a portion of the device display. The rendering the reduced video and webpage is accomplished without a popup window.

Abstract: Techniques for display based on inline frame-based video rendering within browser are disclosed. A plurality of short-form videos is obtained. Thumbnails of one or more short-form videos, from the plurality of short-form videos, are displayed. The displaying thumbnails occurs within a frame on a third-party webpage within a browser window. A user's selection of a video from the short-form videos is received. The video that was selected by a user is reduced. The reducing the video enables the video to be rendered in a portion of the browser window. The reducing the video to render the video in a portion of the browser window includes launching a reloaded version of the third-party webpage to render the video superimposed over the third-party webpage. The video that was reduced is rendered, along with a reloaded copy of the third-party webpage, in the browser window.

Abstract: Disclosed techniques include audio and visual stream rendering modification based on device rotation metric. One or more media streams are received from a server. The one or more media streams are targeted for display on a handheld media stream playback device. An orientation of the handheld device is determined during playback of at least one of the one or more media streams on the handheld device. A change in orientation of the handheld device is detected. The change occurs during playback of the at least one of the one or more media streams. Based on detecting the change in orientation, a rotation angle of the handheld device is calculated. Based on the rotation angle, the playback of at least one of the one or more media streams is changed on the handheld device.