Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program and a method for rendering a virtual object in a real-world environment depicted in image content based on contextual information. A virtual object template is selected. One or more stylizations for the virtual object template are determined based on contextual information associated with a computing device. A virtual object is generated by applying the one or more stylizations to the virtual object template. The virtual object is rendered within a 3D space captured within a camera feed of the computing device.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program and method for performing operations comprising: receiving, by a messaging application, a video feed from a camera of a user device that depicts a face; receiving a request to add a 3D caption to the video feed; identifying a graphical element that is associated with context of the 3D caption; and displaying the 3D caption and the identified graphical element in the video feed at a position in 3D space of the video feed proximate to the face depicted in the video feed.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program and method for performing operations comprising: receiving, by a messaging application, an image generated by a camera of a user; displaying an option to crop an object depicted in the image; determining whether the object is depicted in the image; selectively enabling selection of the option to crop the object based on determining whether the object is depicted in the image; receiving input that selects the option to crop an object depicted in the image when the option is enabled; in response to receiving the input, segmenting the image to separate the object from other portions depicted in the image; extracting the object from the image; and adding the extracted object to a new image as a virtual object.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program and method for rendering three-dimensional captions (3D) in real-world environments depicted in image content. An editing interface is displayed on a client device. The editing interface includes an input component displayed with a view of a camera feed. A first input comprising one or more text characters is received. In response to receiving the first input, a two-dimensional (2D) representation of the one or more text characters is displayed. In response to detecting a second input, a preview interface is displayed. Within the preview interface, a 3D caption based on the one or more text characters is rendered at a position in a 3D space captured within the camera feed. A message is generated that includes the 3D caption rendered at the position in the 3D space captured within the camera feed.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program, and a method for rendering three-dimensional virtual objects within real-world environments. Virtual rendering of a three-dimensional virtual object can be altered appropriately as a user moves around the object in the real-world through utilization of a redundant tracking system comprising multiple tracking sub-systems. Virtual object rendering can be with respect to a reference surface in a real-world three-dimensional space depicted in a camera view of a mobile computing device.

Abstract: Systems and methods are provided for capturing by a camera of a user device, a first image depicting a first environment of the user device; overlaying a first virtual object on a portion of the first image depicting the first environment; modifying a surface of the first virtual object using content captured by the user device; storing a second virtual object comprising the first virtual object with the modified surface; and generating for display the second virtual object on a portion of a second image depicting a second environment.

Abstract: The subject technology receives image data and depth data. The subject technology selects an augmented reality content generator corresponding to a three-dimensional (3D) effect. The subject technology applies the 3D effect to the image data and the depth data based at least in part on the selected augmented reality content generator. The subject technology generates, using a processor, a message including information related to the applied 3D effect, the image data, and the depth data.

Abstract: Systems and methods are provided for determining a location of a selection in a space viewable in a camera view on a display of a computing device, detecting movement of the computing device, and generating a path based on the location of the selection and the movement of the computing device. The systems and methods further provide for generating a three-dimensional (3D) mesh along the path, populating the 3D mesh with selected options to generate a 3D paint object, and causing the generated 3D paint object to be displayed. The systems and methods further provide for receiving a request to send a message comprising an image or video overlaid by the 3D paint object, capturing the image or video overlaid by the displayed 3D paint object, generating the message comprising the image or video overlaid by the 3D paint object, and sending the message to another computing device.

Abstract: Systems and methods are provided for receiving a two-dimensional (2D) image comprising a 2D object; identifying a contour of the 2D object; generating a three-dimensional (3D) mesh based on the contour of the 2D object; and applying a texture of the 2D object to the 3D mesh to output a 3D object representing the 2D object.

Abstract: The subject technology selects a set of augmented reality content generators from available augmented reality content generators, the selected set of augmented reality content generators comprising at least one augmented reality content generator for applying a three-dimensional (3D) effect. The subject technology causes display of a carousel interface including selectable graphical items, each selectable graphical item corresponding to a respective augmented reality content generator. The subject technology receives a selection of a first selectable graphical item from the selectable graphical items, the first selectable graphical item including a first augmented reality content generator for applying a first 3D effect. The subject technology applies, to first image data and first depth data, the first augmented reality content generator corresponding to the selected first selectable graphical item.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program and method for performing operations comprising: receiving, by a messaging application, an image generated by a camera of a user; displaying an option to crop an object depicted in the image; determining whether the object is depicted in the image; selectively enabling selection of the option to crop the object based on determining whether the object is depicted in the image; receiving input that selects the option to crop an object depicted in the image when the option is enabled; in response to receiving the input, segmenting the image to separate the object from other portions depicted in the image; extracting the object from the image; and adding the extracted object to a new image as a virtual object.

Abstract: Systems and methods are provided for initiating transfer of image data corresponding to at least one predetermined level of an image pyramid comprising higher resolution to a graphic processing unit (GPU) of the computing device, calculating, by the central processing unit (CPU) of the computing device, optical flow of at least one predetermined coarse level of the image pyramid, transferring, by the CPU of the computing device, the calculated optical flow of the at least one predetermined coarse level of the image pyramid to the GPU, calculating, by the GPU of the computing device, the optical flow of the at least one predetermined level of the image pyramid comprising higher resolution, and outputting, by the GPU of the computing device, the optical flow of the image data.

Abstract: Systems and methods are provided for determining a location of a selection in a space viewable in a camera view on a display of a computing device, detecting movement of the computing device, and generating a path based on the location of the selection and the movement of the computing device. The systems and methods further provide for generating a three-dimensional (3D) mesh along the path, populating the 3D mesh with selected options to generate a 3D paint object, and causing the generated 3D paint object to be displayed. The systems and methods further provide for receiving a request to send a message comprising an image or video overlaid by the 3D paint object, capturing the image or video overlaid by the displayed 3D paint object, generating the message comprising the image or video overlaid by the 3D paint object, and sending the message to another computing device.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program and method for rendering three-dimensional captions (3D) in real-world environments depicted in image content. An editing interface is displayed on a client device. The editing interface includes an input component displayed with a view of a camera feed. A first input comprising one or more text characters is received. In response to receiving the first input, a two-dimensional (2D) representation of the one or more text characters is displayed. In response to detecting a second input, a preview interface is displayed. Within the preview interface, a 3D caption based on the one or more text characters is rendered at a position in a 3D space captured within the camera feed. A message is generated that includes the 3D caption rendered at the position in the 3D space captured within the camera feed.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program, and a method for rendering a three-dimensional virtual object in a video clip. The method and system include capturing, using a camera-enabled device, video content of a real-world scene and movement information collected by the camera-enabled device during capture of the video content. The captured video and movement information are stored. The stored captured video content is processed to identify a real-world object in the scene. An interactive augmented reality display is generated that: adds a virtual object to the stored video content to create augmented video content comprising the real-world scene and the virtual object and adjusts, during playback of the augmented video content, an on-screen position of the virtual object within the augmented video content based at least in part on the stored movement information.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program and method for performing operations comprising: receiving, by a messaging application, an image from a camera of a user device; receiving input that selects a user-customizable effects option for activating a user-customizable effects mode; in response to receiving the input, displaying an array of a plurality of effect options together with the image proximate to the user-customizable effects option; and applying a first effect associated with a first effect option of the plurality of effect options to the image.

Abstract: In various example embodiments, a system and methods are presented for generation and manipulation of three dimensional (3D) models. The system and methods cause presentation of an interface frame encompassing a field of view of an image capture device. The systems and methods detect an object of interest within the interface frame, generate a movement instruction with respect to the object of interest, and detect a first change in position and a second change in position of the object of interest. The systems and methods generate a 3D model of the object of interest based on the first change in position and the second change in position.

Abstract: Among other things, embodiments of the present disclosure improve the functionality of computer software and systems by facilitating the automatic performance optimization of a software application based on the particular platform upon which the application runs. In some embodiments, the system can automatically choose a set of parameters or methods at run-time from a design space with pre-selected optimization methods and parameters (e.g., algorithms, software libraries, and/or hardware accelerators) for a specific task.

Abstract: A mobile device can generate real-time complex visual image effects using asynchronous processing pipeline. A first pipeline applies a complex image process, such as a neural network, to keyframes of a live image sequence. A second pipeline generates flow maps that describe feature transformations in the image sequence. The flow maps can be used to process non-keyframes on the fly. The processed keyframes and non-keyframes can be used to display a complex visual effect on the mobile device in real-time or near real-time.

Abstract: Systems and methods are provided for capturing by a camera of a user device, a first image depicting a first environment of the user device; overlaying a first virtual object on a portion of the first image depicting the first environment; modifying a surface of the first virtual object using content captured by the user device; storing a second virtual object comprising the first virtual object with the modified surface; and generating for display the second virtual object on a portion of a second image depicting a second environment.