Abstract: Computer-implemented techniques can include obtaining languages for automatic speech recognition, the languages being associated with a plurality of users and receiving an input indicative of a request to provide speech input. In response to receiving the input, a first indication that automatic speech recognition (ASR) is being performed in each of the plurality of languages can be displayed. A speech input from one of the plurality of users can be received and ASR can identify one of the languages associated with the speech input to obtain a detected language. In response to obtaining the detected language, a second indication that ASR is being performed in the detected language can be displayed. The speech input can be transcribed and translated to another one of the plurality of languages to obtain a translated text that can be displayed.

Abstract: A method for encoding a spherical video is disclosed. The method includes mapping a frame of the spherical video to a two dimensional representation based on a projection. Further, in a prediction process the method includes determine whether at least one block associated with a prediction scheme is on a boundary of the two dimensional representation, and upon determining the at least one block associated with the prediction scheme is on the boundary, select an adjacent end block as a block including at least one pixel for use during the prediction process, the adjacent end block being associated with two or more boundaries of the two dimensional representation.

Abstract: Various embodiments of the present disclosure relate generally to systems and methods for generating multi-view interactive digital media representations in a virtual reality environment. According to particular embodiments, a plurality of images is fused into a first content model and a first context model, both of which include multi-view interactive digital media representations of objects. Next, a virtual reality environment is generated using the first content model and the first context model. The virtual reality environment includes a first layer and a second layer. The user can navigate through and within the virtual reality environment to switch between multiple viewpoints of the content model via corresponding physical movements. The first layer includes the first content model and the second layer includes a second content model and wherein selection of the first layer provides access to the second layer with the second content model.

Abstract: Various embodiments of the present disclosure relate generally to systems and methods for generating multi-view interactive digital media representations in a virtual reality environment. According to particular embodiments, a plurality of images is fused into a first content model and a first context model, both of which include multi-view interactive digital media representations of objects. Next, a virtual reality environment is generated using the first content model and the first context model. The virtual reality environment includes a first layer and a second layer. The user can navigate through and within the virtual reality environment to switch between multiple viewpoints of the content model via corresponding physical movements. The first layer includes the first content model and the second layer includes a second content model and wherein selection of the first layer provides access to the second layer with the second content model.

Abstract: A plurality of videos associated with a plurality of video resolutions may be received. A minimum bitrate may be identified for each of the video resolutions based on a first bitrate point associated with the videos where a quality of the videos at a first video resolution that is upscaled to a second video resolution is better than a quality of the videos at the second video resolution at bitrates below the first bitrate point. Furthermore, a maximum bitrate may be identified for each of the video resolutions based on a second bitrate point where the rate of change of the quality of the videos is below a threshold value. The identified minimum bitrates and the maximum bitrates may be assigned to the video resolutions.

Abstract: Computer-implemented techniques can include receiving a selected word in a source language, obtaining one or more parts of speech for the selected word, and for each of the one or more parts-of-speech, obtaining candidate translations of the selected word to a different target language, each candidate translation corresponding to a particular semantic meaning of the selected word. The techniques can include for each semantic meaning of the selected word: obtaining an image corresponding to the semantic meaning of the selected word, and compiling translation information including (i) the semantic meaning, (ii) a corresponding part-of-speech, (iii) the image, and (iv) at least one corresponding candidate translation. The techniques can also include outputting the translation information.

Abstract: A computer-implemented technique includes techniques are presented for user image capture feedback for improved machine language translation. When machine language translation of OCR text obtained from an initial image has a low degree of likelihood of being an appropriate translation, these techniques provide for user image capture feedback to obtain additional images to obtain a modified OCR text, which can result in improved machine language translation results. Instead of user image capture feedback, the techniques may obtain the modified OCR text by selecting another possible OCR text from the initial OCR operation. In addition to additional image capturing, light source intensity and/or a quantity/number of light source flashes can be adjusted. After obtaining the modified OCR text, another machine language translation can be obtained and, if it has a high enough degree of likelihood, it can then be output to a user.

Abstract: Computer-implemented techniques can include capturing, by a microphone associated with a computing device having one or more processors, a speech input from a user, the speech input comprising a single word in a source language, and in response to receiving the speech input from the user, performing a plurality of actions. The plurality of actions can include identifying, by the computing device, the source language of the single word and a target language that is associated with the user, obtaining, by the computing device, one or more translated words that are each a potential translation of the single word to the target language, obtaining, by the computing device, lexicon data for the one or more translated words, the lexicon data relating to at least one semantic meaning of the one or more translated words, and displaying, by the computing device, the lexicon data.

Abstract: Decoding encoded transform coefficients of a current block includes initializing a decoder state of a state machine having Boolean and symbol ANS decoders. The decoder state includes an ANS state and a buffer position within a buffer storing a variable string including the encoded transform coefficients. The transform coefficients are sequentially produced from the variable string using the state machine by processing a binary flag/bit using the Boolean ANS decoder and processing a token using the symbol ANS decoder. Each decoder performs state normalization when the ANS state is outside a valid state range, performs output computation to generate an output value for the binary flag/bit or token using the ANS state and a probability, and updates the ANS state using the output value and the probability as inputs. The decoder state evolution operations may be different. An encoder state machine having Boolean and symbol ANS encoders is also described.

Abstract: Various embodiments describe systems and processes for generating AR/VR content. In one aspect, a method for generating a 3D projection of an object in a virtual reality or augmented reality environment comprises obtaining a sequence of images along a camera translation using a single lens camera. Each image contains a portion of overlapping subject matter, including the object. The object is segmented from the sequence of images using a trained segmenting neural network to form a sequence of segmented object images, to which an art-style transfer is applied using a trained transfer neural network. On-the-fly interpolation parameters are computed and stereoscopic pairs are generated for points along the camera translation from the refined sequence of segmented object images for displaying the object as a 3D projection in a virtual reality or augmented reality environment. Segmented image indices are mapped to a rotation range for display in the virtual reality or augmented reality environment.

Abstract: Various embodiments describe systems and processes for generating AR/VR content. In one aspect, a method for generating a three-dimensional (3D) projection of an object is provided. A sequence of images along a camera translation may be obtained using a single lens camera. Each image contains at least a portion of overlapping subject matter, which includes the object. The object is semantically segmented from the sequence of images using a trained neural network to form a sequence of segmented object images, which are then refined using fine-grained segmentation. On-the-fly interpolation parameters are computed and stereoscopic pairs are generated for points along the camera translation from the refined sequence of segmented object images for displaying the object as a 3D projection in a virtual reality or augmented reality environment. Segmented image indices are then mapped to a rotation range for display in the virtual reality or augmented reality environment.

Abstract: Provided are mechanisms and processes for augmenting multi-view image data with synthetic objects using inertial measurement unit (IMU) and image data. In one example, a process includes receiving a selection of an anchor location in a reference image for a synthetic object to be placed within a multi-view image. Movements between the reference image and a target image are computed using visual tracking information associated with the multi-view image, device orientation corresponding to the multi-view image, and an estimate of the camera's intrinsic parameters. A first synthetic image is then generated by placing the synthetic object at the anchor location using visual tracking information in the multi-view image, orienting the synthetic object using the inverse of the movements computed between the reference image and the target image, and projecting the synthetic object along a ray into a target view associated with the target image.

Abstract: Various embodiments of the present invention relate generally to systems and methods for integrating audio into a multi-view interactive digital media representation. According to particular embodiments, one process includes retrieving a multi-view interactive digital media representation that includes numerous images fused together into content and context models. The process next includes retrieving and processing audio data to be integrated into the multi-view interactive digital media representation. A first segment of audio data may be associated with a first position in the multi-view interactive digital media representation. In other examples, a first segment of audio data may be associated with a visual position or the location of a camera in the multi-view interactive digital media representation.

Abstract: A computer-implemented technique can include receiving a media feed from a speaker computing device representing speech of a speaker user captured by the speaker computing device. The technique can include receiving a plurality of translation requests, each translation request being received from a listener computing device associated with a listener user and corresponding to a request to obtain a translated version of the media feed into a preferred language of the listener user. The technique can include determining the preferred language for each listener user. The technique can include obtaining a machine translated media feed for each of the translation requests, the machine translated media feed corresponding to a translation of the media feed from the source language to the preferred language of the listener user associated with the translation request. The technique can also include outputting the machine translated media feeds to the listener computing devices.

Abstract: A computer-implemented technique can include receiving, at a server from a mobile computing device, the server having one or more processors, an image including a text. The technique can include obtaining, at the server, optical character recognition (OCR) text corresponding to the text, the OCR text having been obtained by performing OCR on the image. The technique can include identifying, at the server, non-textual context information from the image, the non-textual context information (i) representing context information other than the text itself and (ii) being indicative of a context of the image. The technique can include based on the non-textual context information, obtaining, at the server, a translation of the OCR text to a target language to obtain a translated OCR text. The technique can include outputting, from the server to the mobile computing device, the translated OCR text.

Abstract: A computer-implemented technique can include receiving a machine translation input specifying (i) a source text, (ii) a source language of the source text, and (iii) a target language for the source text, and obtaining a machine translation of the source text from the source language to the target language to obtain a translated source text. The technique can include determining whether to swap the source and target languages based on (i) the source text and (ii) at least one language model, and in response to determining to swap the source and target languages: swapping the source and target languages to obtain modified source and target languages, utilizing the translated source text as a modified source text, obtaining a machine translation of the modified source text from the modified source language to the modified target language to obtain a translated modified source text, and outputting the translated modified source text.

Abstract: A technique for selectively distributing OCR and/or machine language translation tasks between a mobile computing device and server(s) includes receiving, at the mobile computing device, an image of an object comprising a text. The mobile computing device can determine a degree of optical character recognition (OCR) complexity for obtaining the text from the image. Based on this degree of OCR complexity, the mobile computing device and/or the server(s) can perform OCR to obtain an OCR text. The mobile computing device can then determine a degree of translation complexity for translating the OCR text from its source language to a target language. Based on this degree of translation complexity, the mobile computing device and/or the server(s) can perform machine language translation of the OCR text from the source language to a target language to obtain a translated OCR text. The mobile computing device can then output the translated OCR text.

Abstract: A technique for selectively distributing OCR and/or machine language translation tasks between a mobile computing device and server(s) includes receiving, at the mobile computing device, an image of an object comprising a text. The mobile computing device can determine a degree of optical character recognition (OCR) complexity for obtaining the text from the image. Based on this degree of OCR complexity, the mobile computing device and/or the server(s) can perform OCR to obtain an OCR text. The mobile computing device can then determine a degree of translation complexity for translating the OCR text from its source language to a target language. Based on this degree of translation complexity, the mobile computing device and/or the server(s) can perform machine language translation of the OCR text from the source language to a target language to obtain a translated OCR text. The mobile computing device can then output the translated OCR text.

Abstract: A computer-implemented technique can include receiving a selection by a user of a single word in a document in a source language, the document being displayed in a viewing application executing at the computing device, obtaining contextual information from the document that is indicative of a context of the selected word, providing the selected word and its contextual information from the viewing application to a different translation application, obtaining potential translated words using the translation application, the selected word, and its contextual information, each potential translated word being a potential translation of the selected word to a different target language that is preferred by the user, and displaying the potential translated words.