Abstract: An online system receives information describing a target group of online system users from a third party system that includes one or more user properties, which may identify an action performed by a an online user within a time interval to include the user in the target group. When an opportunity to present advertisement requests to a user is identified, the online system retrieves actions associated with the user and times associated with the actions. If an action associated with the user matches the action identified by the user properties and occurred within the time interval identified by the user properties, and other characteristics of the user satisfy at least a threshold number of the user properties, the online system includes the user in the target group. Advertisement requests having targeting criteria, including the target group, satisfied by characteristics of the user are evaluated for presentation to the user.

Abstract: This disclosure describes techniques for recognizing gestures performed by a user, including techniques for conserving power when performing finger or hand gesture recognition operations that involve processing electromyography (EMG) data. In one example, a wearable device capable of being worn by a user comprises: a motion detector configured to detect motion of the wearable device; a tissue movement sensor configured to collect tissue movement information associated with motion of muscles or tissues beneath the user's skin; and a gesture detection module.

Abstract: The disclosed visible-light barrier may include a light-scattering layer that preferentially scatters visible light over infrared light. The light-scattering layer may include (1) a substantially transparent or translucent material and (2) at least one of TiO2 or ZnO particles dispersed within the substantially transparent or translucent material at a concentration of from approximately 0.02 wt % to approximately 2 wt %. Various other articles, devices, systems, and methods are also disclosed.

Abstract: In one example, an apparatus comprises: a plurality of photodiodes, one or more charge sensing units, one or more analog-to-digital converters (ADCs), and a controller. The controller is configured to: enable the each photodiode to generate charge in response to a different component of the incident light; transfer the charge from the plurality of photodiodes to the one or more charge sensing units to convert to voltages; receive a selection of one or more quantization processes of a plurality of quantization processes corresponding to a plurality of intensity ranges; based on the selection, control the one or more ADCs to perform the selected one or more quantization processes to quantize the voltages from the one or more charge sensing units to digital values representing components of a pixel of different wavelength ranges; and generate a pixel value based on the digital values.

Abstract: A virtual reality (VR) system is described that enables users to create, model, and visualize augmented reality experiences at physical, real-world locations, without needing to physically be at the location for which the augmented experience is being modeled. In one example, the VR system includes a head-mounted device (HMD), an augmented reality content development system, and a rendering engine. The HMD is configured to output VR content representing a physical environment for an augmented reality experience. The augmented reality content development system is configured to identify, based on an input, augmented reality content with which to augment the physical environment for the augmented reality experience. The rendering engine is configured to render, via the HMD, the VR content representing the physical environment, augmented with the augmented reality content.

Abstract: Various embodiments set forth a foveated display system and components thereof. The foveated display system includes a peripheral display module disposed in series with a foveal display module. The peripheral display module is configured to generate low-resolution, large field of view imagery for a user's peripheral vision. The foveal display module is configured to perform foveated rendering in which high-resolution imagery is focused towards a foveal region of the user's eye gaze. The peripheral display module may include a diffuser that is disposed within a pancake lens, which is a relatively compact design. The foveal display module may include a Pancharatnam-Berry Phase grating stack that increases the steering range of a beam-steering device such that a virtual image can be steered to cover an entire field of view visible to the user's eye.

Abstract: Systems on a chip (SoCs) include security logic configured to increase resistance to fault injection attacks (FIAs). The security logic includes a monitoring circuit and a cascaded series of substitution-boxes (S-Boxes) having a circuit delay that is designed to match (or most closely match) the computing device critical path length. The monitoring circuit monitors the number of iterations required for the cascaded series of S-Boxes to return to an initial value and generates an error signal if the monitored loop length is different from the expected loop length. In some examples, the error signal is received by a mitigation processor that executes one or more processes aimed at mitigating the attack.

Abstract: Various embodiments set forth high-resolution liquid crystal displays and components thereof. In some embodiments, light emitted by a high-resolution green color liquid crystal display is combined, via a combiner, with light emitted by at least one lower-resolution red and blue color liquid crystal display. The red and blue color display(s) may include a single display or two displays positioned on opposing sides of the combiner. The combiner may be a dichroic or polarization-based combiner. Combined light from the green color display and the red and blue color display(s) is passed through collimating optics, such as a pancake lens or a Fresnel lens, toward a viewer's eye.

Abstract: The disclosed computer-implemented method may include a fluidic device comprising a chamber, an inlet port coupled to the chamber and configured to convey fluid to the chamber, and an outlet port coupled to the chamber and configured to convey the fluid from the chamber. The fluidic device may also have a restricting region that (1) is dimensioned to restrict a flow of the fluid through the outlet port when the pressure in the chamber is below a threshold level and (2) is configured to move in a manner that allows a flow rate of the fluid through the outlet port to increase when pressure in the chamber reaches the threshold level.

Abstract: One embodiment of the present disclosure sets forth a method that includes receiving one or more two-dimensional images of one or more light patterns incident on an eye proximate to an eye region of a near-eye display device, and computing a gaze direction associated with the eye based on the one or more two-dimensional images.

Abstract: Embodiments described herein pertain to a machine-learning approach for shading. A system may determine a number of pixels associated with a viewpoint of a viewer. The system may determine, for each of the pixels, (1) a view direction based on the viewpoint and a pixel position of that pixel and (2) and a surface orientation of a surface visible to that pixel. The system may generate, using a first machine-learning model, a latent space representation of ambient lighting information associated with the pixels based on respective view directions and surface orientations. The system may determine color values for the pixels by processing the latent space representation of ambient lighting information using a second machine-learning model.

Abstract: An exemplary imaging device includes an optical sensor having an optical axis, a lens positioned to focus light from a scene onto the optical sensor, a matrix of variable-phase optical elements that are dimensioned to introduce at least two different phase delays into a wavefront of a light signal received from the scene through the lens, a housing that secures the matrix of variable-phase optical elements between the optical sensor and the lens, and a processing subsystem programmed to determine a phase difference associated with the light signal based on the at least two different phase delays. Related systems and methods are also provided.

Abstract: In one embodiment, a computing system may receive a target image with a first number of bits per color and access a seed mask from a storage media. The system may generate a set of masks based on the seed mask. Each of the masks may include a number of first dot patterns that observe a spatial stacking property. The system may generate a number of images based on the target image and the set of masks. Each of the images may have a second number of bits per color smaller than the first number of bits per color. The system may display the images sequentially in time domain on a display for representing the target image. The images may have a number of second dot patterns for representing corresponding grayscale values. The second dot patterns of the images may observe a temporal stacking property across the images.

Abstract: In one embodiment, a computing system may access a video including a first frame and a second frame. The computing system may determine first sampling locations for the first frame and determine second sampling locations for the second frame by transforming the first sampling locations to the second frame according to an optical flow between the first frame and the second frame. The computing system may select a subset of the second sampling locations based on a comparison between pixels in the first frame corresponding to the first sampling locations and pixels in the second frame corresponding to the second sampling locations. The computing system may define one or more rejection areas in the second frame based on the subset of the second sampling locations to determine third sampling locations in areas outside of the rejection areas. The computing system may generate a sample of the video.

Abstract: A method for providing imagery to a user on a display includes receiving head tracking data and using the head tracking data to determine a change in an orientation of a user's head. The method also includes blending pixels of a previous frame with corresponding pixels of a current frame. The corresponding pixels of the current frame are identified using the change in the orientation of the user's head. The method also includes providing foveated imagery to the user on the display using the blended pixels and multiple tiles.

Abstract: This disclosure describes efficient communication of surface texture data between system on a chip (SOC) integrated circuits. An example system includes a first integrated circuit, and at least one second integrated circuit communicatively coupled to the first integrated circuit by a communication interface. The first integrated circuit, upon determining that surface texture data of a frame to be rendered for display by the second SoC integrated circuit is to be updated, (a) transmits the surface texture data in one or more update packets to the second integrated circuit using the communication interface, and (b) transmits a command to the second integrated circuit indicating that the surface texture data of the frame has been updated using the communication interface.

Abstract: An artificial reality system is described that renders, presents, and controls user interface elements within an artificial reality environment, and performs actions in response to one or more detected gestures of the user. In one example, an artificial reality system comprises a head-mounted display configured to output artificial reality content; a stylus; a stylus action detector configured to detect movement of the stylus, detect a stylus selection action, and after detecting the stylus selection action, detect further movement of the stylus; a UI engine configured to generate stylus movement content in response to detecting movement of the stylus, and generate a UI input element in response to detecting the stylus selection action; and a rendering engine configured to render the stylus movement content and the UI input element as overlays to the artificial reality content, and update the stylus movement content based on the further movement of the stylus.

Abstract: In one embodiment, a computing device may determine a virtual content to be displayed with a scene of a real-world environment. The device may generate an image depicting the virtual content. Using one or more sensors, the device may detect characteristics of the scene of the real-world environment. Based on the image and the characteristics of the scene, the device may determine that a visual enhancement is to be applied to the virtual content depicted in the image to enhance a contrast between the depicted virtual content and the scene. The device may generate a visually-enhanced image depicting the virtual content by applying the visual enhancement to the virtual content depicted in the image. The device may display the visually-enhanced image of the virtual content on a display of the computing device, wherein the scene of the real-world environment is visible through the display.

Abstract: In general, this disclosure describes techniques for using a random number generator to affect the lengths of clock cycles in a clock waveform that drives the timing of operations performed by processing circuitry. In one example, the processing circuitry includes a central processing unit and a clock generator. The clock generator is configured, upon receiving an indication of a boot command for the processing circuitry, generate a random number using a true random number generator and generate, based at least in part on the random number, an output clock waveform indicating at least a length of a clock cycle for the central processing unit. The central processing unit is configured to execute a boot sequence for at least the processing circuitry using the output clock waveform.

Abstract: In one embodiment, a method includes projecting a triangle primitive of an object defined in three-dimensional space onto a two-dimensional space, accessing a distortion map generated based on distortion characteristics of a display system, distorting a grid representation of a screen of the display system using the distortion map, determining a visibility of the triangle primitive relative to pixels of the screen by comparing the projected triangle primitive and the distorted grid representation of the screen, rendering an image based on the determined visibility of the triangle primitive, the rendered image being configured to be displayed by the screen of the display system having the distortion characteristics.

Abstract: An optical device for a head-mounted display device includes a first partial reflector and a second partial reflector positioned relative to the first partial reflector so that the second partial reflector receives first light transmitted through the first partial reflector and reflects at least a portion of the first light toward the first partial reflector as second light. At least a portion of the second light is reflected by the first partial reflector as third light, and at least a portion of the third light is transmitted through the second partial reflector. At least one of the first partial reflector or the second partial reflector includes a reflective holographic element.

Abstract: An audio system includes a sound output device, a microphone, and processing circuitry. The microphone is configured to capture environmental audio. The processing circuitry is configured to analyze the environmental audio to identify one or more properties of environmental audio conditions. The processing circuitry is configured to adjust one or more sound presentation parameters based on the one or more properties of the environmental audio conditions to account for the environmental audio conditions. The processing circuitry is configured to operate the sound output device to output audio according to the one or more sound presentation parameters.

Abstract: In one embodiment, a system may access an input image of an object captured by cameras, and the input image depicts appearance information associated with an object. The system may generate a first mesh of the object based on features identified from the input image of the object. The system may generate, by processing the first mesh using a machine-learning model, a position map that defines a contour of the object. Each pixel in the position map corresponds to a three-dimensional coordinate. The system may further generate a second mesh based on the position map, wherein the second mesh has a higher resolution than the first mesh. The system may render an output image of the object based on the second mesh. The system disclosed in the present application can render a dense mesh which has a higher resolution to provide details which cannot be compensated by texture information.

Abstract: In general, this disclosure describes a suspend mode feature for artificial reality systems and, more specifically, system configurations and techniques for generating a suspend mode environment during execution of an artificial reality application, and presenting and controlling user interface (UI) elements within the suspend mode environment. The suspend mode feature enables a user interacting with one or more other users (or one or more avatars corresponding to the one or more other users) within an artificial reality (AR) environment to suspend the AR environment from the user's perspective. While in the suspend mode environment, the user is able to view and select the other users from the AR environment. For example, the user may select another user to friend, message, mute, block, or report that user from the suspend mode environment.

Abstract: In one embodiment, one or more computing systems executes a computer program defining (1) a bound value that references a bound variable and (2) a terminating bound value that references computation instructions for deriving a runtime value from at least the bound value of the bound variable. A computation orchestration layer of a programming framework initializes, according to the computer program, a subscription tracker that tracks subscription relationships between the bound variable and the terminating bound value. The computation orchestration layer uses, in response to receiving an indication of an update to the bound value of the bound variable, the subscription tracker to determine that the terminating bound value subscribes to updates to the bound value of the bound variable. The computation orchestration layer sends an update stream associated with the update to a callback function associated with the terminating bound value.

Abstract: In one embodiment, a method includes displaying, for one or more displays of a virtual VR device, a first output image comprising a passthrough view of a real-world environment. The method includes identifying, using one or more images captured by one or more cameras of the VR display device, a real-world object in the real-world environment. The method includes receiving a user input indicating a first dimension corresponding to the real-world object. The method includes automatically determining, based on the first dimension, a second and third dimension corresponding to the real-world object. The method includes rendering, for the one or more displays of the VR display device, a second output image of a VR environment. The VR environment includes a MR object that corresponds to the real-world object. The MR object is defined by the determined first, second, and third dimensions.

Abstract: A wearable device equipped with sensors or transducers capable of detecting a vibration propagating through two different mediums. In some embodiments, the two mediums are air and a surface contacted by a user. In such an embodiment, the sensors or transducers include a microphone and an accelerometer. In some embodiments, the device may perform some or all of the signal processing and/or logic used to determine that a contact occurred and the location of the contact using a processor programmed with a set of computer-executable instructions. In some embodiments, the device may transmit data or signals to a processor located in a different device, such as a mobile phone or artificial reality headset, for signal processing and/or the execution of logic.

Abstract: An example device may include a light source, an optical element, and an encapsulant layer. A light beam generated by the light source may be received by the optical element, and redirected into the encapsulant layer. The optical element may include a high-index material, for example, with a refractive index of at least approximately 1.5 at the wavelength of the light beam. The light source may be a semiconductor light source, such as a light emitting diode or a laser. The optical element may be embedded in the encapsulant layer, and the optical element may have a curved exit surface. Refraction at the exit surface of the optical element may redirect the light beam towards a target.

Abstract: A method is described for utilizing NIL materials with switchable mechanical properties. The method comprises applying an imprint mask to a nano-imprint lithography (NIL) material layer. The NIL material layer is comprised of a NIL material with a modulus level below a flexibility threshold. The NIL material layer has an internal property, that when changed, causes a change in the modulus level of the NIL material. The method further comprises detaching the imprinted NIL material layer from the imprint mask, with the low modulus level of the NIL material causing a shape of the imprinted NIL material layer to remain unchanged after detachment. A modulus level of the NIL material is increased by changing an internal property of the NIL material, with the modulus level increased beyond a strength threshold to create a first imprint layer that has a structure that remains unaffected by a subsequent process.

Abstract: Disclosed herein are related to a system and a method for controlling a virtual reality based on a physical object. In one aspect, a shape of a hand of a user corresponding to a surface or a structure of a physical object is detected. In one aspect, according to the detected shape of the hand, an interactive feature for the surface or the structure of the physical object is generated in a virtual reality or augmented reality application. In one aspect, a user interaction with the interactive feature is detected. In one aspect, an action of the virtual reality or augmented reality application is initiated, in response to detecting the user interaction with the interactive feature.

Abstract: A method for using a virtual camera location to display image data to one device from another device associated with a plurality of cameras. The method includes receiving image data from a plurality of cameras associated with a second user device associated with a second user. The method further includes establishing a virtual camera location different from positions of the plurality of cameras. The method further includes providing an image using the image data on a first user display of a first user device based at least on a viewpoint of the virtual camera location.

Abstract: In one embodiment, a computing system may determine, for a current frame to be displayed and using an eye tracking system, a current eye position of a viewer. The system may determine a first array of scaling factors based on the determined current eye position of the viewer. The system may retrieve one or more second arrays of scaling factors used for correcting one or more proceeding frames of the current frame. The system may determine a third array of scaling factors based on the first array of scaling factors determined based on the current eye position and the one or more second arrays of scaling factors used for correcting the proceeding frames. The system may adjust pixel values of the current frame based at least on the third array of scaling factors. The system may output the current frame with the adjusted pixel values to a display.

Abstract: In general, the disclosure describes techniques for wireless communications between multiple devices of an artificial reality system using virtual channels. In one example, a first computing device of a multi-device system, comprising: one or more first processors coupled to one or more memory devices; and a supervisory processor configured to establish a plurality of virtual channels over a physical wireless channel between the first computing device and a second computing device, wherein each of the virtual channels is associated with a different pair of processors comprising a source processor selected from one of the first processors or second processors of the second computing device and a destination processor selected from one of the first processors or the second processors, wherein, for each virtual channel and associated pair of processors, the source processor is configured to communicate application data with the destination processor using the virtual channel.

Abstract: In one embodiment, a computing system may access a first mask associated with a first color component and a first pixel in an image, and a second mask associated with a second color component and a second pixel in the image. The system may access first component values of the first color component in a first pixel region containing the first pixel, and second component values of the second color component in a second pixel region containing the second pixel. The system may modify the first component values using the first mask, and the second component values using the second mask. The system may cause the modified first and second component values to be displayed by light-emitting elements of the first and second color components. The first and second masks may be generated based on relative positions of the first light-emitting elements and the second light-emitting elements.

Abstract: This disclosure describes various examples of a system which uses a multi-bank, multi-port shared memory system that may be implemented as part of a system on a chip. The shared memory system may have particular applicability in the context of an artificial reality system, and may be designed to have distributed or varied latency for one or more memory banks and/or one or more components or subsystems within the system on a chip. The described shared memory system may be logically a single entity, but physically may have multiple memory banks, each accessible by any of a number of components or subsystems. In some examples, the memory system may enable concurrent, common, and/or shared access to memory without requiring, in some situations, full locking or arbitration.

Abstract: In some examples, a method includes forming a material layer on a substrate, partially polymerizing a component of the material layer, to form fluid-filled droplets within a partially polymerized matrix, deforming the material layer to form anisotropic fluid-filled droplets, and further polymerizing the partially polymerized matrix to form an anisotropic voided polymer, including anisotropic voids in a polymer matrix. The anisotropic voids may include anisotropic nanovoids. Example methods may further include depositing electrodes on the anisotropic voided polymer so that at least a portion of the anisotropic voided polymer is located between the electrodes. Examples may include forming electroactive elements including an anisotropic nanovoided polymer, and devices (such as sensors and/or actuators) including electroactive elements.

Abstract: Aspects of the present disclosure are directed to an artificial reality system orchestrating interactions between virtual object “augments.” The orchestration can include linking, which can be forming two or more augments into a combination, embedding an augment within an existing combination, or triggering an action mapped to the linking of those augments. Another type of orchestration can include extracting, which can refer to taking an augment out of an existing combination, either by removing it from the combination or copying the augment to leave a version in the combination and having another version outside the combination.

Abstract: In one embodiment for generating passthrough, a computing system may access images of an environment captured by cameras of a device worn by a user. The system may generate, based on the images, depth measurements of objects in the environment. The system may generate a mesh covering a field of view of the user and then update the mesh based on the depth measurements to represent a contour of the objects in the environment. The system may determine a first viewpoint of a first eye of the user and render a first output image based on the first viewpoint and the updated mesh. The system may then display the first output image on a first display of the device, the first display being configured to be viewed by the first eye of the user.

Abstract: Disclosed herein are a system, a method and a device for switching between different communications protocols. A head wearable display can transmit, to a console during a first session interval, a first request to switch from a first link using a first communication protocol to a second link using a second communication protocol. The first request can include at least one characteristic of the first link. The head wearable display can receive, from the console, a first response to the first request, to perform a switch from the first link to the second link. The first response can indicate a wake up time and a second session interval to perform the switch. The head wearable display can switch from the first link to the second link in the second session interval and after the wake up time.

Abstract: In some examples, a device includes a multilayer structure, a first electrode, and a second electrode, where the multilayer structure is located at least in part between the first electrode and the second electrode, and the multilayer structure includes a nanovoided polymer layer, and a solid layer. The solid layer may include a non-nanovoided layer. The nanovoided polymer layer may be an electroactive layer. The device may further include a control circuit configured to apply an electrical potential between the first electrode and the second electrode, which may induce a mechanical deformation of the multilayer.

Abstract: Systems and methods for rendering VR content at a VR rendering device include identifying, by a content facilitator, that that first content presented via a first device to a user includes metadata indicating availability of second content. The second content includes at least one of a three-dimensional (3D) video, 360° image, or virtual reality (VR) content. The content facilitator provides an interface element to the user at the first device responsive to the metadata indicating availability of the second content. The content facilitator receives a user selection of the interface element. The content facilitator provides access to the second content via a VR rendering device for the user responsive to the user selection. The VR rendering device is configured to render the second content.

Abstract: Methods and systems for integrating functionality of XR devices and non-XR devices are disclosed herein. An XR device can identify and obtain one or more content items being displayed on non-XR devices and create virtual objects associated with the content item in an artificial reality environment of the XR device. A user of the XR device can interact with the virtual objects, such as by moving the virtual objects to new locations within the artificial reality environment or interact with the virtual objects to view the content items. The user of the XR device can also provide one or more gestures or commands to send content items or identifiers of content items associated with the virtual objects to non-XR devices for display at the non-XR devices.

Abstract: A method may include displaying, to a user, a first color in a first area and a second color in a second area, where (1) the second color has a longer wavelength than the first color and (2) the first and second color have an expected longitudinal chromatic aberration for a human eye. The method may also include receiving, from the user, an indication of whether the user perceives (1) the first area as being clearer than the second area or (2) the second area as being clearer than the first area. The method may further include determining, based on the indication of the user and the expected longitudinal chromatic aberration, information about a refractive error of the user's vision. Various other methods, systems, and devices are also disclosed.

Abstract: Disclosed herein are related to a system and a method of remotely rendering artificial reality. In one approach, a console determines an expected transmission end time of a first session for wireless transmission of image data indicating an image of artificial reality to a head wearable display. In one approach, the console determines an expected process end time of generating additional data associated with the image of the artificial reality. In one approach, the console compares the expected transmission end time and the expected process end time. The console may transmit, to the head wearable display in the first session, the additional data together with the image data, in response to the expected process end time being less than the expected transmission end time.

Abstract: A method includes projecting a single color illumination light having a first color on a liquid crystal on silicon display device, thereby obtaining a single color image light having the first color from the liquid crystal on silicon display device. The method also includes receiving image light having at least a second color that is different from the first color from a display panel that is different from a liquid crystal on silicon display device and combining the single color image light and the image light for projection toward an eye.

Abstract: In one embodiment, a method includes accessing one or more surfaces of an artificial reality scene. The one or more surfaces are generated based on one or more images of the artificial reality scene rendered at a first rate and based on a first viewing position. The method includes generating subframes at a second rate higher than the first rate. Each of the subframes is generated by determining a second viewing position, determining a visibility of the one or more surfaces of the artificial reality scene from the second viewing position, generating, based on the determined visibility of the one or more surfaces, color values of the subframe corresponding to output positions of a display, and providing the color values of the subframe for display.

Abstract: Methods and systems for providing illumination for depth sensing are provided. In one example, an apparatus comprises an illuminator, an optical sensor, and a controller. The illuminator comprises a photonic integrated circuit (PIC) and a diffractive optical element (DOE). The PIC including a laser source and at least one waveguide including optical turning features. The at least one waveguide propagates light transmitted by the laser source along a first axis parallel to a top surface of the PIC. The optical turning features diverts the light to form collimated light beams to exit the at least one waveguide and the top surface along a second axis. The DOE can diffract and project the collimated light beams. The optical sensor can detect the collimated light beams reflected off an object. The controller can determine a depth of the object with respect to the apparatus based on the detection.

Abstract: Systems and methods for generating composite sets of data based on sensor data from different sensors are disclosed. Exemplary implementations may capture a color image including chromatic information; capture a depth image; generate inertial signals conveying values that are used to determine motion parameters; determine the motion parameters based on the inertial signals; generate a re-projected depth image as if the depth image had been captured at the same time as the color image, based on the interpolation of motion parameters; and generate a composite set of data based on different kinds of sensor data by combining information from the color image, the re-projected depth image, and one or more motion parameters.

Abstract: In one embodiment, a method includes receiving, from a client system associated with a first user, a first audio input. The method includes generating multiple transcriptions corresponding to the first audio input based on multiple automatic speech recognition (ASR) engines. Each ASR engine is associated with a respective domain out of multiple domains. The method includes determining, for each transcription, a combination of one or more intents and one or more slots to be associated with the transcription. The method includes selecting, by a meta-speech engine, one or more combinations of intents and slots from the multiple combinations to be associated with the first user input. The method includes generating a response to the first audio input based on the selected combinations and sending, to the client system, instructions for presenting the response to the first audio input.