Abstract: Methods and systems for generating a graphical user interface for a user task are described. An example method for generating a graphical user interface for a user task includes receiving, from a user, a natural language prompt describing the user task, and generating, based on an output of a large language model configured to process the natural language prompt, a task-driven data model comprising a task object that includes multiple entity objects and multiple dependencies between objects. The method then includes determining, for each attribute of at least one of the multiple entity objects, a first label indicative of a user interface component for the entity object, rendering the graphical user interface that includes a panel comprising the user interface component, and providing the graphical user interface to a display device. An example system includes one or more processors configured to implement the above-described method.

Abstract: Methods and systems for generating malleable overview-detail user interfaces are described. An example method includes displaying, on the graphical user interface, a webpage comprising an overview view that corresponds to an initial ordering of multiple items and at least one detail view that corresponds to a selected item, and each item comprising multiple attributes. The method includes receiving a first input that toggles an attributes mode, and automatically identifying one or more attributes of the plurality of attributes on the graphical user interface. Then, the method includes receiving a second input that identifies a particular attribute from the one or more attributes, and modifying the graphical user interface to display the initial ordering of the plurality of items such that the particular attribute is displayed or hidden for each item in the overview view. An example system includes at least one processor configured to implement the above-described method.

Abstract: In various examples, a video effect is displayed in a live video stream in response to determining a portion of an audio stream of the live video stream corresponds to a text segment of a script associated with the video effect and detecting performance of a gesture. For example, during presentation of the script, the audio stream is obtained to determine if a portion of the audio stream corresponds to the text segment. In response to the portion of the audio stream corresponding to the text segment, detecting performance of a gestures and causing the video effect to be displayed.

Abstract: A computer system for generating graphics content receives text or information specifying an amount of spoken language and uses NLP to extract linguistic structures associated with the text or the amount of spoken language to determine mappings between the linguistic structures and the graphics content based at least in part on a predefined grammar. The predefined grammar may specify a target context for matching to arguments associated with the linguistic structures and may specify one or more corresponding graphics elements having associated appearances, layouts and/or graphics effects. The computer system then generates the graphics content associated with the text or the amount of spoken language.

Abstract: A method and system for managing workflows receives a text string being typed within a data document and executes a connection engine that performs natural language processing (NLP) to extract words and phrases having keywords corresponding to data operations, parse the text string into nested nodes including sub-phrases of arguments and keywords. The arguments and keywords are assembled into one or more complete data operation which is executed to return matching results from within a dataset as dependent phrase candidates to complete the text string. The writer selects a candidate from the dependent phrase candidates in response to which the connection engine creates a persistent text-data connection between the selected candidate and the dataset. This persistent text-data connection automatically updates the selected candidate when one or more of the dataset, arguments, and keywords are modified.

Abstract: A finger-mounted stylus for performing touch-based input on a touchscreen includes a fingertip case configured to attach to a user fingertip, an extension arm that is coupled to the fingertip case and includes a conductive tip, wherein the extension arm is configured to position the conductive tip away from the fingertip case, and control circuitry configured to apply an electric charge to the conductive tip when the conductive tip is in contact with or proximate to the touchscreen.

Abstract: The unified system for bimanual interactions provides a lightweight and integrated interface that allows the user to efficiently interact with and manipulate content in the user interface. The system is configured to detect a multi-finger interaction on the touchscreen and to differentiate whether the user intends to pan, zoom or frame a portion of the user interface. Generally, the framing interaction is identified by detection of the user's thumb and forefinger on the touchscreen, which cooperate to define a focus area between vectors extending outwards from the user's thumb and forefinger. Upon a determination that the user intends to interact with or manipulate content within the focus area, the unified system for bimanual interactions provides an indication of the objects that are located within the focus area and contextual menus for interacting with the objects.

Abstract: A system and method are disclosed for using a touch sensing system capable of sensing location of a finger or object above a touch surface to inform a touch response system in an electronic device of a predicted future user input event or motion data in advance of an actual touch event. Current user input is sensed via the touch sensing system and data reflecting hover information is created. A model of user interaction with a touch surface is applied to the data representative of the user input to create data reflecting a prediction of a future user input event. In an embodiment, prior to occurrence of the predicted user input event, a predicted location and a predicted time at which the predicted future user input event will occur are provided to a touch response system.

Abstract: A system and method are disclosed for using a touch sensing system capable of sensing location of a finger or object above a touch surface to inform a touch response system in an electronic device of a predicted future user input event or motion data in advance of an actual touch event. Current user input is sensed via the touch sensing system and data reflecting hover information is created. A model of user interaction with a touch surface is applied to the data representative of the user input to create data reflecting a prediction of a future user input event. In an embodiment, prior to occurrence of the predicted user input event, a predicted location and a predicted time at which the predicted future user input event will occur are provided to a touch response system.

Abstract: The unified system for bimanual interactions provides a lightweight and integrated interface that allows the user to efficiently interact with and manipulate content in the user interface. The system is configured to detect a multi-finger interaction on the touchscreen and to differentiate whether the user intends to pan, zoom or frame a portion of the user interface. Generally, the framing interaction is identified by detection of the user's thumb and forefinger on the touchscreen, which cooperate to define a focus area between vectors extending outwards from the user's thumb and forefinger. Upon a determination that the user intends to interact with or manipulate content within the focus area, the unified system for bimanual interactions provides an indication of the objects that are located within the focus area and contextual menus for interacting with the objects.

Abstract: A finger-mounted stylus for performing touch-based input on a touchscreen includes a fingertip case configured to attach to a user fingertip, an extension arm that is coupled to the fingertip case and includes a conductive tip, wherein the extension arm is configured to position the conductive tip away from the fingertip case, and control circuitry configured to apply an electric charge to the conductive tip when the conductive tip is in contact with or proximate to the touchscreen.

Abstract: A system and method are disclosed for using a touch sensing system capable of sensing location of a finger or object above a touch surface to inform a touch response system in an electronic device of a predicted future user input event or motion data in advance of an actual touch event. Current user input is sensed via the touch sensing system and data reflecting hover information is created. A model of user interaction with a touch surface is applied to the data representative of the user input to create data reflecting a prediction of a future user input event. In an embodiment, prior to occurrence of the predicted user input event, a predicted location and a predicted time at which the predicted future user input event will occur are provided to a touch response system.

Abstract: A system and method are disclosed for using a touch sensing system capable of sensing location of a finger or object above a touch surface to inform a touch response system in an electronic device of a predicted future user input event or motion data in advance of an actual touch event. Current user input is sensed via the touch sensing system and data reflecting hover information is created. A model of user interaction with a touch surface is applied to the data representative of the user input to create data reflecting a prediction of a future user input event. In an embodiment, prior to occurrence of the predicted user input event, a predicted location and a predicted time at which the predicted future user input event will occur are provided to a touch response system.

Abstract: A system and method are disclosed for using a touch sensing system capable of sensing location of a finger or object above a touch surface to inform a touch response system in an electronic device of a predicted future user input event or motion data in advance of an actual touch event. Current user input is sensed via the touch sensing system and data reflecting hover information is created. A model of user interaction with a touch surface is applied to the data representative of the user input to create data reflecting a prediction of a future user input event. In an embodiment, prior to occurrence of the predicted user input event, a predicted location and a predicted time at which the predicted future user input event will occur are provided to a touch response system.

Abstract: The unified system for bimanual interactions provides a lightweight and integrated interface that allows the user to efficiently interact with and manipulate content in the user interface. The system is configured to detect a multi-finger interaction on the touchscreen and to differentiate whether the user intends to pan, zoom or frame a portion of the user interface. Generally, the framing interaction is identified by detection of the user's thumb and forefinger on the touchscreen, which cooperate to define a focus area between vectors extending outwards from the user's thumb and forefinger. Upon a determination that the user intends to interact with or manipulate content within the focus area, the unified system for bimanual interactions provides an indication of the objects that are located within the focus area and contextual menus for interacting with the objects.

Abstract: The unified system for bimanual interactions provides a lightweight and integrated interface that allows the user to efficiently interact with and manipulate content in the user interface. The system is configured to detect a multi-finger interaction on the touchscreen and to differentiate whether the user intends to pan, zoom or frame a portion of the user interface. Generally, the framing interaction is identified by detection of the user's thumb and forefinger on the touchscreen, which cooperate to define a focus area between vectors extending outwards from the user's thumb and forefinger. Upon a determination that the user intends to interact with or manipulate content within the focus area, the unified system for bimanual interactions provides an indication of the objects that are located within the focus area and contextual menus for interacting with the objects.

Abstract: A finger-mounted stylus for performing touch-based input on a touchscreen includes a fingertip case configured to attach to a user fingertip, an extension arm that is coupled to the fingertip case and includes a conductive tip, wherein the extension arm is configured to position the conductive tip away from the fingertip case, and control circuitry configured to apply an electric charge to the conductive tip when the conductive tip is in contact with or proximate to the touchscreen.

Abstract: A finger-mounted stylus for performing touch-based input on a touchscreen includes a fingertip case configured to attach to a user fingertip, an extension arm that is coupled to the fingertip case and includes a conductive tip, wherein the extension arm is configured to position the conductive tip away from the fingertip case, and control circuitry configured to apply an electric charge to the conductive tip when the conductive tip is in contact with or proximate to the touchscreen.

Abstract: A system and method are disclosed for using a touch sensing system capable of sensing location of a finger or object above a touch surface to inform a touch response system in an electronic device of a predicted future user input event or motion data in advance of an actual touch event. Current user input is sensed via the touch sensing system and data reflecting hover information is created. A model of user interaction with a touch surface is applied to the data representative of the user input to create data reflecting a prediction of a future user input event. In an embodiment, prior to occurrence of the predicted user input event, a predicted location and a predicted time at which the predicted future user input event will occur are provided to a touch response system.