Abstract: Systems and methods for content snippet generation, storage, and suggestion can include obtaining a user input, segmenting a sub-portion of displayed content based on the user input, and generating a content snippet that includes the segmented content and source data associated with the displayed content. A generative model can be leveraged to categorize the content snippet, determine similar content, and determine when to surface the content snippet to a user.

Abstract: Visual search in an operating system of a computing device can process and provide additional information on the content being provided for display. The computing device can include an operating system that includes a visual search interface that obtains and processes display data associated with content currently being provided for display. The visual search interface can generate display data based on the current content provided for display, process the display data with one or more on-device machine-learned models, and provide additional information to the user. The visual search interface may transmit data associated with the display data to perform additional data processing tasks. Application suggestions may be determined and provided based on the visual search data.

Abstract: Systems and methods for unimodal visual search can include obtaining an image query, processing the image query to generate an intent determination, determining a particular prompt based on the intent determination, determining a plurality of search results, and processing the particular prompt, the image query and at least a subset of the plurality of search results to generate a model-generated response. The generative model can perform different response renderings based on the different prompts associated with different intents.

Abstract: Visual search in an operating system of a computing device can process and provide additional information on the content being provided for display. The computing device can include an operating system that includes a visual search interface that obtains and processes display data associated with content currently being provided for display. The visual search interface can generate display data based on the current content provided for display, process the display data with one or more on-device machine-learned models, and provide additional information to the user. The visual search interface may transmit data associated with the display data to perform additional data processing tasks. Application suggestions may be determined and provided based on the visual search data.

Abstract: Systems and methods for content processing can include obtaining a gesture input and display data, determining content selected by the gesture input, classifying the gesture, and performing a particular data processing action based on the content selection and the gesture classification. The particular data processing action can vary based on gesture classification. The content selection determination can include determining a gesture mask and then determining the features of the displayed content item that are within the gesture mask.

Abstract: Systems and methods for content processing can include obtaining a gesture input and display data, determining content selected by the gesture input, classifying the gesture, and performing a particular data processing action based on the content selection and the gesture classification. The particular data processing action can vary based on gesture classification. The content selection determination can include determining a gesture mask and then determining the features of the displayed content item that are within the gesture mask.

Abstract: Systems and methods for content processing can include obtaining a gesture input and display data, determining content selected by the gesture input, classifying the gesture, and performing a particular data processing action based on the content selection and the gesture classification. The particular data processing action can vary based on gesture classification. The content selection determination can include determining a gesture mask and then determining the features of the displayed content item that are within the gesture mask.

Abstract: Visual search in an operating system of a computing device can process and provide additional information on the content being provided for display. The computing device can include an operating system that includes a visual search interface that obtains and processes display data associated with content currently being provided for display. The visual search interface can generate display data based on the current content provided for display, process the display data with one or more on-device machine-learned models, and provide additional information to the user. The visual search interface may transmit data associated with the display data to perform additional data processing tasks. Application suggestions may be determined and provided based on the visual search data.

Abstract: Systems and methods for content snippet generation, storage, and suggestion can include obtaining a user input, segmenting a sub-portion of displayed content based on the user input, and generating a content snippet that includes the segmented content and source data associated with the displayed content. A generative model can be leveraged to categorize the content snippet, determine similar content, and determine when to surface the content snippet to a user.

Abstract: Visual search in an operating system of a computing device can process and provide additional information on the content being provided for display. The computing device can include an operating system that includes a visual search interface that obtains and processes display data associated with content currently being provided for display. The visual search interface can generate display data based on the current content provided for display, process the display data with one or more on-device machine-learned models, and provide additional information to the user. The visual search interface may transmit data associated with the display data to perform additional data processing tasks. Application suggestions may be determined and provided based on the visual search data.

Abstract: Visual search in an operating system of a computing device can process and provide additional information on the content being provided for display. The computing device can include an operating system that includes a visual search interface that obtains and processes display data associated with content currently being provided for display. The visual search interface can generate display data based on the current content provided for display, process the display data with one or more on-device machine-learned models, and provide additional information to the user. The visual search interface may transmit data associated with the display data to perform additional data processing tasks. Application suggestions may be determined and provided based on the visual search data.

Abstract: Visual search in an operating system of a computing device can process and provide additional information on the content being provided for display. The computing device can include an operating system that includes a visual search interface that obtains and processes display data associated with content currently being provided for display. The visual search interface can generate display data based on the current content provided for display, process the display data with one or more on-device machine-learned models, and provide additional information to the user. The visual search interface may transmit data associated with the display data to perform additional data processing tasks. Application suggestions may be determined and provided based on the visual search data.

Abstract: Visual search in an operating system of a computing device can process and provide additional information on the content being provided for display. The computing device can include an operating system that includes a visual search interface that obtains and processes display data associated with content currently being provided for display. The visual search interface can generate display data based on the current content provided for display, process the display data with one or more on-device machine-learned models, and provide additional information to the user. The visual search interface may transmit data associated with the display data to perform additional data processing tasks. Application suggestions may be determined and provided based on the visual search data.