Abstract: A computer implemented method includes accessing training data that includes images, associated alternative text, and proximately located text providing instructions describing a user interface. The images are paired with text captions derived from the proximately located text and image's alternative text, training a vision and language model in a self-supervised manner using language masking, region masking, and image-text alignment techniques on respective image region features and tokenized text captions. Fine-tuning of the vision and language model is performed to obtain a specialized model representing user interface elements and associated functions.

Abstract: Systems and methods are described that are generally directed to generating a general task embedding representing task information. In examples, the generated task embedding may include predicted task information such that, rather being underspecified, the task embedding representative of the task may include additional specified information, where the task embedding can then be utilized in many different models and applications. Thus, task data may be received and at least a portion of the task data may be encoded using an encoder. Based on one or more outputs generated by the encoder and a type embedding associated with the task data, a task intent may be extracted or otherwise predicted based on the task data and one or more type encodings associated with the task data. The intent extractor may be trained on multiple auxiliary tasks with weak supervision that provide semantic augmentation to under-specified task texts.

Abstract: For a given input query specifying a task to be performed on a website, the correct sequence of actions (or UI script) is machine learned without having any previous knowledge about the website or the query. To learn the correct UI script, a task agent is created that performs multiple task agent runs comprising different sequences of actions of UI elements on the website (e.g., buttons, text fields, menus, and the like). The states of the webpages are monitored after each action of a UI element is performed. Tasklets are created that include the performed sequences of actions for each task agent as well as their assigned scores, and the correct UI script is chosen from the tasklets based on the scores (e.g., tasklet with the highest score).

Abstract: Systems and methods for extracting and sharing application-related user data are disclosed. A method may include extracting in-app data for at least one of the plurality of apps running on a computing device, the in-app data including content consumed by a user while the at least one app is running, and/or at least one user action taken in connection with the content. Using an entity template associated with the app, a plurality of text strings within the in-app data are classified into at least one of a plurality of data types specified by the template. At least one user data item (UDI) may be generated by combining at least a portion of the classified plurality of text strings, the at least one UDI being accessible by a second app, an operating system running on the, a service of the operating system, and/or a service running on at least another device.

Abstract: The automatic generation of one or more task-oriented conversational bots is disclosed. Illustratively, systems and methods are provided that allow for tracing the interactions of one or more computing applications inclusive of the interaction with one or more programmatic elements of the one or more computing applications, interaction with the graphical user interface(s) of the one or more computing applications, and/or the operation of the one or more computing environments on which the one or more computing applications are executing to collect various state data. The state data can be illustratively graphed to show the overall execution paths of one or more functions/operations of the one or more computing applications for use in generating one or more instructions representative of a desired task-oriented conversational bot that can be operatively executed through one or more application program interfaces of the one or more computing applications.

Abstract: Computer systems and methods for generating and interacting with a micro-service framework are provided. A micro-service corresponds to one or more deep link/API calls that carry out some particular function. A static analysis of an app is conducted, from one or more starting sources of the app to identify one or more valid and feasible execution paths, as well as corresponding input parameters within the app. Each valid execution path with corresponding input parameters represent a “deep link” or “API” for that app. The information regarding the deep link is collected and stored as a micro-service in a micro-service catalog. A micro-service framework is implemented that receives a micro-service request (i.e., a request that the micro-service be carried out on behalf of a computer user) from a UX client and executes that micro-service request via execution of the deep link.

Abstract: Various technologies described herein pertain to prefetching content units. A prefetch request is transmitted to a server from a client device. The prefetch request includes data indicative of probabilities of slots for content units being available during an upcoming time period. The probabilities can be based on likely interaction with application(s) executed by the client device during the upcoming time period. Prefetched content units assigned to the client device for the upcoming time period can be received from the server responsive to the prefetch request. One or more of the prefetched content units can be served for display on a display screen of the client device during execution the application(s). Further, statuses of the prefetched content units can be monitored, and information that specifies a subset of the prefetched content units that are unlikely to be displayed on the display screen prior to corresponding deadlines for expiration can be transmitted.

Abstract: The automatic generation of one or more task-oriented conversational bots is disclosed. Illustratively, systems and methods are provided that allow for tracing the interactions of one or more computing applications inclusive of the interaction with one or more programmatic elements of the one or more computing applications, interaction with the graphical user interface(s) of the one or more computing applications, and/or the operation of the one or more computing environments on which the one or more computing applications are executing to collect various state data. The state data can be illustratively graphed to show the overall execution paths of one or more functions/operations of the one or more computing applications for use in generating one or more instructions representative of a desired task-oriented conversational bot that can be operatively executed through one or more application program interfaces of the one or more computing applications.

Abstract: Various technologies described herein pertain to prefetching content units. A prefetch request is transmitted to a server from a client device. The prefetch request includes data indicative of probabilities of slots for content units being available during an upcoming time period. The probabilities can be based on likely interaction with application(s) executed by the client device during the upcoming time period. Prefetched content units assigned to the client device for the upcoming time period can be received from the server responsive to the prefetch request. One or more of the prefetched content units can be served for display on a display screen of the client device during execution the application(s). Further, statuses of the prefetched content units can be monitored, and information that specifies a subset of the prefetched content units that are unlikely to be displayed on the display screen prior to corresponding deadlines for expiration can be transmitted.

Abstract: Systems and methods for extracting and sharing application-related user data are disclosed. A method may include extracting in-app data for at least one of the plurality of apps running on a computing device, the in-app data including content consumed by a user while the at least one app is running, and/or at least one user action taken in connection with the content. Using an entity template associated with the app, a plurality of text strings within the in-app data are classified into at least one of a plurality of data types specified by the template. At least one user data item (UDI) may be generated by combining at least a portion of the classified plurality of text strings, the at least one UDI being accessible by a second app, an operating system running on the, a service of the operating system, and/or a service running on at least another device.

Abstract: Various technologies described herein pertain to allocating content units to client devices. Content units can be requested from an exchange by a server. The content units can be received by the server from the exchange, where the content units each has a corresponding deadline for expiration. Further, an indication of a penalty can be transmitted to the exchange if one of the content units is not displayed by one or more client devices upon expiration of the corresponding deadline.

Abstract: Systems and methods for extracting and sharing application-related user data are disclosed. A method may include extracting in-app data for at least one of the plurality of apps running on a computing device, the in-app data including content consumed by a user while the at least one app is running, and/or at least one user action taken in connection with the content. Using an entity template associated with the app, a plurality of text strings within the in-app data are classified into at least one of a plurality of data types specified by the template. At least one user data item (UDI) may be generated by combining at least a portion of the classified plurality of text strings, the at least one UDI being accessible by a second app, an operating system running on the, a service of the operating system, and/or a service running on at least another device.

Abstract: Various technologies described herein pertain to prefetching content units. A prefetch request is transmitted to a server from a client device. The prefetch request includes data indicative of probabilities of slots for content units being available during an upcoming time period. The probabilities can be based on likely interaction with application(s) executed by the client device during the upcoming time period. Prefetched content units assigned to the client device for the upcoming time period can be received from the server responsive to the prefetch request. One or more of the prefetched content units can be served for display on a display screen of the client device during execution the application(s). Further, statuses of the prefetched content units can be monitored, and information that specifies a subset of the prefetched content units that are unlikely to be displayed on the display screen prior to corresponding deadlines for expiration can be transmitted.

Abstract: Techniques are described for validating stateful app links. Validation can be performed when stateful app links are created, activated, shared, or at other times. Validation can be performed to determine whether a stateful app link has a dependency on a resource external to the mobile application. Validation can also be performed to detect other issues, such as security issues, privacy issues, or other issues.

Abstract: Computer systems and methods for generating and interacting with a micro-service framework are provided. A micro-service corresponds to one or more deep link/API calls that carry out some particular function. A static analysis of an app is conducted, from one or more starting sources of the app to identify one or more valid and feasible execution paths, as well as corresponding input parameters within the app. Each valid execution path with corresponding input parameters represent a “deep link” or “API” for that app. The information regarding the deep link is collected and stored as a micro-service in a micro-service catalog. A micro-service framework is implemented that receives a micro-service request (i.e., a request that the micro-service be carried out on behalf of a computer user) from a UX client and executes that micro-service request via execution of the deep link.

Abstract: The privacy-preserving cookie generator implementations described herein create a privacy-preserving data structure (also called a privacy-preserving cookie herein) that is used to provide personalization for online services without user tracking. In some implementations the privacy-preserving cookie generator encodes a user profile (for example, based on a user's online activity) into a data structure that has naturally occurring noise and that efficiently supports noise addition. In one implementation a Bloom filter is used to create the encoded profile. Additional noise is injected into the encoded profile to create an obfuscated user profile in the form of a privacy-preserving data structure. The privacy-preserving data structure or cookie can be attached to an online service request and sent over a network to an online service provider which can use it fulfill the services request, providing a somewhat personalized result while the user's privacy is maintained.

Abstract: Examples of the disclosure enable a plurality of applications to be managed. Some examples include a mobile device that receives a template associated with an application, captures one or more user interactions with the application, and determines one or more user interests by comparing the one or more user interactions with a plurality of entities included in the template. Examples of the disclosure enable the mobile device to analyze the user interactions to provide valuable information about user interest, preference, and/or intent and provide tailored services to a user.

Abstract: This document relates to trend response management. One example can detect a trending topic and identify content associated with the trending topic. The example can take an action relating to the content that decreases a trend-induced processing spike and/or enhances a user search experience associated with the trending topic.

Abstract: Techniques are described for validating stateful app links. Validation can be performed when stateful app links are created, activated, shared, or at other times. Validation can be performed to determine whether a stateful app link has a dependency on a resource external to the mobile application. Validation can also be performed to detect other issues, such as security issues, privacy issues, or other issues.

Abstract: Techniques are described for dynamically generating stateful app links. For example, page launcher events can be intercepted during user interaction with a mobile application. During the interaction, a request can be received to dynamically generate a stateful app link to a current page of the mobile application. In response to the request, a stateful app link to the current page can be created. The stateful app link can then be output. When the stateful app link is activated later, it returns to the current page of the mobile application. In some implementations, user interface events are also captured and used when creating the stateful app link.