Abstract: Embodiments of the present invention provide systems, methods, and computer storage media directed to providing talk back automation for applications installed on a mobile device. To do so actions (e.g., talk back features) can be created, via the digital assistant, by recording a series of events that are typically provided by a user of the mobile device when manually invoking the desired action. At a desired state, the user may select an object that represents the output of the application. The recording embodies the action and can be associated with a series of verbal commands that the user would typically announce to the digital assistant when an invocation of the action is desired. In response, the object is verbally communicated to the user via the digital assistant, a different digital assistant, or even another device. Alternatively, the object may be communicated to the same application or another application as input.

Abstract: Embodiments of the present invention provide systems, methods, and computer storage media directed to providing talk back automation for applications installed on a mobile device. To do so actions (e.g., talk back features) can be created, via the digital assistant, by recording a series of events that are typically provided by a user of the mobile device when manually invoking the desired action. At a desired state, the user may select an object that represents the output of the application. The recording embodies the action and can be associated with a series of verbal commands that the user would typically announce to the digital assistant when an invocation of the action is desired. In response, the object is verbally communicated to the user via the digital assistant, a different digital assistant, or even another device. Alternatively, the object may be communicated to the same application or another application as input.

Abstract: Computer-implemented methods for telemetry data processing and representations for foundation models are provided. Aspects include generating a graph data structure based on processing telemetry data associated with at least one computer system, where the graph data structure is representative of a set of events and a set of entities associated with the at least one computer system. Aspects include generating a textual representation of behavioral information associated with at least one entity of the set of entities and at least one event of the set of events, based on one or more subgraphs included in the graph data structure. Aspects include obtaining an analysis result associated with the at least one computer system based on processing the textual representation by one or more models.

Abstract: Techniques for verifying the identity of users transferring between devices are disclosed. An example method includes receiving device usage data by monitoring user-device interactions of a first set of users interacting with a first device and a second set of users interacting with a second device. The method also includes extracting features from the data and aggregating feature samples from the user-device interactions for first set of users and the second set of users. The method also includes generating a score for each feature based on an analysis the feature samples. The feature score indicates a degree to which the first set of feature samples and the second set of feature samples differentiate between the devices. The method also includes identifying features as transferrable if the score is below a specified threshold, and generating a new behavior model by modifying existing user behavior data according to the transferrable features.

Abstract: Embodiments of the present invention provide systems, methods, and computer storage media directed to providing talk back automation for applications installed on a mobile device. To do so actions (e.g., talk back features) can be created, via the digital assistant, by recording a series of events that are typically provided by a user of the mobile device when manually invoking the desired action. At a desired state, the user may select an object that represents the output of the application. The recording embodies the action and can be associated with a series of verbal commands that the user would typically announce to the digital assistant when an invocation of the action is desired. In response, the object is verbally communicated to the user via the digital assistant, a different digital assistant, or even another device. Alternatively, the object may be communicated to the same application or another application as input.

Abstract: A mobile application development system includes a developer portal that receives an application from a developer and provides a routing library to the developer to augment the application. An offline analysis system analyzes the application to (i) determine a set of activities that a handler within the application is programmed to resume in response to respective resumption requests from a host operating system and (ii) determine parameters for each of the activities. The offline analysis system generates a set of links that each corresponds to a respective one of the activities. The routing library, installed as part of the augmented application onto a user device, receives a link, from the user device's operating system, that identifies a first activity. The routing library includes instructions for generating a first resumption request based on parameters corresponding to the first activity and transmitting the first resumption request to the augmented application's handler.

Abstract: User verification using a mobile interaction can include receiving interaction data associated with a user of an application operating on a mobile device, the interaction data is generated during an interactive session of the user with the application on the mobile device. A plurality of features are extracted from the interaction data and one or more feature vectors are generated from the plurality of features. The plurality of features are aggregated to the one or more feature vectors and embedded within each feature vector. The embedded plurality of features are then projected to a global feature space by comparing a history of interactive sessions associated with the user and a history of interactive sessions associated with a plurality of different users for generating a global training dataset. Finally, a verification model is generated as a global binary classification model using the global training dataset.

Abstract: Embodiments of the present invention provide systems, methods, and computer storage media directed to providing talk back automation for applications installed on a mobile device. To do so actions (e.g., talk back features) can be created, via the digital assistant, by recording a series of events that are typically provided by a user of the mobile device when manually invoking the desired action. At a desired state, the user may select an object that represents the output of the application. The recording embodies the action and can be associated with a series of verbal commands that the user would typically announce to the digital assistant when an invocation of the action is desired. In response, the object is verbally communicated to the user via the digital assistant, a different digital assistant, or even another device. Alternatively, the object may be communicated to the same application or another application as input.

Abstract: Techniques for verifying the identity of users transferring between devices are disclosed. An example method includes receiving device usage data by monitoring user-device interactions of a first set of users interacting with a first device and a second set of users interacting with a second device. The method also includes extracting features from the data and aggregating feature samples from the user-device interactions for first set of users and the second set of users. The method also includes generating a score for each feature based on an analysis the feature samples. The feature score indicates a degree to which the first set of feature samples and the second set of feature samples differentiate between the devices. The method also includes identifying features as transferrable if the score is below a specified threshold, and generating a new behavior model by modifying existing user behavior data according to the transferrable features.

Abstract: Embodiments described herein are generally directed towards systems and methods relating to a crowd-sourced digital assistant and system. In particular, embodiments facilitate the intuitive creation and distribution of action datasets that include computing events or tasks that can be reproduced when an associated command, stored in an action dataset, is determined received by a digital assistant device. The digital assistant device described herein can generate new action datasets, on-board new action datasets, and receive new action datasets or updates to existing action datasets. Each digital assistant device in the described system can participate in the building of action datasets, so as to crowd-source a dialect that can be understood by a digital assistant device.

Abstract: User verification using a mobile interaction can include receiving interaction data associated with a user of an application operating on a mobile device, the interaction data is generated during an interactive session of the user with the application on the mobile device. A plurality of features are extracted from the interaction data and one or more feature vectors are generated from the plurality of features. The plurality of features are aggregated to the one or more feature vectors and embedded within each feature vector. The embedded plurality of features are then projected to a global feature space by comparing a history of interactive sessions associated with the user and a history of interactive sessions associated with a plurality of different users for generating a global training dataset. Finally, a verification model is generated as a global binary classification model using the global training dataset.

Abstract: Embodiments described herein are generally directed towards systems and methods relating to a crowd-sourced digital assistant and system. In particular, embodiments facilitate the intuitive creation and distribution of action datasets that include computing events or tasks that can be reproduced when an associated command, stored in an action dataset, is determined received by a digital assistant device. The digital assistant device described herein can generate new action datasets, on-board new action datasets, and receive new action datasets or updates to existing action datasets. Each digital assistant device in the described system can participate in the building of action datasets, so as to crowd-source a dialect that can be understood by a digital assistant device.

Abstract: Embodiments of the present invention provide systems, methods, and computer storage media directed to providing talk back automation for applications installed on a mobile device. To do so actions (e.g., talk back features) can be created, via the digital assistant, by recording a series of events that are typically provided by a user of the mobile device when manually invoking the desired action. At a desired state, the user may select an object that represents the output of the application. The recording embodies the action and can be associated with a series of verbal commands that the user would typically announce to the digital assistant when an invocation of the action is desired. In response, the object is verbally communicated to the user via the digital assistant, a different digital assistant, or even another device. Alternatively, the object may be communicated to the same application or another application as input.

Abstract: A mobile application development system includes a developer portal that receives an application from a developer and provides a routing library to the developer to augment the application. An offline analysis system analyzes the application to (i) determine a set of activities that a handler within the application is programmed to resume in response to respective resumption requests from a host operating system and (ii) determine parameters for each of the activities. The offline analysis system generates a set of links that each corresponds to a respective one of the activities. The routing library, installed as part of the augmented application onto a user device, receives a link, from the user device's operating system, that identifies a first activity. The routing library includes instructions for generating a first resumption request based on parameters corresponding to the first activity and transmitting the first resumption request to the augmented application's handler.

Abstract: A mobile application development system includes a developer portal that receives an application from a developer and provides a routing library to the developer to augment the application. An offline analysis system analyzes the application to (i) determine a set of activities that a handler within the application is programmed to resume in response to respective resumption requests from a host operating system and (ii) determine parameters for each of the activities. The offline analysis system generates a set of links that each corresponds to a respective one of the activities. The routing library, installed as part of the augmented application onto a user device, receives a link, from the user device's operating system, that identifies a first activity. The routing library includes instructions for generating a first resumption request based on parameters corresponding to the first activity and transmitting the first resumption request to the augmented application's handler.

Abstract: Embodiments of the present invention provide systems, methods, and computer storage media directed to providing talk back automation for applications installed on a mobile device. To do so actions (e.g., talk back features) can be created, via the digital assistant, by recording a series of events that are typically provided by a user of the mobile device when manually invoking the desired action. At a desired state, the user may select an object that represents the output of the application. The recording embodies the action and can be associated with a series of verbal commands that the user would typically announce to the digital assistant when an invocation of the action is desired. In response, the object is verbally communicated to the user via the digital assistant, a different digital assistant, or even another device. Alternatively, the object may be communicated to the same application or another application as input.

Abstract: A mobile application development system includes a developer portal that receives an application from a developer and provides a routing library to the developer to augment the application. An offline analysis system analyzes the application to (i) determine a set of activities that a handler within the application is programmed to resume in response to respective resumption requests from a host operating system and (ii) determine parameters for each of the activities. The offline analysis system generates a set of links that each corresponds to a respective one of the activities. The routing library, installed as part of the augmented application onto a user device, receives a link, from the user device's operating system, that identifies a first activity. The routing library includes instructions for generating a first resumption request based on parameters corresponding to the first activity and transmitting the first resumption request to the augmented application's handler.

Abstract: A search system includes a query receiver, a live scrape system, and a content provider. The query receiver is configured to receive a first query related to a first application state of a first application. The first query is based on a user-requested search from a user device. The live scrape system is configured to operate a device including an operating system in which an instance of the first application executes. The live scrape system is configured to, responsive to the first query, and within the device, (i) navigate to the first application state in the instance of the first application and (ii) extract content from the instance of the first application. The content provider is configured to provide the extracted content to the user device for presentation to a user of the user device in real time.

Abstract: A mobile application development system includes a developer portal that receives an application from a developer and provides a routing library to the developer to augment the application. An offline analysis system analyzes the application to (i) determine a set of activities that a handler within the application is programmed to resume in response to respective resumption requests from a host operating system and (ii) determine parameters for each of the activities. The offline analysis system generates a set of links that each corresponds to a respective one of the activities. The routing library, installed as part of the augmented application onto a user device, receives a link, from the user device's operating system, that identifies a first activity. The routing library includes instructions for generating a first resumption request based on parameters corresponding to the first activity and transmitting the first resumption request to the augmented application's handler.

Abstract: Embodiments described herein are generally directed towards systems and methods relating to a crowd-sourced digital assistant system and related methods. In particular, embodiments describe techniques for effectively searching, modifying, identifying parameter values, and determining features for selecting action datasets for distribution to digital assistant devices based on commands received therefrom. Action datasets include computing events or tasks that can be reproduced when a command is received by a digital assistant device and communicated to the server device.