Abstract: Techniques and solutions are described for improving user interfaces, such as by analyzing user interactions with a user interface with a machine learning component. The machine learning component can be trained with user interaction data that includes an interaction identifier and a timestamp. The identifiers and timestamps can be used to determine the duration of an interaction with a user interface element, as well as patterns of interactions. Training data can be used to establish baseline or threshold values or ranges for particular user interface elements or types of user interface elements. Test data can be obtained that includes identifiers and timestamps. The time taken to complete an interaction with a user interface element, and optionally an interaction pattern, can be analyzed. If the machine learning component determines that an interaction time or pattern is abnormal, various actions can be taken, such as providing a report or user interface guidance.

Abstract: Systems and methods include acquisition of data representing one or more user interactions with a user interface of an application, determination of a user workflow from a plurality of user workflows based on the acquired data, determination of one of a plurality of trained models based on the determined user workflow, each of the plurality of trained models associated with a respective one of the plurality of user workflows, and generation of an inference based on the data using the determined trained model.

Abstract: Technologies are provided for iterative machine learning and relearning. A training dataset can be divided into a plurality of training data blocks which can be used to generate separate machine learning models. The accuracy of the machine learning models can be assessed using a test dataset. Training data blocks which result in models with good accuracy can be combined into larger training data blocks which can then be used to generate new machine learning models. The process of combining training data blocks can be repeated as long as the resulting machine learning model has acceptable accuracy. However, if a model for a combined training data block has a poorer accuracy than the machine learning models for its component training data blocks, then the combined training data block and its machine learning model can be forgotten and its component training data blocks (and their associated machine learning models) can be relearned.

Abstract: A method of simplifying automated testing within an integrated development environment (IDE) for a user having a visual impairment is disclosed. An access mechanism is provided for selecting automated testing scripts from within the IDE by responding to audio communications describing one or more access commands. An execution mechanism is provided to the user for executing one or more of the selected automated testing scripts. A result mechanism is provided to the user, the result mechanism including a summary of a result of the executing of the one or more selected testing scripts, the summary based on an analysis of console output of the IDE, the analysis including performing feature extraction and natural language processing on the console output to generate a natural language description of the result. An action mechanism is provided to perform an additional action that is selected based on the type of the result.

Abstract: A method of simplifying automated testing within an integrated development environment (IDE) for a user having a visual impairment is disclosed. An access mechanism is provided for selecting automated testing scripts from within the IDE by responding to audio communications describing one or more access commands. An execution mechanism is provided to the user for executing one or more of the selected automated testing scripts. A result mechanism is provided to the user, the result mechanism including a summary of a result of the executing of the one or more selected testing scripts, the summary based on an analysis of console output of the IDE, the analysis including performing feature extraction and natural language processing on the console output to generate a natural language description of the result. An action mechanism is provided to perform an additional action that is selected based on the type of the result.

Abstract: Techniques and solutions are described for improving user interfaces, such as by analyzing user interactions with a user interface with a machine learning component. The machine learning component can be trained with user interaction data that includes an interaction identifier and a timestamp. The identifiers and timestamps can be used to determine the duration of an interaction with a user interface element, as well as patterns of interactions. Training data can be used to establish baseline or threshold values or ranges for particular user interface elements or types of user interface elements. Test data can be obtained that includes identifiers and timestamps. The time taken to complete an interaction with a user interface element, and optionally an interaction pattern, can be analyzed. If the machine learning component determines that an interaction time or pattern is abnormal, various actions can be taken, such as providing a report or user interface guidance.