Abstract: A method includes receiving, at an electronic device, a command directed to a first application operated by the electronic device. Features presented by the first application in response to interactions with the first application are captured at the electronic device. Data communicated with the first application via the interactions with the first application are captured at the electronic device. A task is learned based on the captured features and communicated data.

Abstract: Computer-based content understanding can include segmenting an image into a plurality of blocks, wherein each block includes textual information from the image. For each block of the plurality of blocks, encoded feature data is generated by encoding visual information of the block and visual information of one or more neighboring blocks from the plurality of blocks and encoded textual data is generated by encoding the textual information of the block and the textual information of the one or more neighboring blocks. Further, using an entity class prediction model, one or more tokens of the block are classified into one or more entity classes based on a combination of the encoded textual data and the encoded feature data. A plurality of entities can be extracted from the image based on the entity classes of the plurality of blocks.

Abstract: A method for learning a task includes capturing first information associated with at least one application executed by an electronic device. A sequence of user interface interactions for the at least one application is recorded. Second information are extracted from the sequence of user interface interactions. Events, action or a combination thereof are filtered from the second information using the first information. Recognition is performed on each element from the first information to generate a semantic ontology. An executable sequential event task bytecode is generated from each element of the semantic ontology and the filtered second information.

Abstract: A method includes receiving, at an electronic device, a command directed to a first application operated by the electronic device. Features presented by the first application in response to interactions with the first application are captured at the electronic device. Data communicated with the first application via the interactions with the first application are captured at the electronic device. A task is learned based on the captured features and communicated data.

Abstract: A method for learning a task includes capturing first information associated with at least one application executed by an electronic device. A sequence of user interface interactions for the at least one application is recorded. Second information are extracted from the sequence of user interface interactions. Events, action or a combination thereof are filtered from the second information using the first information. Recognition is performed on each element from the first information to generate a semantic ontology. An executable sequential event task bytecode is generated from each element of the semantic ontology and the filtered second information.