Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for evaluating interactions with a user interface are disclosed. In one aspect, a method includes identifying a set of mutation events specifying changes to the structure of a user interface that occurred during the user session, and generating user interface states specifying different structures of the user interface throughout the given user session. Based at least in part on the user interface states, playback data that present visual changes of the user interface corresponding to the set of mutation events that occurred during the user session are generated. Session activity data describing user interactions that occurred during the user session are also generated. At least a portion of the playback data and the session activity data are output to a requesting device.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for evaluating interactions with a user interface of an application are disclosed. In one aspect, a method includes, for each of a plurality of different user sessions of a native application, accessing frame bundles that each include data representing content presented by a frame of a user interface of the native application at a given time. Each frame bundle includes at least a portion of a view tree of the native application used to generate the user interface at the given time and data specifying content presented by each view of the portion of the view tree. Based on the frame bundles, playback data are generated that present visual changes of the user interface corresponding to changes to the view trees.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for protecting user privacy in the playback of user sessions are described. In one aspect, a method includes accessing, for a user session with one or more user interfaces, event data that includes interface data specifying a structure of the user interface(s), and, for each of one or more user interface elements for which content was presented by the user interface(s) during the user session, an encrypted content element including the content of the user interface element encrypted using a public key corresponding to a rule enabling recording of the content of the user interface element and data identifying the rule. Playback of the user session is generated including, for each of the interface element(s), decrypting the encrypted content element for the user interface element and presenting the decrypted content during the playback of the user session.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for evaluating interactions with a user interface are disclosed. In one aspect, a method includes identifying a set of mutation events specifying changes to the structure of a user interface that occurred during the user session, and generating user interface states specifying different structures of the user interface throughout the given user session. Based at least in part on the user interface states, playback data that present visual changes of the user interface corresponding to the set of mutation events that occurred during the user session are generated. Session activity data describing user interactions that occurred during the user session are also generated. At least a portion of the playback data and the session activity data are output to a requesting device.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for clustering user interface event data for analysis and retrieval are disclosed. In one aspect, a system includes a data store and computer(s) that interact with the data store and execute instructions that cause the computer(s) to receive, for a user interface event, event data specifying a structure of a user interface presented during the user session. The event is assigned to a respective cluster based on a comparison of the structure of the user interface specified by the event data to a user interface structure that represents the respective cluster. For each cluster, a user interface attribute indicative of a user interface state of user interfaces specified by the event data in the cluster is determined. User interface state groups are generated based on the user interface attribute for each cluster.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for evaluating interactions with a user interface of an application are disclosed. In one aspect, a method includes, for each of a plurality of different user sessions of a native application, accessing frame bundles that each include data representing content presented by a frame of a user interface of the native application at a given time. Each frame bundle includes at least a portion of a view tree of the native application used to generate the user interface at the given time and data specifying content presented by each view of the portion of the view tree. Based on the frame bundles, playback data are generated that present visual changes of the user interface corresponding to changes to the view trees.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for providing user interface development tools. In one aspect, a method includes receiving console logs that specify sequences of user interface events that occurred during user sessions. For each of multiple user sessions, a received console log for the user interface events that occurred during that user session is identified. Playback data that presents visual changes of one or more user interfaces of the publisher that occurred during that user session is generated. An interactive interface is generated and provided. The interface includes session replay area in which the playback data presents the visual changes of the one or more user interfaces of the publisher that occurred during that user session and a console log area that presents console entries from the console log for the user interface events that occurred during that user session.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for providing user interface development tools. In one aspect, a method includes receiving, from a client device, network request data that specifies, for each of multiple network requests that were transmitted from the client device to a server during a user session, a requested user interface element and a load time that indicates a duration of time between a time at which the network request was initiated and a time at which the user interface element was loaded in a user interface at the client device. An interactive interface is generated. The interactive interface includes a session replay area that presents playback of visual changes of the one or more user interfaces that occurred during the user session and a network request area that presents a list of network requests that were transmitted during the user session.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for evaluating interactions with a user interface are disclosed. In one aspect, a method includes identifying a set of mutation events specifying changes to the structure of a user interface that occurred during the user session, and generating user interface states specifying different structures of the user interface throughout the given user session. Based at least in part on the user interface states, playback data that present visual changes of the user interface corresponding to the set of mutation events that occurred during the user session are generated. Session activity data describing user interactions that occurred during the user session are also generated. At least a portion of the playback data and the session activity data are output to a requesting device.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for providing user interface development tools. In one aspect, a method includes receiving, from a client device, network request data that specifies, for each of multiple network requests that were transmitted from the client device to a server during a user session, a requested user interface element and a load time that indicates a duration of time between a time at which the network request was initiated and a time at which the user interface element was loaded in a user interface at the client device. An interactive interface is generated. The interactive interface includes a session replay area that presents playback of visual changes of the one or more user interfaces that occurred during the user session and a network request area that presents a list of network requests that were transmitted during the user session.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for evaluating interactions with a user interface of an application are disclosed. In one aspect, a method includes, for each of a plurality of different user sessions of a native application, accessing frame bundles that each include data representing content presented by a frame of a user interface of the native application at a given time. Each frame bundle includes at least a portion of a view tree of the native application used to generate the user interface at the given time and data specifying content presented by each view of the portion of the view tree. Based on the frame bundles, playback data are generated that present visual changes of the user interface corresponding to changes to the view trees.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for evaluating interactions with a user interface of an application are disclosed. In one aspect, a method includes, for each of a plurality of different user sessions of a native application, accessing frame bundles that each include data representing content presented by a frame of a user interface of the native application at a given time. Each frame bundle includes at least a portion of a view tree of the native application used to generate the user interface at the given time and data specifying content presented by each view of the portion of the view tree. Based on the frame bundles, playback data are generated that present visual changes of the user interface corresponding to changes to the view trees.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for evaluating interactions with a user interface are disclosed. In one aspect, a method includes identifying a set of mutation events specifying changes to the structure of a user interface that occurred during the user session, and generating user interface states specifying different structures of the user interface throughout the given user session. Based at least in part on the user interface states, playback data that present visual changes of the user interface corresponding to the set of mutation events that occurred during the user session are generated. Session activity data describing user interactions that occurred during the user session are also generated. At least a portion of the playback data and the session activity data are output to a requesting device.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating and presenting engagement heatmaps. In one aspect, a method includes generating and providing, to a client device, an interactive interface that includes a session replay area that presents playback of user interfaces presented during a user session. A detection is made, based on interaction data received from the client device, of user interaction with an interface control for viewing engagement heatmaps. In response to detecting the user interaction with the interface control, playback is paused. The interactive interface is updated to present, for each of two or more user interface elements presented by the given user interface, an engagement heatmap visualization that visually represents a relative quantity of user interactions with the user interface element when the user interface element was presented by the given interface.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for protecting user privacy in the playback of user sessions are described. In one aspect, a method includes accessing, for a user session with one or more user interfaces, event data that includes interface data specifying a structure of the user interface(s), and, for each of one or more user interface elements for which content was presented by the user interface(s) during the user session, an encrypted content element including the content of the user interface element encrypted using a public key corresponding to a rule enabling recording of the content of the user interface element and data identifying the rule. Playback of the user session is generated including, for each of the interface element(s), decrypting the encrypted content element for the user interface element and presenting the decrypted content during the playback of the user session.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for providing user interface development tools. In one aspect, a method includes receiving console logs that specify sequences of user interface events that occurred during user sessions. For each of multiple user sessions, a received console log for the user interface events that occurred during that user session is identified. Playback data that presents visual changes of one or more user interfaces of the publisher that occurred during that user session is generated. An interactive interface is generated and provided. The interface includes session replay area in which the playback data presents the visual changes of the one or more user interfaces of the publisher that occurred during that user session and a console log area that presents console entries from the console log for the user interface events that occurred during that user session.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating and presenting engagement heatmaps. In one aspect, a method includes generating and providing, to a client device, an interactive interface that includes a session replay area that presents playback of user interfaces presented during a user session. A detection is made, based on interaction data received from the client device, of user interaction with an interface control for viewing engagement heatmaps. In response to detecting the user interaction with the interface control, playback is paused. The interactive interface is updated to present, for each of two or more user interface elements presented by the given user interface, an engagement heatmap visualization that visually represents a relative quantity of user interactions with the user interface element when the user interface element was presented by the given interface.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for evaluating interactions with a user interface of an application are disclosed. In one aspect, a method includes, for each of a plurality of different user sessions of a native application, accessing frame bundles that each include data representing content presented by a frame of a user interface of the native application at a given time. Each frame bundle includes at least a portion of a view tree of the native application used to generate the user interface at the given time and data specifying content presented by each view of the portion of the view tree. Based on the frame bundles, playback data are generated that present visual changes of the user interface corresponding to changes to the view trees.