Abstract: A load balanced system may incorporate instrumented systems within a group of managed devices and distribute workload among the devices to meet both load balancing and data collection. A workload distributor may communicate with and configure several managed devices, some of which may have instrumentation that may collect trace data for workload run on those devices. Authentication may be performed between the managed devices and the workload distributor to verify that the managed devices are able to receive the workloads and to verify the workloads prior to execution. The workload distributor may increase or decrease the amount of instrumentation in relation to the workload experienced at any given time.

Abstract: Input sequence information may be analyzed and quantified using n-gram analysis of inputs received by an application. The sequences of inputs may be represented by n-grams, and the frequency of the various n-grams may indicate the ‘real world’ uses of the application in production, which may be compared to a test suite whose coverage may be quantified using a similar n-gram analysis. A coverage factor may compare the observed inputs to the application in production to the test suite for the application. The n-grams may be further quantified or prioritized by resource utilization and several visualizations may be generated from the data.

Abstract: Visualizing execution of an application in a trace coverage graph includes receiving module trace data gathered during execution of an executable module. The module trace data includes data representing sequences of execution between individual executable components. A trace coverage graph is generated and displayed from the module trace data. The trace coverage graph includes graph nodes that each correspond to a different executable component. The trace coverage graph also includes graph edges that each visually connects two of the graph nodes. Each graph edge visually indicates an amount of trace data present for the sequence of execution between the graph nodes that are connected by the graph edge. Thicker graph edges represent the presence of a larger amount of trace data, and thinner graph edges represent the presence of a lesser amount of trace data.

Abstract: Periodicity similarity between two different tracer objectives may be used to identify additional input parameters to sample. The tracer objectives may be individual portions of a large tracer operation, and each of the tracer objectives may have separate set of input objects for which data may be collected. After collecting data for a tracer objective, other tracer objectives with similar periodicities may be identified. The input objects from the other tracer objectives may be added to a tracer objective and the tracer objective may be executed to determine a statistical significance of the newly added objective. An iterative process may traverse multiple input objects until exhausting possible input objects and a statistically significant set of input objects are identified.

Abstract: A tracing management system may use cost analyzes and performance budgets to dispatch tracing objectives to instrumented systems that may collect trace data while running an application. The tracing management system may analyze individual tracing workloads for processing, storage, and network performance costs, and select workloads to deploy based on a resource budget that may be set for a particular device. In some cases, complementary tracing objectives may be selected that maximize consumption of resources within an allocated budget. The budgets may allocate certain resources for tracing, which may be a mechanism to limit any adverse effects from tracing when running an application.

Abstract: An event chain visualization of performance data may show the execution of monitored elements as bars on a timeline, with connections or other relationships connecting the various bars into a sequential view of an application. The visualization may include color, shading, or other highlighting to show resource utilization or performance metrics. The visualization may be generated by monitoring many events processed by an application, where each bar on a timeline may reflect multiple instances of a monitored element and, in some case, the aggregated performance.

Abstract: An analysis system may perform network analysis on data gathered from an executing application. The analysis system may identify relationships between code elements and use tracer data to quantify and classify various code elements. In some cases, the analysis system may operate with only data gathered while tracing an application, while other cases may combine static analysis data with tracing data. The network analysis may identify groups of related code elements through cluster analysis, as well as identify bottlenecks from one to many and many to one relationships. The analysis system may generate visualizations showing the interconnections or relationships within the executing code, along with highlighted elements that may be limiting performance.

Abstract: Periodicity similarity between two different tracer objectives may be used to identify additional input parameters to sample. The tracer objectives may be individual portions of a large tracer operation, and each of the tracer objectives may have separate set of input objects for which data may be collected. After collecting data for a tracer objective, other tracer objectives with similar periodicities may be identified. The input objects from the other tracer objectives may be added to a tracer objective and the tracer objective may be executed to determine a statistical significance of the newly added objective. An iterative process may traverse multiple input objects until exhausting possible input objects and a statistically significant set of input objects are identified.

Abstract: Comparisons of different versions of an application may be compared using a behavior model of the application. A behavior model may be derived from n-gram analysis of observations of the application in production. The behavior model may include sequences of inputs received by the application or functions performed by the application, where each sequence is an n-gram observed in tracer data. Each n-gram may be coupled with a resource consumption to give a behavior model with performance data. A regression analysis may apply a behavior model derived from a first version of an application to the performance observations of a new version to create an expected performance metric for the new version. A similarly calculated metric from a previous version may be compared to the metric from a new version to determine an improvement or degradation of performance.

Abstract: An analysis system may perform network analysis on data gathered from an executing application. The analysis system may identify relationships between code elements and use tracer data to quantify and classify various code elements. In some cases, the analysis system may operate with only data gathered while tracing an application, while other cases may combine static analysis data with tracing data. The network analysis may identify groups of related code elements through cluster analysis, as well as identify bottlenecks from one to many and many to one relationships. The analysis system may generate visualizations showing the interconnections or relationships within the executing code, along with highlighted elements that may be limiting performance.

Abstract: A visualization system for a tracer may include a processing pipeline that may generate tracing data, preprocess the data, and visualize the data. The preprocessing step may include a mechanism to process user-defined expressions or other executable code. The executable code may perform various functions including mathematical, statistical, aggregation with other data, and others. The preprocessor may perform malware analysis, test the functionality, then implement the executable code. A user may be presented with an editor or other text based user interface component to enter and edit the executable code. The executable code may be saved and later recalled as a selectable transformation for use with other data streams.

Abstract: A module-specific tracing mechanism may trace the usage of a module on behalf of the module developer. The module may be used by multiple application developers, and the tracing system may collect and summarize data for the module in each of the different applications. The data may include usage data as well as performance data. Usage data may include anonymized data for each time the module may be invoked and called, and performance data may include the processing time, memory consumption, and other metrics. The module-specific tracing may be enabled or disabled by an application developer.

Abstract: Regression testing of an application may gather performance tests for multiple functions within an application and determine when performance changes from one version of the application to another. The analysis may be further broken down by input sequences that may be processed by various functions. A detailed regression analysis may be presented as a heat map or other visualizations. A regression testing system may be launched during a build process by automatically launching a set of performance tests against an application. In many cases, the application may be executed in a system with a known or consistent performance capabilities. The application may be executed and tested in a new version and at least one prior version on the same hardware and software execution environment, so that results may be normalized from one execution run to another. A regression testing system may be deployed as a paid-for service that may integrate into a source code repository.

Abstract: A visualization system for a tracer may include a processing pipeline that may generate tracing data, preprocess the data, and visualize the data. The preprocessing step may include a mechanism to process user-defined expressions or other executable code. The executable code may perform various functions including mathematical, statistical, aggregation with other data, and others. The preprocessor may perform malware analysis, test the functionality, then implement the executable code. A user may be presented with an editor or other text based user interface component to enter and edit the executable code. The executable code may be saved and later recalled as a selectable transformation for use with other data streams.

Abstract: N-grams of input streams or functions executed by an application may be analyzed to identify security breaches or other anomalous behavior. A histogram of n-grams representing sequences of executed functions or input streams may be generated through baseline testing or production use. An alerting system may compare real time n-gram observations to the histogram of n-grams to identify security breaches or other changes in application behavior that may be anomalous. An alert may be generated that identifies the anomalous behavior. The alerting system may be trained using known good datasets and may identify deviations as bad behavior. The alerting system may be trained using known bad datasets and may identify matching behavior as bad behavior.

Abstract: A force directed graph may display recent activities of a message passing system as highlighted features over a larger graph. The force directed graph may display a superset of nodes and edges representing processes and message routes, then display recent activities as highlighted elements within the larger superset. The highlighted elements may display messages passed or computation performed during a recent time element of a time series. In some embodiments, the effects of activities may be displayed by decaying the highlighted visual elements over time.

Abstract: A database of module performance may be generated by adding tracing components to applications, as well as by adding tracing components to modules themselves. Modules may be reusable code that may be made available for reuse across multiple applications. When tracing is performed on an application level, the data collected from each module may be summarized in module-specific databases. The module-specific databases may be public databases that may assist application developers in selecting modules for various tasks. The module-specific databases may include usage and performance data, as well as stability and robustness metrics, error logs, and analyzes of similar modules. The database may be accessed through links in module description pages and repositories, as well as through a website or other repository.

Abstract: A tracing system may trace applications and their modules, and may make module-specific data available through various interfaces. The tracing system may collect tracer data while an application executes, and may preprocess the data into application-specific and module-specific databases. An analysis engine may further analyze and process these databases to create application-specific views and module-specific views into the data. The application-specific views may be intended for a developer of the application, while the module-specific views may have a public version accessible to everybody and a module developer version that may contain additional details that may be useful to the module developer.

Abstract: A financing mechanism for intellectual property asset development finances the portfolio at a liquidation value over the life of the asset. A client company may assign their intellectual property assets to a holding company, and the client company may pay a payment to the holding company over the life of the asset. The client company may retain exclusive rights to the asset while payments are timely, but some or all of the rights may revert to the holding company when payments are not timely. The client company may have the option to purchase the assets or the holding company during the life of the assets.

Abstract: A module-specific tracing mechanism may trace the usage of a module on behalf of the module developer. The module may be used by multiple application developers, and the tracing system may collect and summarize data for the module in each of the different applications. The data may include usage data as well as performance data. Usage data may include anonymized data for each time the module may be invoked and called, and performance data may include the processing time, memory consumption, and other metrics. The module-specific tracing may be enabled or disabled by an application developer.