Abstract: A processor-implemented method for generating a test suite within a time requirement is provided. The processor-implemented method includes executing a rule selection operation to determine candidate test cases utilizing attributes corresponding to each of the candidate test cases to produce selected test cases. The processor-implemented method includes determining whether an estimated testing execution time of the selected test cases is equal to or less than the time requirement. The processor-implemented method includes generating the test suite based on the selected test cases when the estimated testing execution time is equal to or less than the time requirement.

Abstract: A system receives one or more messages including information related to one or more issues associated with a software application installed on a user device. In response to receiving the message, a processor in the system identifies a user context associated with the user device, wherein the user context comprises usage information related to the software application. The processor further analyzes the received information and the user context associated with the user device to determine one or more errors associated with the software application. One or more software updates that are available and associated with the one or more errors are determined. The system automatically installs one or more of the determined software updates on the user device. Embodiments of the present invention further include a method and computer program product for managing and installing one or more software updates in substantially the same manner as is described above.

Abstract: In partitioning a graph database, a plurality of vertices of the graph database is assigned to a plurality of nodes. The vertices of the graph database are connected by edges that indicate relationships between the vertices. A vertex of the graph database is designated as a super-vertex that is split into a truncated vertex and at least one vertex representative.

Abstract: Disclosed embodiments relate to systems and methods for automatically detecting and addressing security risks in code segments. Techniques include accessing a plurality of code segments developed for execution in a network environment, automatically identifying a first code segment from the plurality of code segments for analysis, automatically performing a first code-level security risk assessment for the first code segment, and determining a first security risk level for the first code segment based on the application programming interface risk level. The first code-level security risk assessment may be performed based on at least one of an application programming interface risk level, an embedded credentials risk level, and a target resource risk level. Further techniques may include determining a second security risk level for a modified version of the first code segment; and enabling a comparison between the first security risk level and the second security risk level.

Abstract: In some database systems, multiple servers may be deployed with different configurations based on stacks, applications, or services associated with the servers. Such systems may perform server validations to determine whether the servers are deployed with the correct configurations. A testing server may validate one or more target servers using metadata and parameters of the target servers. The testing server may receive parameters corresponding to a configuration file, and the testing server may select a subset of validation routines to perform for each target server based on metadata in the configuration file. In this way, the testing server may perform the same validation process for each target server, but may perform a different subset of validation routines. The testing server may execute code containing the selected subset of validation routines remotely against the target server, and may determine whether the server was configured correctly.

Abstract: A method for integration-testing a microservices-based application is performed by an automated software-testing system by using mock software entities. The mock entities emulate the behavior of other microservices or external software applications that would normally interact with the tested microservices in a production environment. The mock entities are managed by a mock server that communicates with the test system through a messaging system. The mock server determines how the mock entities should respond to the tested application's service requests by referring to test guidelines stored in a behavior store. During testing, the test system dynamically updates the behavior store in response to the tested application's service requests and to other indicators of the current state of the tested application. If the tested application interacts with the mock entities during the test as expected, the test system deems the application to have passed the integration test.

Abstract: Automated software-testing systems and computer program products use mock software entities to perform integration testing on a microservices-based application. The mock entities emulate the behavior of other microservices or external software applications that would normally interact with the tested microservices in a production environment. The mock entities are managed by a mock server that communicates with the test system through a messaging system. The mock server determines how the mock entities should respond to the tested application's service requests by referring to test guidelines stored in a behavior store. During testing, the test system dynamically updates the behavior store in response to the tested application's service requests and to other indicators of the current state of the tested application. If the tested application interacts with the mock entities during the test as expected, the test system deems the application to have passed the integration test.

Abstract: In an embodiment, a system is configured to replay and/or reconstruct execution events and system states in real time or substantially in real time starting from the point when execution of a target program has stopped to the point when the user desires to step through the target program's execution in order to debug the software. In an embodiment, a system is configured to efficiently collect trace data that is sufficient to reconstruct the state of a computer system at any point of time from the start of execution to the time execution was stopped. Efficient and effective debugging of the software can be performed using embodiments of the disclosed methods, systems, and devices.

Abstract: For each detected bug, historical code with similar characteristics and bug corrections from a historical bug dataset can be displayed in a source code editor. Relevant training and/or testing data can be found by comparing an internal representation of the code under development with an internal representation of the original buggy code in the historical bug dataset. Training and/or testing data that is relevant to the current code can be distinguished from irrelevant training and/or testing data by determining that the code syntax tokens from the current and historical data overlap to at least a specified percentage. Code can be devolved into a set of metrics. The degree of overlap between the metric sets can be determined. If a computed risk factor for the bug correction meets or exceeds a specified threshold, the bug correction can be automatically applied. Additional testing can be automatically performed on and/or added to the corrected code.

Abstract: Method and apparatus for detecting anomalous flights. Embodiments collect sensor data from a plurality of sensor devices onboard an aircraft during a flight. A plurality of feature definitions are determined, where a first one of the feature definitions specifies one or more of the plurality of sensor devices and an algorithm for deriving data values from sensor data collected from the one or more sensor devices. Embodiments determine whether anomalous activity occurred during the flight using an anomaly detection model, where the anomaly detection model describes a pattern of normal feature values for at least the feature definition, and comprising comparing feature values calculated from the collected sensor data with the pattern of normal feature values for the first feature definition. A report specifying a measure of the anomalous activity for the flight is generated.

Abstract: Code monitoring and optimization may include code being monitored for potential security violations during an active code creation session. The procedure may include at least one of monitoring code for potential security violations during an active code creation session occurring on a client device, comparing the monitored code with an aggregate rule set stored in a repository, identifying at least one of a security violation or an error based on the comparing and identifying a correction to the at least one of the security violation or the error during the active code creation session.

Abstract: A mechanism for providing context simulation. A method includes receiving context relay from a client machine. The context relay includes context input set (CIS) and configuration difference (diff) data. The method also includes applying the diff data to an application so that configuration settings of the application are same as the configuration settings of a first application on the client machine. The method also includes retrieving message information from the CIS and sending the message information to the application. The method further includes executing the application in same interval as the first application being executed on the client machine.

Abstract: An apparatus updates module information containing information on interfaces of modules contained in source code of a test-run target, when an update of an interface in the source code is detected. The apparatus refers to test-case information containing pieces of information on plural test cases, and selects a run-target test case from among the plural test cases, based on test run times taken in previous runs of the plural test cases. The apparatus generates a dummy module to include simplified code by simplifying code in source code of a second module to be statically linked to a first module called during a previous run of the selected run-target test case, so that the simplified code is not related to implementation of the interface indicated in the updated module information. The apparatus compiles the first module and the dummy module, and issues an instruction to run the selected run-target test case.

Abstract: A method and system for dynamically controlling publication of a changing web page. A delta vector of a change in position of a user interface control in a web browser window is invoked. The change in position occurred between a first time before a change to a web page that is already public was received and a second time after the change to the web page was implemented. A click speed component is invoked to obtain information on transition times between user interface controls including the user interface control within a click flow. A user interface control weight value is determined for the user interface control having undergone the change in position. A disruptiveness factor value is determined based on the delta vector and the corresponding user interface control weight. A test pattern to be applied to the changed web page is selected depending on the determined disruptiveness factor.

Abstract: A system for use in displaying software components to a user is described herein. A system comprising one or more processors and one or more non-transitory memory storage devices storing computer instructions configured to run on the one or more processors and to perform accessing a data repository of a plurality of data repositories comprising at least one software component data file associated with a software component comprising a software code and an object data. Retrieving the at least one software component data file executing the software code and the object data associated with the at least one software component data file to render a demonstration image of the software component associated with the at least one software component data file. Generating a component display website to be displayed and comprising a component display area associated with the software component associated with the at least one software component data file.

Abstract: A computer system may identify a cryptographic application programming interface (API) call for a program. The cryptographic API call may include a first variable. The computer system may determine that the first variable is a static value. The computer system may tag the first variable. The computer system may determine that the cryptographic API call will be executed. The computer system may replace the first variable with a second variable during execution of the program. The computer system may execute the cryptographic API call with the second variable.

Abstract: Embodiments include a method for generation of test scenarios based on risk analysis. The method includes receiving a first set of code test scenarios, the first set of code test scenarios configured to test one or more first code components, the first code components each including first code attributes. The method also includes determining a probability of finding defects in the first code components based on the first code attributes. The method also includes generating a second set of code test scenarios based on the determining the probability of finding defects in the first code components based on the first code attributes, where the second set of one or more test scenarios is configured to test one or more second code components based on the second code attributes.

Abstract: A system and apparatus for monitoring networks and quality-of-service elements within networks. Probes placed at strategic locations throughout the network activate at predetermined intervals or upon predetermined events, conduct tests emulating actual user experience, and report the results of the tests to a controller. The data collected by the controller is used to maintain the integrity of the network and to evaluate the user experience within the network.

Abstract: Generally, this disclosure provides systems, devices, methods and computer readable media for launching MPI-based processes and applications on multiple platform types. A method may include determining a type of an OS associated with a processing node platform on which the MPI-based process is to be launched. The method may also include launching a proxy module on the processing node platform, the proxy module configured to launch the MPI-based process. The launching of the proxy module may include operations that are based on the determination of the OS type of the processing node platform. These operations may include the execution of a secure shell for remote process launching and/or requesting a remote process launching from a system service module executing on the processing node platform.

Abstract: Particular embodiments described herein provide for a network element that can be configured to receive, from an electronic device, a request to access a network service. In response to the request, the network element can send data related to the network service to the electronic device and add a test link to the data related to the network service. The network element can also be configured to determine if the test link was successfully executed and classify the electronic device as untrusted if the test link was not successfully executed.

Abstract: Disclosed is a system and method for multi-channel testing of a software application is disclosed. The system may generate a set of test case definitions for testing a software application. The system may further allocate a sub-set of test case definitions, from the set of test cases definitions, to each test execution channel from the set of test execution channels based on a set of execution parameters specified by a user and the primary execution channel as well as the secondary execution channel associated with each test case definition. Furthermore, using the test case definitions associated with each test case execution channel, the system may enable testing of the software application at respective test execution channels and accordingly report the test execution results to the user.

Abstract: A debug session identifier representing a debug session for the web application is stored in local storage. The web application is located in multi-node cloud storage. The local storage resides on a developer computer system that executes outside of the multi-node cloud system. A stateless debugger node and a stateless target tester node are selected from at least one node pool in the multi-node cloud system. The stateless debugger node and the stateless target tester node are separate from each other. An internal connection is provided between the stateless debugger node and the stateless target tester node within the multi-node cloud system based on the debug session identifier during the debug session. The script that is injected into the web application is debugged while the web application executes on the stateless target tester node.

Abstract: A method for testing an updated version of an existing software application. The method may comprise analyzing a user interface screen of the updated version of the existing software application to identify previously existing controls and updated controls and automatically capturing, via a capture engine, each of the updated controls present on the user interface screen of the updated version, wherein the automatic capturing is initiated by a user selecting a learn screen function. The method may further comprise automatically associating, via a rules base, control descriptions with each of the automatically captured updated controls and one or more testing actions with each of the updated controls, thereby generating a plurality of test steps each comprising one of the updated controls, a particular associated control description, and a particular testing action. The method may then comprise generating an updated test component comprised of the plurality of test steps.

Abstract: For implementing cognitive feature based code level updates, an apparatus is disclosed. In some embodiments, the apparatus includes a detection module that detects components operating at a client site, collects information associated with the detected components, and identifies defects in the detected components based on the collected information. The apparatus includes a network module that transfers the collected information to a host apparatus and receives a flash issuance from the host apparatus based on the collected information transferred to the host apparatus. The apparatus includes a repair module that retrieves a fix list from the flash issuance received from the host apparatus and implements the fix list for the detected components when defects are identified in the detected components.

Abstract: A continuous integration system and method for testing source code is disclosed. In particular, the continuous integration system allows developers to create predefined tasks for builds—each task comprises information to perform a particular build step. In certain embodiments, developers can select a particular task from a set of built-in tasks to add to their build configuration without having to know or determine the underlying build logic required to perform the task (such as the commands needed to execute the task, the artefacts required for the task, etc.). In particular, the disclosed systems and methods allow developers to add one or more predefined tasks in their build configuration, without adding the build logic associated with the task in the configuration file.

Abstract: Example implementations relate to automatically identifying regressions. Some implementations may include a data capture engine to capture data points during test executions of the application under test. The data points may include, for example, test action data and application action data. Additionally, some implementations may include a data correlation engine to correlate each of the data points with a particular test execution of the test executions, and each of the data points may be correlated based on a sequence of events that occurred during the particular test execution. Furthermore, some implementations may also include a regression identification engine to automatically identify, based on the correlated data points, a regression between a first version of the application under test and a second version of the application under test.

Abstract: According to an aspect of the present inventive concept there is provided a method for identifying a cause for a failure of a computer-implemented test performed on a latest software code revision in a sequence of a plurality of software code revisions, the method comprising: identifying a first revision of said sequence of revisions for which said test passes and a second revision of said sequence of revisions which is subsequent to the first revision and for which said test fails, determining a first change between the software code of the first revision and the software code of the second revision, generating a first modified version of the software code of the latest revision using the determined first change, determining whether said test passes or fails when performed on the first modified version, in response to determining that said test fails when performed on the first modified version: identifying a third revision and a fourth revision of said sequence which are intermediate the latest revision

Abstract: A method may include obtaining a plurality of synthesized method invocations using a plurality of objects and a plurality of methods of a software program. The method may also include determining a prioritization of the plurality of synthesized method invocations based on one or more of: relationships between one or more characteristics of each of the plurality of synthesized method invocations and a fault location; and relationships between the one or more characteristics and an error report that corresponds to the fault location. The method may also include selecting a synthesized method invocation from the plurality of synthesized method invocations for repair operations with respect to the fault location based on a corresponding prioritization of the selected synthesized method invocation. In addition, the method may include performing repair operations with respect to the fault location and the selected synthesized method invocation.

Abstract: The invention discloses a method for an audio/video live broadcast in a HTML5-based browser. There is a WebRTC-based communication channel built between a client end and a host end to do transmission for a video stream and an audio stream as well as a message transmission. After a communication is built between a client end and a host end, the client-end browser will receive data and process them while the network server only conducts works for verification, connections and heartbeat detection. A work to push streams does not need to be conducted by the server and thus the loading on the network server is reduced.

Abstract: Aspects of the disclosure relate to conducting automated software testing using a centralized controller and one or more distributed test host servers. A computing platform may receive a test execution request. Subsequently, the computing platform may retrieve test specification details information and may identify one or more tests to execute. Then, the computing platform may generate one or more remote test execution commands directing a test host server farm to execute the one or more tests. In addition, generating the one or more remote test execution commands may include constructing one or more command line instructions to be executed by the test host server farm and inserting the one or more command line instructions into the one or more remote test execution commands. Thereafter, the computing platform may send the one or more remote test execution commands to the test host server farm.

Abstract: A classification machine learning model is trained to predict the likelihood that a software program is likely to have a software bug in the future. The model is based on features from different source code files having changes made to fix a software bug and source code files having changes that were not made for a bug fix. The features include a time-weighted bug density, a time-weighted addition factor, and a time-weighted deletion factor for a source code file and its dependent code, a page rank, and complexity features representing a number of different types of code elements in the source code file.

Abstract: A system and method for developing a multi-layered test suite for an operating platform including a framework layer and a system layer includes executing a first test suite against a version of the operating platform modified based on a software faults. A first counter is incremented if a first test suite executed against the modified version of the operating platform fails. A second test suite can be executed against the modified version of the operating platform and test cases may be added to the first test suite based on whether the second test suite passes or fails.

Abstract: A data system is provided for analyzing and maintaining data obtained from one or more data sources on which the data system depends. The system includes a primary database including current values used by the system and a collection of executable algorithms used to generate the data maintained in the primary database. In response to receiving a notification regarding a change in one of the data sources, a dependency database is used to establish an execution order for algorithms of the algorithm collection that are directly or indirectly dependent on the changed data. The algorithms identified in the execution order are then executed in accordance with the execution order and the corresponding result is stored in the primary database. The system may include data harvesters adapted to recognize changes in the data sources and to generate and transmit corresponding change notifications when such changes occur.

Abstract: The present disclosure relates to automatically detecting software defects and determining the source of the defect. An example method generally includes receiving an output data set from a host server executing a test operation and comparing the output data set to a performance model. A test system identifies that a defect exists in software executing on the host server based on determining that the output data set deviates from the performance model by more than a threshold. The test system retrieves a source code update from a source code repository that was committed to the source code repository before the execution of the test operation. The test system compares a scope of the source code update to the defect and notifies a development team related to the source code update of the defect upon determining that the scope of the source code update and the defect match.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for linearizing the commit history of a code base. One of the methods includes receiving a request to attribute source code contributions to a particular snapshot in a project having a revision graph with multiple branches. Source code contributions occurring in each non-merge commit on a single path through the revision graph are attributed to a responsible developer entity of the non-merge commit. Each merge commit is processed on the single path including identifying contributions introduced by the merge commit, identifying one or more respective responsible developer entities for the contributions occurring in the one or more feature branch snapshots from one or more feature branches, and attributing source code contributions occurring in the one or more feature branch snapshots from the one or more feature branches to the identified one or more respective responsible developer entities.

Abstract: Apparatus and methods for an intelligent audit engine are provided. Product development artifacts may be associated with artifact tags and stored in an artifact repository. The audit engine may retrieve an artifact for audit. The audit engine may identify a set of development guidelines in a rule repository based on their artifact tags. The audit engine may determine whether the development guidelines have been satisfied based on stored product testing data. A failed development guideline may initiate a workflow that includes notification of a project stakeholder regarding the failure. The notification may include remediation measures to be taken by the stakeholder. The validation process may iterate until all guidelines have passed validation. A machine-learning algorithm may prioritize a development guideline for future audits to increase efficiency. The machine learning algorithm may educate stakeholders to reduce audit failures.

Abstract: In some implementations, a server device may receive, from a client device, a script identifier, that identifies a script, and an indication of whether the script was successfully executed by the client device. The server device may determine whether a condition associated with disabling the script is satisfied based on the script identifier, the indication of whether the script was successfully executed by the client device, and stored information determined based on a plurality of indications of whether the script was successfully executed by a plurality of other client devices. The server device may selectively cause a data structure to be updated to indicate that the script is disabled based on determining whether the condition is satisfied. The data structure may be used to indicate whether the script is enabled prior to execution of the script.

Abstract: A system and method for controlling quality of performance of a digital application is disclosed. Performance data representing test results of a requested performance test on the application is obtained. A performance metric is optimized by analyzing the performance data and classifying analysis results of the analyzing into multiple categories, including an immediate remediation category. An automated modification is initiated based on a first analysis result classified into the immediate remediation category. An automated verification performance test determines a measure of improvement in performance of the application based on the modification by comparing a first verification result of a first performance test without the automated modification with a second verification result of a second performance test of the digital application with the automated front end modification.

Abstract: The present invention may be a method, a system, and/or a computer program product. An embodiment of the present invention provides a computer-implemented method for associating one or more test cases with one or more source code parts for a test target, or vice versa. The method comprises the followings: obtaining one or more historical data in which one or more modification histories made in one or more source codes are associated with one or more test operation histories made in the test target after the one or more modifications; for each of the historical data, determining whether the test operation history has one or more parts which are identical or similar with those in one or more test cases; and if the determination is positive, associating the one or more test cases with one or more source code parts in the one or more modification histories which have been associated with the test operation history, or vice versa.

Abstract: Systems and methods for uploading tenant code to a multi-tenant system. Tenant Java source code is sent to the multi-tenant system over an MDL interface, and stored in a repository of the multi-tenant system as source code metadata. The tenant code may be extracted and executed when a predetermined event occurs.

Abstract: An apparatus and method for controlling mobile phone settings can include setting up an external speaker or an auxiliary port providing an audio signal output. In particular, the method runs to disable speaker and other audio extension ports thereby providing more control to the caller and preventing mishandling at the receiver's mobile device. The method includes initiating the call, from the calling device, with a non-speaker call signal setting. The receiving mobile interprets such signal and automatically disables the speaker/aux ports of the receiver mobile during the call duration. The apparatus and method implements a speaker disabled call initiated by the sender with the inclusion of a flag set to indicate the speaker disable feature in the signaling mobile channel of the network and be processed and carried over the network to the receiving mobile. The destination mobile resolves the flag to modify device settings to disable speaker and other audio extension ports.

Abstract: Described embodiments provide systems and methods for routing service requests. The system includes a first network of computing devices including a server hosting a service. The system includes a release router in the first network, the release router configured to receive a request for the service, the request forwarded to the release router responsive to resolving a hostname specified in the request using a DNS mapping the hostname to the release router, the hostname associated with the service hosted by the server. The release router is configured to identify a relay agent registered with the release router for debugging the service, the relay agent executed by a test platform in a second network, and to forward the request to the test platform in the second network, wherein the test platform resolves the hostname specified in the request using a local DNS mapping the hostname to a localhost address.

Abstract: A system for equivalence class analysis-based automated requirements-based test case generation includes a control processor, a data store containing textual design requirements, a textual converter unit structured to convert the textual design requirements to a machine-readable version of design requirements, a requirement partition unit configured to partition the machine-readable design requirements into one or more sets of related design requirements, an equivalence class partition unit configured to process the machine-readable design requirements and input/output variables into a set of equivalence classes, an equivalence class analyzer unit structured to analyze the set of equivalence classes to generate equivalence class tests and identify uncovered input space, and a boundary class analyzer unit structured to identify boundaries of the equivalence classes and generate boundary value tests and robustness tests.

Abstract: Methods, systems, and devices for wireless communication are described. A user equipment (UE) configured with several component carriers (CCs) may select one or more CCs for fast activation. The UE may, in various examples, autonomously select the CCs for fast activation or may receive signaling indicating such CCs. In some cases, the UE may place the selected CCs in a semi-active state by detecting and synchronizing the CCs while refraining from control channel monitoring. In other examples, the UE may identify a set of CCs that may be activated quickly based on channel conditions or that have activation interdependency, such as CCs in a physical uplink control channel (PUCCH) group. The UE may then trigger an activation for one of the selected CCs, determine it is capable of fast activation, and begin monitoring, e.g., for control information, after a reduced delay period based on the fast activation.

Abstract: A knowledge graph is built based on a corpus stored in the computer system. The corpus includes a set of events and each event includes a respective set of entities. A first set of entities is identified in the NLQ. The first set of entities is used to identify a first set of significant events in the selected corpus in a first search depth. A second set of entities is identified in the first set of significant events. The knowledge graph determines which ones of the second set of entities are related to the entities in the first set of entities to produce a filtered second set of entities. The filtered second set of entities is used to identify a second set of significant events in the selected corpus in a second search depth. Members of the first and second set of significant events are presented to a user.

Abstract: Mechanisms are provided for evaluating test cases for testing a software product based on a requirements change. The mechanisms analyze a test case corpus to identify a plurality of first relationships between elements of test cases in the test case corpus and generate a test case relationship model based on the identified plurality of first relationships. The mechanisms receive a proposed requirements change to change one or more requirements of the software product and then perform a search of the test case relationship model to identify test case relationships corresponding to the proposed requirements change. The mechanisms identify a subset of test cases affected by the proposed requirements change and generate an output specifying the identified subset of test cases.

Abstract: An anomaly estimation apparatus includes a collection section that collects vehicle data, a feature amount calculation section that calculates a feature amount from the vehicle data and stores the feature amount and a place corresponding thereto, an anomaly determination section that determines whether an anomaly occurrence point is present based on the feature amount, an accumulation section that, if the anomaly occurrence point is present, uses the vehicle data at the anomaly occurrence point and an anomaly periphery point to generate estimation data, an information generation section that uses the estimation data to generate causality information representing causality between an anomaly caused at the anomaly occurrence point and an anomaly caused at the anomaly periphery point, and an estimation section that, if the anomaly occurrence point is present, uses the causality information to estimate transition of the anomaly from the anomaly occurrence point to the anomaly periphery point.

Abstract: A system, computer-implemented method and computer program product for monitoring one or more software development life cycles is provided. The system comprises one or more agents configured to collect data pertaining to one or more Software Development Life Cycles (SDLCs) from one or more tools. The system further comprises a data aggregator and co-relator configured to convert the collected data to one or more object formats and co-relate the converted data by storing the converted data in a graph database as one or more nodes and corresponding relationships and properties, wherein each of the one or more nodes comprise converted data corresponding to a specific tool. Furthermore, the system comprises an indexed database configured to index the stored data. In addition, the system comprises a dashboard viewer configured to render the indexed data on one or more pre-configured dashboards for one or more users.

Abstract: Described technologies facilitate cloud validation using validation as a service (VaaS). A cloud validation service provider acquires and securely stores certification tests developed by cloud component providers, integrated solution providers, and others. Each test's executable portion tests hardware or software of a candidate cloud. The candidate may be on the premises of an enterprise, or instead be a hosted cloud on the premises of a hoster off the premises of the entity that pays for the hosting. Monitored testing is done using an infrastructure in the candidate cloud or in a public cloud. Results are uploaded to the VaaS provider, which provides an analysis of test results for use in determining whether to validate the candidate cloud. Test execution agents may be VaaS-cloud-resident or candidate-cloud-resident, and may use a mutex to prevent simultaneous execution of tests. Testing may be accomplished even when the candidate cloud has no internet-exposed communication endpoint.

Abstract: Technology is described for testing applications. Multiple testing threads that are to be used for testing an application may be identified. The multiple testing threads may be dispatched for execution using a unit testing framework. The multiple testing threads may be executed in parallel to test the application using testing context information that is related to each testing thread. Performance metrics may be collected for the application that is tested. The performance metrics may be transmitted from the multiple testing threads after parallel execution of the multiple testing threads.