Abstract: A method and an apparatus for generating software test reports are provided. The method includes following steps: providing a testing platform that supports retrieving one or more documents such as screenshots or DOM-like documents related to screen content of an application under test (AuT) and analyzing the documents to obtain description data of the screen content; selecting to execute a test report generator, and querying the description data, multiple test scripts ever executed on a system under test (SuT) and multiple test actionables from the testing platform by the test report generator, so as to evaluate a test trace of the AuT, calculate at least one test actionable and test data adapted for the AuT, and return the calculated test actionable to the testing platform; and executing the test actionable on the AuT in the SuT by the testing platform, so as to generate test reports of the AuT.

Abstract: A method and computer program for generating code coverage information during testing of a code sequence are described, in which the code sequence comprises decisions, each having one or more conditions as inputs. The method includes executing the code sequence on target processing circuitry under the control of test stimuli and maintaining, in memory, a code coverage table for at least one decision. When a decision is evaluated, a bitstring is created within a storage element, each position in the bitstring being associated with one of the conditions and the value in that position representing the value of that condition used in evaluating the decision. The bitstring is used to identify the entry, in the code coverage table associated with the evaluated decision, for that combination of values of the conditions, and a confirmation value is recorded in that entry, indicating that the decision has been evaluated for that entry.

Abstract: Methods, systems, and computer-readable media are disclosed herein for a system and framework that tests end-user applications for failures, data validation, and performance indicators. In aspects, multiple use-modeling programs that mimic user interactions are used to concurrently run unique instances of the application in real-time to simulate a real-world scenario, at scale and with a full load. Whether data, operations, and/or functions of the end-user application fail or are successful is automatically documented in real-time, while performance is concurrently measured.

Abstract: The system determines data which indicate memory usage by a process and information related to a number of instances of the process. The system determines for the process: a first average amount of memory; a first standard deviation; a first value indicating a difference between a current amount of memory used by the process and the first average amount; a second average number of process instances; a second standard deviation; and a second value indicating a difference between a current number of process instances and the second average number. The system performs a corrective action to address a memory spike associated with the process responsive to comparing at least one of: the current amount of memory used by the process with a watermark; the first value with the first standard deviation scaled; the current number of instances with a watermark; and the second value with the second standard deviation scaled.

Abstract: A system and method for automated chatbot testing to provide training and quality assurance of conversational artificial intelligence systems, comprising a chatbot testing administrator interface which allows chatbot makers to define what a chatbot is supposed to do, create test scripts to test the performance of the chatbot, and review the results of the chatbot tests, a chatbot testing server which provides and interface between chatbot testing agents and the administrator interface, instantiates chatbot agents and distributes them across available hardware and runs testing programs which activate, configure, and deactivate chatbot testing agents as needed. A plurality of chatbot agents may be running in parallel to provide automated testing based upon test script configuration.

Abstract: Systems and methods for automated test data microservices are provided. Test versions of software (such as an Application Programming Interface (API)) may be configured to automatically generate test data and to call a microservice to manage the test data. The microservice may automatically add and remove the test data from an operational data store to facilitate the testing process and to automatically perform setup and teardown stages of the testing process.

Abstract: In some implementations, a device may receive action information, for an action from multiple actions, that indicates at least one of code associated with the action or one or more inputs required to complete the action. The device may receive workflow information, for a workflow included in a set of workflows, that indicates one or more actions from the multiple actions and an order in which the one or more actions are to be completed for the workflow. The device may receive, via an application programming interface (API) call, a request to complete the workflow The device may identify the workflow information associated with the workflow in a data structure. The device may perform the workflow based on the workflow information and the one or more inputs by executing code for the one or more actions indicated by the workflow information and in the order indicated by the workflow information.

Abstract: A system including a data repository storing metrics describing operational behavior of software programs executing in an enterprise system. The system also includes an application programming interface (API) gateway configured to receive the metrics. The system also includes an ingestion layer configured to ingest the metrics to form the ingested metrics. The system also includes a tumbling window processor configured to process the ingested metrics and the events into heat maps, sort the heat maps into the time slices, and populate the time slices with the ingested metrics.

Abstract: In implementations of the present disclosure, a solution for optimization of a learning network in an equivalent class space is provided. In this solution, base paths running through layers of a learning network are determined. Each node utilizes an activation function with a scaling invariant property to process an input from a node of a previous layer, each base path comprises a single node in each layer, and processing in the base paths is linearly independent from each other. A combined value of parameters associated with nodes in each base path is updated. A parameter associated with a node is used to adjust an input obtained from a node of a previous layer. Values of parameters associated with nodes in the base paths are updated based on updated combined values of parameters. Through this solution, optimization efficiency can be improved and more accurate optimized values of parameters are achieved.

Abstract: A distributed system for software verification includes a plurality of processors, each of which actively executes a verified software to perform a verifying operation of a corresponding target object under no external instruction, wherein the plurality of processors independently perform verifying operations of different target objects; a switch including a plurality of first connection ports electrically coupled to the plurality of processors, respectively, and a second connection port electrically coupled to each of the plurality of first connection ports; and a data collector electrically coupled to the second connection port. An information report is generated and sent to the data collector via the corresponding first connection port and the second connection port in response to the verifying operation.

Abstract: Techniques for testing may include: receiving a first command at a framework, wherein the first command includes first information identifying a first test to be executed and a first collector that collects data during execution of the first test; and responsive to receiving the first command, perform processing to execute the first command, wherein the processing is coordinated by the framework and wherein the processing includes: executing the first test; collecting test data, wherein the test data is collected by the first collector during execution of the first test; and generating a report regarding the test data collected while executing the first test.

Abstract: A computer-implemented method of mapping a process model of a software application executed by a hosting platform. A series of actions taken by a user with respect to user interfaces generated by the software application are simulated through one or more application programming interface (API) calls. The user interfaces include a form and the series of actions includes opening the form. A plurality of user interface fields of the form are then identified through one or more other API calls executed while impersonating a session of a user under test. At least some of the user interface fields are set to known values and the form is submitted to the software application after the fields have been set. Changes to field-related and record-related information resulting from the submission are then gathered and a process model of the software application is determined based upon the gathered information.

Abstract: Features of each ordered step of a recorded test script for an application under test (AUT) are generated. The ordered steps are segmentally organized over actions based on feature relatedness of the ordered steps. Each action includes a contiguous subset of the ordered steps. The actions are hierarchically organized within a tree based on an ordered feature relatedness of the actions. The tree has nodes respectively corresponding to the actions. The actions are abstractedly related to one another based on an unordered feature relatedness of the actions to generate abstracted actions that each generalize a subset of the actions. Abstracted action groups are generated based on the abstracted actions and the tree. Each abstracted action group includes abstracted actions mapping to corresponding nodes of the tree within a same branch. The ordered steps of the recorded test script are collapsed into a series of instantiations of the abstracted action groups.

Abstract: The present invention is a process by which an engineer can provide as input the design, functional verification goals, and other abstract design details, and receive as output an agent which can be integrated into traditional test benches and will generate stimuli to automatically hit the functional coverage goals for the design. The present invention may employ a system which includes a learning configurator, a pre-trained learning test generator, and a test bench. The pre-trained learning test generator is communicatively coupled to the generator and notably comprises a learning algorithm.

Abstract: A GUI testing device may be configured to execute a testing state machine for interacting with a software application to generate an initial screen of a GUI. The GUI testing device may be configured to determine a current state in the testing state machine based upon a matching trigger target in the initial screen to a given state. The current state may include an operation, and the operation may associate with a trigger target to operate on. The trigger may include a source state, a destination state, and a trigger target. The operation may include a user input operation, and an operation trigger target. The GUI testing device may be configured to perform the operation on the matching trigger target in the initial screen to generate a next screen of the GUI, and advance from the current state to a next state based upon the trigger.

Abstract: Systems, computer program products, and methods are described herein for testing an application with dynamically linked security tests. The present invention may be configured to perform, using a request engine, based on the first data, and based on test protocols stored in a first data structure, a first security test on an application. The present invention may be further configured to determine, based on determining that the application failed the first security test and based on a second data structure, whether the first security test is linked to one or more other security tests, where the second data structure includes security test sequences linking security tests and/or data to transmit from the first security test to the one or more other security tests, and provide, to the first queue, one or more other security tests to which the first security test is linked.

Abstract: This disclosure relates generally to a human machine interface (HMI) application, and more particularly to modular testing kits and testing framework for testing the HMI application. In one embodiment, a method for developing a modular testing kit for testing a modular component of the HMI application is disclosed. The method may include generating a plurality of test cases for testing the modular component based on a definition of each of a set of interfaces configured to implement the modular component on a target platform, developing the modular testing kit to execute the plurality of test cases by simulating each of the set of interfaces, and creating a test script for each of the plurality of test cases using the modular testing kit.

Abstract: A method includes generating a behavioral state for an endpoint device based on actor identities and corresponding subject identities for a plurality of operations wherein for each operation, a respective actor represented by a respective actor identity performs the operation upon a respective subject represented by a respective subject identity. Performance of a later operation by an actor with an actor identity upon a subject with a subject identity is recorded and the actor identity and the subject identity are used to determine that the performance of the later operation does not match the behavioral state and indicates a security risk.

Abstract: A method of performing auto-naming of content includes: receiving an auto-naming command for the content; performing auto-naming of the content by using different parameters according to different content types to obtain at least one auto-naming result for the content; and displaying the auto-naming result.

Abstract: A script directing functional testing of an application specifies values for properties of a specific graphical user interface (GUI) and specifies an image of the specific GUI element. A computing device detects whether the application has displayed the specific GUI element by evaluating the values for the properties of the specific GUI element against GUI elements that the application has displayed. If unsuccessful, the computing device detects whether the application has displayed the specific GUI element by evaluating the image of the specific GUI element against a screen image including the GUI elements that the application has displayed.

Abstract: Systems, methods, and computer-readable media are described for selecting, at build time, a respective compiler and/or a respective set of compiler options for each section of code to be compiled such that the compiler/compiler options selected for each code section are optimized for that code section with respect to one or more metrics. Abstract syntax tree (AST) analysis and semantic analysis may be performed at build time for each section of code to identify the compiler/compiler options that produce compiled object code for that code section that maximizes or minimizes a desired metric. The metric according to which compiler/compiler option optimization is performed may be any suitable metric including, without limitation, performance, binary size, security, reliability, scalability, and so forth.

Abstract: Aspects of the present disclosure relate to systems and methods that help automate the generation of a test vector. Every functional product, be it a service, device, or combination thereof, has one or more documents associated with it. These documents may include such documentation as: (1) Release Notes; (2) Configuration Guides; (3) command line interfaces (CLIs)/application program interfaces (APIs); (4) Data Sheets; (5) Installation Guides; (6) User Manuals; (7) Errata notices; and (8) other documentation. In embodiments, command data-models and ranges and sequence of valid inputs to the data-model are extracted from natural language expressions in documentation related to a specific product. In embodiments, this extracted information is used to generate one or more table of attributes and properties or constraints about which a test vector may be generated for testing a command.

Abstract: In one example, a method for testing a multi-threaded application includes running an initial test of the multi-threaded application and collecting thread generation data and determining the thread hierarchy. The thread execution is then modified to produce a modified configuration and a second test is run with the modified configuration. A device for testing of multi-threaded applications is also provided.

Abstract: The present invention is a process by which an engineer can provide as input the design, functional verification goals, and other abstract design details, and receive as output an agent which can be integrated into traditional test benches and will generate stimuli to automatically hit the functional coverage goals for the design. The present invention may employ a system which includes a learning configurator, a learning-based test generator, and a test bench. The learning test generator is communicatively coupled to the generator and notably comprises a learning algorithm.

Abstract: A rollback refinement process for software testing using software models in a model-based testing environment. The method includes: identifying a model having a set of model elements assigned relevance scores based on correspondence between each model element in the set of model elements and a pre-defined suspect element, the model producing an expected behavior when running a test case; generating a reduced model on condition that the reduced model include only model elements assigned a threshold reduction score corresponding to the relevance scores; determining that the reduced model does not reproduce the expected behavior when running the test case; rolling back a model element into the reduced model to produce a revised model; and responsive to the revised model not reproducing the expected behavior, performing a rollback refinement of the revised model based on a number of model elements rolled back into the reduced model.

Abstract: Certain aspects of the present disclosure provide techniques for managing the deployment of software applications based on application performance data. Embodiments include transmitting, to a performance testing system, information identifying a test build of a software application component in a first deployment environment in a software development pipeline to test. Performance information for the test build of the software application component is received from the performance testing system. The received performance information for the test build of the software application component is compared to one or more of predetermined performance parameters or historical performance data for the software application component. Based on the comparing, the system takes one or more actions to manage deployment of the test build of the software application component to a second deployment environment in the software development pipeline.

Abstract: A context-sensitive process guidance system provides process guidance based on context data from a system used in performing the process. The system may be a work item tracking system or similar project management tool. Process guidance may provide information to a user performing actions related to a process. Context data may be derived from various sources, including the state of an application at the time of the request for process guidance, the user making the request and the role assigned to that user in the process, or the data displayed for the user who made the request. The context-sensitive process guidance system may display context-sensitive process guidance content and provide links to related process guidance content based on the context data.

Abstract: Example implementations relate to separating verifications from test executions. Some implementations may include a data capture engine that captures data points during test executions of the application under test. The data points may include, for example, application data, test data, and environment data. Additionally, some implementations may include a data correlation engine that correlates each of the data points with a particular test execution state of the application under test based on a sequence of events that occurred during the particular test execution state. Furthermore, some implementations may also include a test verification engine that, based on the correlation of the data points, verifies an actual behavior of the application under test separately from the particular test execution state.

Abstract: A system and method for remediating and redeploying out of compliance applications and cloud services uses integrated vulnerability information. The system and method may use Conda packages and compliance reports to perform the monitoring, remediation and redeployment.

Abstract: This invention generally relates to a process, system and computer code for enabling users to create adapters that enable application automation by collecting automation information; locate application controls and tracking changes between an older and a newer version of the application, such changes to include addition of one or more new data fields, removal of one or more data fields, change in data field type (i.e. type of data held in the field); change field layout; and change the underlying technology framework of the application; to present the changes using an exception management model to the user, so user can by way of example provide feedback in a visual instead of programmatic manner; store the changes, so as to make the adapters resilient to application changes and upgrades; and incorporating the changes to upgrade the application.

Abstract: A system and an operating method thereof include a system under test (SUT) having a hardware array of flash storages, partitions including logical volumes, a kernel subsystem including an operating system, and an application layer including services, applications, and/or systems. The application layer receives test drivers and corresponding test fixtures from an external source. The SUT is segmented into subsystems of interest, and a different test using a different test fixture is applied to each subsystem. The tests are monitored and the results of the tests are collected and verified against suitable benchmarks. Results and associated data are archived.

Abstract: In an embodiment, a modeling component generated in a first modeling environment can be implemented in a second modeling environment. The modeling component is executed in the first modeling environment. A first behavior of the modeling component in the first modeling environment is obtained based on the executing. The modeling component is then executed in the second modeling environment. A second behavior of the modeling component in the second modeling environment is obtained based on the executing the modeling component in the second modeling environment. The first behavior is compared to the second behavior. Based on the comparing, it is verified that the second behavior of the modeling component complies with one or more modeling requirements of the second modeling environment. The second behavior of the modeling component may be implemented in the second modeling environment.

Abstract: The computing device receives information representing a design of an experiment. The design of the experiment comprises a plurality of test cases. Each element of a test case of the design is a test condition for testing one of factors for the experiment. The computing device generates a graphical representation based on the information. The graphical representation comprises a plurality of axes for respective ones of factors used in a fractal sequence. Each of the plurality of axes has two endpoints comprising a first endpoint that corresponds to a first level of a respective factor, and a second endpoint that corresponds to a second level of the respective factor. The computing device plots, on the graphical representation, data corresponding to one or more test cases of the plurality of test cases. The computing device displays an interactive graphical user interface comprising the graphical representation with the plotted data.

Abstract: Systems, methods, and apparatus for automatically parallelizing code segments are provided. For example, an environment includes a profiling agent, a parallelization agent, and a verification agent. The profiling agent executes a code segment and generates a profile of the executed code segment. The parallelization agent analyzes the code segment to determine whether a parallelizable portion is present in the code segment. When a parallelizable portion is present, the parallelization agent determines, based on the profile of the executed code segment, whether to parallelize the parallelizable portion of the code segment. If it is determined to parallelize the parallelizable portion of the code segment, the parallelization agent automatically parallelizes the parallelizable portion of the code segment. The verification agent verifies the functionality and/or correctness of the parallelized code segment.

Abstract: A method and device for design driven development based automation testing is disclosed. The method includes receiving a design document comprising an overview of a proposed product. The method further includes extracting functionality of the proposed product based on an analysis of at least one design object in the design document. The method includes converting the functionality of the proposed product into at least one test script based on the at least one design object. The method includes replacing one or more of the at least one design object with a corresponding real object in a developed product associated with the proposed product. The method further includes executing the at least one test script to test the developed product, in response to the replacing.

Abstract: A user interface (“UI”) code coverage system and process determines the coverage by test code of UI Components in a computer program. In other words, the UI code coverage system and process determine whether the test code tests a UI Component. In at least one embodiment, the UI code coverage system and process identify the particular UI Components tested by the test code, provide coverage (i.e. tested) statistics, provide a visualization of the UI Component tested, provide a visualization of tested and not-tested (also referred to respectively as “covered” and “not covered”) UI Components of a UI Screen.

Abstract: A method includes receiving, by a computing system that includes an application repository, an updated version of an application. The method also includes determining whether the particular computing device contributes additional test scope for the updated version of the application beyond existing test scope for the updated version of the application that is contributed by the group of computing devices. The method further includes, sending the updated version of the application to the particular computing device for installation in response to determining that the particular computing device contributes additional test scope.

Abstract: A device management apparatus includes circuitry configured to: acquire an updating program for updating a device; transmit the updating program to the device; receive introduction success/failure information indicating whether the updating program is successfully introduced, from the device; acquire, if the introduction success/failure information indicates success in introduction, a verification program for verifying an operation of the device updated with the updating program; and verify whether the operation of the updated device can be guaranteed based on the verification program.

Abstract: A system, method and computer-readable medium for testing a service-based application identifies a set of use cases for the service-based application. Each use case in the set is mapped to a test case to generate a first set of test cases. Each of the test cases is refined to generate an updated set of test cases and the updated set of test cases is executed.

Abstract: A method includes receiving, via a processor, a model, a test case, and one or more suspect elements. The method includes calculating, via the processor, a relevance score for each model element of the model based on the one or more suspect elements. The method includes setting, via the processor, a threshold reduction score. The method includes generating, via the processor, a reduced model by reducing the model based on the relevance scores and the threshold reduction score. The method includes evaluating, via the processor, the reduced model based on the test case. The method includes detecting, via the processor, the reduced model reproduces an expected behavior. The method includes outputting, via the processor, a result model in response to detecting the reduced model reproduces the expected behavior. The method includes modifying, via the processor, an application associated with the model based on the reduced model.

Abstract: Techniques for configuring a test system that enable simple specification of a degree of concurrency in testing separate functional portions of a semiconductor device. For a test flow with multiple sub-flows, the pins accessed in connection with each sub-flow may define a flow domain. Site regions, each associated with a flow domain, may be defined. Tester sites may be associated with each of these flow domain specific site regions and independently operating resources may be assigned to these tester sites. A second portion of the defined site regions may be associated with tester sites, but resources assigned to these site regions may be accessed from multiple flow domains. Test blocks, even if not developed for concurrent execution, may be executed concurrently using resources in the flow domain specific site regions. Flexibility is provided to share resources through the use of the second portion of the site regions.

Abstract: A computer-implemented method, computer program product, and system is provided for determining requirements for testing software. In an implementation, a method may include inspecting contents of a test case, including source code of the test case. The method may also include identifying at least one of: at least one characteristic of a test machine and at least one characteristic of a resource required to execute the test case correctly. The method may further include compiling a list of requirements for the test case to execute correctly based upon, at least in part, the at least one of the at least one characteristic of the test machine and the at least one characteristic of the resource.

Abstract: A computer documentation validation method, system, and computer program product, include translating a natural language of a computer software documentation into a machine instruction, detecting an error in the computer software documentation during an execution of the machine instruction on a software for the computer software documentation, and highlighting a location in the computer software documentation including the error detected by the detecting.

Abstract: A method and a system to create a software test environment on demand are described. An example system includes a dependency module to, upon receiving a command identifying a primary function to be created in a test environment, identify one or more dependencies of the primary function. The dependencies are other functions or databases that the primary function depends upon. The dependency module generates a topology of the test environment that indicates the relationship of the dependencies to the primary function. A provisioning module provisions a plurality of pools based on the topology. An enterprise service bus (ESB) routing module updates ESB routing of the primary function to route to the plurality of pools in the test environment. A credentials module provides credentials of the pools in the test environment.

Abstract: Methods, systems, and computer programs are presented for managing the sensory stimulus generated by a head mounted display (HMD). One method includes an operation for identifying the game state of a game being executed for display on a head mounted display (HMD). Further, the method includes an operation for determining the game intensity value for the game state based on user intensity ratings obtained from user play of the game. The user intensity ratings represent the level of sensory stimulus presented to users during the user play of the game via the HMDs of the respective users. Further, the method includes an operation for presenting the game intensity value to the HMD for rendering during execution of the game.

Abstract: Techniques described herein relate to generating new assessment items and updating existing assessment items. Input data may be received corresponding to the addition, removal, or modification of assessment components within assessment items, and may cause immediate generation and validation of corresponding markup language data blocks, thereby allowing for interactive construction and automated encoding of assessment items. Additional techniques described herein relate to determining compatible scoring types for assessment items and generating and embedding markup language data blocks corresponding to assessment item scoring data.

Abstract: A method and associated system for continuously optimizing data archive management scheduling. A flow network is modeled, which creates vertexes organized in multiple levels and creating multiple edges sequentially connecting the vertexes of the multiple levels. The multiple levels consist of N+1 levels denoted as LEVEL0, LEVEL1, . . . , LEVELN, wherein N is at least 2, wherein LEVEL0 includes a source vertex, wherein LEVEL1 includes at least one task vertex associated with at least one archive management task, wherein EDGEK is directed from the vertex of LEVELK?1 to the vertex of LEVELK for K=1, . . . , N, wherein EDGE1 indicates that a first task associated with a first task vertex of LEVEL1 is scheduled by a job scheduler, and wherein EDGE1 has a first weight equal to a time unit to perform the first task.

Abstract: A computer system includes creating a test suite, wherein the test suite includes a plurality of test cases for execution on a plurality of test agents. The method distributes a first portion of test cases to any available test agents, wherein each test case out of the first portion of test cases does not have any associated preconditions. The receives test results and event information for a first test case out of the first portion of test cases from a first test agent. Responsive to determining the event information for the first test case includes a satisfied condition for a second test case with one or more associated preconditions, the method determines whether the satisfied condition for the second test case relates to a global variable or local variable.

Abstract: A deployment package descriptor may be obtained. One or more application files that include application content in a plurality of different data formats may be obtained. An analysis of the application content included in the obtained one or more application files may be initiated, based on a plurality of different data formats and descriptive information associated with the obtained application files, the descriptive information included in the obtained deployment package descriptor.

Abstract: This apparatus is provided with a storage device and a CPU. The storage device has two or more modules stored therein. The CPU acquires, from the storage device, a first module and a second module from among the modules stored in the storage device. Furthermore, the CPU creates a message that indicates whether a hierarchical structure of the first module matches a hierarchical structure of the second module.