Abstract: Systems and methods for automatically identifying in a dataset insufficient data for learning, or records with anomalous combinations of feature values, by partition of numeric and/or categorical data space into human-interpretable regions are disclosed. The method comprises: receiving a dataset of numeric and/or categorical features with a plurality of observations. Calculating observation density for each observation according to a distance or anomaly based metric, and receiving a density measurement. Partitioning the dataset along the numeric and/or categorical features according to the density measurement of each observation by a perpendicular cut along the feature spaces, receiving a map of a plurality of hyper-rectangular shapes representing various levels of density including empty spaces.

Abstract: A computer that receives a set of names of coverage events. The computer creates, by a machine learning-based technique, groups from the set of received names of the coverage events based on the set of names of the coverage events. The computer generates a cross product coverage model from the created groups and identifies subgroups of uncovered events for each of the created groups.

Abstract: A system and method for generating data slices for validating a classifier and validating the classifier. The classifier is trained using a training data set to train the underlying machine learning algorithm. Data is passed through the trained classifier to obtain results. The results are scored to determine the likelihood that the classifier correctly classified the data. Features are identified in the data set that can be used to validate the classifier. Based on the identified features at least one data slice in the data set is identified. The classifier is validated using the at least one data slice.

Abstract: An example system includes a processor to receive a template-aware coverage data that tracks probabilities of events in a list of events being hit for a set of test-templates over a first and second predetermined period of time. The processor is to generate a hit prediction score for each combination of unhit event in the events and each test-template in the set of test-templates of the second predetermined period of time. The hit prediction score indicates a probability of an unhit event being hit by a particular test-template in a future third predetermined period of time based on the template-aware coverage data and similarities between the events and the test-templates. The processor is to generate a template score for each test-template based on the hit prediction scores for each test-template. The processor is to select a test-template from the set of test-templates based on the template score.

Abstract: A method, apparatus and product for testing a system under development. The method comprises obtaining information including a current version of a code, and predicting that a subset of code elements from the current version of the code are bug-prone elements. The method comprises determining one or more coverage events corresponding to the bug-prone elements. The method further comprises determining a testing policy based on the one or more coverage events that correspond to the bug-prone elements, wherein the testing policy is determined based on a statistical analysis of coverage likelihood of tests generated based on a test template for each coverage event of the one or more coverage events. The method further comprises implementing the testing policy.

Abstract: A method, apparatus and product for reducing a number of test templates in a test suite. The method comprises determining, for a first test template of the test suite, a first probabilities vector comprising a first plurality of coverage probabilities with respect to a set of coverage events. The method comprises determining, for a second test template of the test suite, a second probabilities vector comprising a second plurality of coverage probabilities with respect to the set of coverage events. The method further comprises determining that the first test template is statistically dominant over the second test template based on the first probabilities vector and based on the second probabilities vector. The method further comprises providing an output based on the determination of the statistically dominant test template.

Abstract: A method, apparatus and product for reducing a number of test templates in a test suite. The method comprises determining, for a first test template of the test suite, a first probabilities vector comprising a first plurality of coverage probabilities with respect to a set of coverage events. The method comprises determining, for a second test template of the test suite, a second probabilities vector comprising a second plurality of coverage probabilities with respect to the set of coverage events. The method further comprises determining that the first test template is statistically dominant over the second test template based on the first probabilities vector and based on the second probabilities vector. The method further comprises providing an output based on the determination of the statistically dominant test template.

Abstract: A method, and apparatus and a computer program product for determining coverage in hardware verification based on relations between coverage events. The method comprises generating an over-approximation model of the hardware being verified to perform formal verification thereof with respect to a target coverage event being utilized in the verification process along with a set of coverage events. A score indicating an estimated conditional probability to hit the target coverage event in the verification process, given that the coverage event is hit in the verification process, may be determined for each coverage event based on the formal verification. The method further comprises selecting test suits to be executed in the verification process based on the scores and the test suits probability to hit each coverage event. The verification process may be the performed the selected test suits in order to cover the target coverage event.

Abstract: A method, apparatus and product for testing a system under development. The method comprises obtaining information including a current version of a code, and predicting that a subset of code elements from the current version of the code are bug-prone elements. The method comprises determining one or more coverage events corresponding to the bug-prone elements. The method further comprises determining a testing policy based on the one or more coverage events that correspond to the bug-prone elements, wherein the testing policy is determined based on a statistical analysis of coverage likelihood of tests generated based on a test template for each coverage event of the one or more coverage events. The method further comprises implementing the testing policy.

Abstract: An example system includes a processor to receive a template-aware coverage data that tracks probabilities of events in a list of events being hit for a set of test-templates over a first and second predetermined period of time. The processor is to generate a hit prediction score for each combination of unhit event in the events and each test-template in the set of test-templates of the second predetermined period of time. The hit prediction score indicates a probability of an unhit event being hit by a particular test-template in a future third predetermined period of time based on the template-aware coverage data and similarities between the events and the test-templates. The processor is to generate a template score for each test-template based on the hit prediction scores for each test-template. The processor is to select a test-template from the set of test-templates based on the template score.

Abstract: A computer that receives a set of names of coverage events. The computer creates, by a machine learning-based technique, groups from the set of received names of the coverage events based on the set of names of the coverage events. The computer generates a cross product coverage model from the created groups and identifies subgroups of uncovered events for each of the created groups.

Abstract: A method, apparatus and product for solving CSP comprising vectors of unknown size. The method comprises generating a structural skeleton tree of a problem description, wherein the structural skeleton tree comprises a node representing a vector of unknown size and a node representing a size of the vector; determining a vector size Constraint Satisfaction Problem (CSP) based on the structural skeleton tree, wherein said determining comprises projecting over-approximated constraints on the size of the vector based on operators used on the vector or elements thereof; solving the vector size CSP to determine the size of the vector; modifying the structural skeleton tree to set the size of the vector and to include nodes for each element in the vector, whereby obtaining a CSP; and solving the CSP.

Abstract: A computer-implemented method, an apparatus and a computer program for automatically extracting useful information for functional verification. The method comprising performing repeatedly both operating an instruction generator associated with a Design Under Test (DUT), whereby a generated instruction is determined, the generated instruction having one or more instruction attributes; and collecting information relating to the generated instruction. Based on the generated instruction and the collected information, a classification technique is utilized to classify the information based on the instruction attributes.

Abstract: A method, apparatus and product to be used in verification. The method comprising: based on a test generation input that defines a plurality of requirements automatically determining a mutated test generation input, wherein the mutated test generation input defining a mutated requirement which is absent from the test generation input, wherein the mutated requirement is based on a requirement of the plurality of requirements and contradicts, at least in part, the plurality of requirements; and generating one or more test-cases based on the mutated test generation input, whereby the one or more test-cases violate at least one requirement of the test generation input.

Abstract: A method, apparatus and product for generating elements based on generation streams. The method comprises: obtaining one or more generation streams, wherein the streams comprise elements, wherein each element is a formal specification of an operation that stimulates a system, wherein based on each of the generation streams one or more alternative stimuli for the system can be generated, which stimuli comprises operations according to the elements; and generating a stimuli in accordance with the one or more generation streams, wherein the stimuli comprises at least one hybrid operation, wherein the hybrid operation complies simultaneously with two or more elements of the one or more generation stream, whereby the stimuli is comprised of a number of operations that is smaller than a sum of the numbers of elements of the one or more generation streams.

Abstract: Systems and methods for modeling test space for verifying system behavior, using one or more auxiliary variables, are provided. The method comprises implementing a functional coverage model including: one or more attributes, wherein respective values for the attributes are assigned according to a test plan, and one or more constraints defining restrictions on value combinations assigned to the attributes, wherein the restrictions are Boolean expressions defining whether said value combinations are valid; determining a set of valid value combinations for the attributes that satisfy the restrictions to define the test space for verifying the system behavior; and determining relevant auxiliary variables and a corresponding function for said auxiliary variables to reduce the complexity associated with modeling the test space.

Abstract: A method, apparatus and product. The method comprising automatically determining an abstract CSP based on a formally defined problem having interconnected sub-problems, wherein the abstract CSP corresponds to the problem, wherein the abstract CSP has a reduced complexity in comparison to a CSP representing the problem, wherein the abstract CSP captures the interconnection between the sub-problems and reduces the details of each sub-problem, wherein the abstract CSP comprises constraints over variables, wherein each variable having an associated domain; and repeatedly: (1) propagating constraints of the abstract CSP to reduce domains of the abstract CSP; (2) selecting a sub-problem to solve; (3) solving the sub-problem; and (4) updating the abstract CSP with values in accordance with the solution of the sub-problem. Whereby, a solution to the formally defined problem is determined based on the solutions to the sub-problems.

Abstract: Coverage analysis may be performed using reverse time. The coverage analysis may be based on last hit data. The last hit data may comprise a timestamp indicating a last time in which a coverage event was covered. Utilizing last hit data instead of first hit data as is known in the art enables distinction between coverage goals that were never covered and coverage goals that were not covered lately.

Abstract: A computer-implemented method, an apparatus and a computer program for automatically extracting useful information for functional verification. The method comprising performing repeatedly both operating an instruction generator associated with a Design Under Test (DUT), whereby a generated instruction is determined, the generated instruction having one or more instruction attributes; and collecting information relating to the generated instruction. Based on the generated instruction and the collected information, a classification technique is utilized to classify the information based on the instruction attributes.

Abstract: A logic verification program, method and system that segments simulation results and then processes the resulting segments separately, and optionally in parallel, reduces memory and other system requirements and improves efficiency of verification of digital logic designs. The verification process fixes up event dependency check for past-directed checkers by including additional information with each segment after an initial segment that describes at least a portion of a state of the logic design, so that resultant events in the current segment that are caused by events in the previous segment(s) can be traced back to those events. Future directed checks are fixed-up by either repeating a failed check with a concatenation of the current segment and a next segment, or by providing an overlap between segments to ensure that the expected time duration between a causative event and the resulting event are included within the same segment file.