Abstract: A method, apparatus and a product for determining a performance measurement of predictors. The method comprises obtaining a dataset comprising data instances. Each data instance is associated with a label; obtaining a predictor. The predictor is configured to provide a prediction of a label for a data instance; determining a plurality of data slices that are subsets of the dataset. computing, for each data slice in the plurality of data slices and based on an application of the predictor on each data instance that is mapped to the data slice, a performance measurement that is indicative of a successful label prediction for a data instance comprised by the data slice, whereby obtaining a plurality of performance measurements; based on the plurality of performance measurements, computing a performance measurement of the predictor over the dataset; if the performance measurement of the predictor is below a threshold, performing a mitigating action.

Abstract: A method, system and computer program product, the method comprising: obtaining a first model trained upon cases and labels, the first model providing a prediction in response to an input case; obtaining a second model trained using the cases and indications whether a predictions of the first model are correct, the second model providing a correctness prediction for the first; determining a case for which the second model predicts that the first provides an incorrect prediction; further training the first model also on a first corpus including the case and a label, thereby improving performance of the first model; providing the case to the first model to obtain a first prediction; and further training the second model also on a second corpus including the case and a correctness label, the correctness label being “correct” if the first prediction is equal to the label, thereby improving performance of the second model.

Abstract: A method, apparatus and product for identifying data drifts.

Abstract: A method, apparatus and a computer program product for automated data slicing based on an Artificial Neural Network (ANN). The method comprising: obtaining an ANN, wherein the ANN is configured to provide a prediction for a data instance, wherein the ANN comprises a set of nodes having interconnections therebetween; determining an attribute vector based on a subset of the nodes of the ANN; determining, based on the attribute vector, a plurality of data slices; obtaining a testing dataset comprising testing data instances; computing, for each data slice, a performance measurement of the ANN over the data slice, wherein said computing is based on an application of the ANN on each testing data instance that is mapped to the data slice; and performing an action based on at least a portion of the performance measurements of the data slices.

Abstract: A method and a computer program product for identification and improvement of machine learning (ML) under-performance The method comprises slicing data of ML model based on a functional model representing requirements of a system utilizing the ML model. The functional model comprises a set of attributes and respective domain of values. Each data slice is associated with a different valuation of one or more attributes of the functional model. Each data instance of the ML model is mapped to one or more data slices, based on valuation of the attributes. A performance measurement of the ML model over is computed for each data slice, based on an application of the ML model on each data instance that is mapped to the data slice. A Determination whether ML model adheres to a target performance requirement may be performed based on the performance measurements of the data slices.

Abstract: From metadata corresponding to a narrative text, a first encoding is constructed, the first encoding comprising a standardized text string, the first encoding formed according to an encoding scheme. A specified portion of the standardized text string of the first encoding is marked as an anchor term. A correspondence between the first encoding and a second encoding is tested using the encoding scheme and a Natural Language Processing engine, responsive to finding the anchor term within the narrative text. The second encoding corresponds to a text window. The text window comprises a portion of the narrative text comprising an instance of the anchor term and a word within a predetermined distance from the instance. Responsive to the second encoding being identical to the first encoding, the narrative text is annotated, the annotating creating new data linking the narrative text with the second encoding.

Abstract: A method, apparatus and a product for determining a performance measurement of predictors. The method comprises obtaining a dataset comprising data instances. Each data instance is associated with a label; obtaining a predictor. The predictor is configured to provide a prediction of a label for a data instance; determining a plurality of data slices that are subsets of the dataset. computing, for each data slice in the plurality of data slices and based on an application of the predictor on each data instance that is mapped to the data slice, a performance measurement that is indicative of a successful label prediction for a data instance comprised by the data slice, whereby obtaining a plurality of performance measurements; based on the plurality of performance measurements, computing a performance measurement of the predictor over the dataset; if the performance measurement of the predictor is below a threshold, performing a mitigating action.

Abstract: Systems, methods, and computer-readable media are described for expanding test space coverage for testing performed on a System Under Test (SUT) through iterative test case generation from combinatoric pairwise outputs. At each test case generation iteration, a new set of test vectors is generated that provides complete pairwise coverage of the test space but that does not include any overlapping test vector with any previously generated set of test vectors. As such, cumulative m-wise test space coverage (where 2<m?n) is incrementally increased through each iteration until the iterative process ceases when a desired percentage of m-wise test space coverage is achieved.

Abstract: A method, system and computer program product, the method comprising: obtaining a first model trained upon cases and labels, the first model providing a prediction in response to an input case; obtaining a second model trained using the cases and indications whether a predictions of the first model are correct, the second model providing a correctness prediction for the first; determining a case for which the second model predicts that the first provides an incorrect prediction; further training the first model also on a first corpus including the case and a label, thereby improving performance of the first model; providing the case to the first model to obtain a first prediction; and further training the second model also on a second corpus including the case and a correctness label, the correctness label being “correct” if the first prediction is equal to the label, thereby improving performance of the second model.

Abstract: A method, system and computer program product, the method comprising: receiving a user document describing at least a part of a system to be tested; computing a similarity measurement between the user document and documents in a document corpus, each describing at least part of a system and associated with a combinatorial model representing the at least part of the system, wherein the combinatorial model comprises a set of attributes, a respective domain for each attribute defining possible values for the attribute, and restrictions, wherein each restriction comprises at least one attribute, a respective value and a logical operator, based upon the similarity measurement, selecting selected documents from the document corpus; obtaining one or more combinatorial models corresponding to the selected documents; selecting elements from the combinatorial models; generating an initial combinatorial model for the user document, wherein said generating comprises adding the elements to the initial combinatorial model.

Abstract: A method, apparatus and a computer program product for automated data slicing based on an Artificial Neural Network (ANN). The method comprising: obtaining an ANN, wherein the ANN is configured to provide a prediction for a data instance, wherein the ANN comprises a set of nodes having interconnections therebetween; determining an attribute vector based on a subset of the nodes of the ANN; determining, based on the attribute vector, a plurality of data slices; obtaining a testing dataset comprising testing data instances; computing, for each data slice, a performance measurement of the ANN over the data slice, wherein said computing is based on an application of the ANN on each testing data instance that is mapped to the data slice; and performing an action based on at least a portion of the performance measurements of the data slices.

Abstract: A method and a computer program product for identification and improvement of machine learning (ML) under-performance The method comprises slicing data of ML model based on a functional model representing requirements of a system utilizing the ML model. The functional model comprises a set of attributes and respective domain of values. Each data slice is associated with a different valuation of one or more attributes of the functional model. Each data instance of the ML model is mapped to one or more data slices, based on valuation of the attributes. A performance measurement of the ML model over is computed for each data slice, based on an application of the ML model on each data instance that is mapped to the data slice. A Determination whether ML model adheres to a target performance requirement may be performed based on the performance measurements of the data slices.

Abstract: A method, apparatus and product for identifying data drifts. The method comprising: obtaining a seen dataset, wherein the seen dataset comprises seen instances, each of which comprising feature values in a feature space; determining a first measurement of a statistical metric of the seen dataset; obtaining an unseen dataset, wherein the unseen dataset comprises unseen instances, each of which comprising features values in the feature space; determining a second measurement of the statistical metric of the unseen dataset; identifying a data drift in the unseen dataset with respect to the seen dataset based on the first and second measurements of the statistical metric; and performing a responsive action based on the identification of the data drift.

Abstract: A method, apparatus and product for identifying data drifts.

Abstract: From metadata corresponding to a narrative text, a first encoding is constructed, the first encoding comprising a standardized text string, the first encoding formed according to an encoding scheme. A specified portion of the standardized text string of the first encoding is marked as an anchor term. A correspondence between the first encoding and a second encoding is tested using the encoding scheme and a Natural Language Processing engine, responsive to finding the anchor term within the narrative text. The second encoding corresponds to a text window. The text window comprises a portion of the narrative text comprising an instance of the anchor term and a word within a predetermined distance from the instance. Responsive to the second encoding being identical to the first encoding, the narrative text is annotated, the annotating creating new data linking the narrative text with the second encoding.

Abstract: Systems, methods, and computer-readable media are described for expanding test space coverage for testing performed on a System Under Test (SUT) through iterative test case generation from combinatoric pairwise outputs. At each test case generation iteration, a new set of test vectors is generated that provides complete pairwise coverage of the test space but that does not include any overlapping test vector with any previously generated set of test vectors. As such, cumulative m-wise test space coverage (where 2<m?n) is incrementally increased through each iteration until the iterative process ceases when a desired percentage of m-wise test space coverage is achieved.

Abstract: A method, system and computer program product, the method comprising: receiving a partial description of a combinatorial model defining a test space and comprising a set of attributes, a respective domain for each attribute defining possible values for the attribute, and restrictions, each comprising an attribute, a respective value and a logical operator, the partial description comprising an attribute and a value thereof; computing a similarity measurement between the partial description and each model in a corpus, based on similarity of attributes and domains, and not on a test space defined by compared models, whereby similarity between models is indifferent to a size of the test space of the models; selecting models from the corpus according to the computed similarity measurements; determining elements from the selected models to be suggested for modifying the partial description; and outputting the suggestions to modify the partial description useful for completing the combinatorial model.

Abstract: A method, system and computer program product, the method comprising: receiving a user document describing at least a part of a system to be tested; computing a similarity measurement between the user document and documents in a document corpus, each describing at least part of a system and associated with a combinatorial model representing the at least part of the system, wherein the combinatorial model comprises a set of attributes, a respective domain for each attribute defining possible values for the attribute, and restrictions, wherein each restriction comprises at least one attribute, a respective value and a logical operator, based upon the similarity measurement, selecting selected documents from the document corpus; obtaining one or more combinatorial models corresponding to the selected documents; selecting elements from the combinatorial models; generating an initial combinatorial model for the user document, wherein said generating comprises adding the elements to the initial combinatorial model.

Abstract: A method, system and computer program product, the method comprising: receiving a partial description of a combinatorial model defining a test space and comprising a set of attributes, a respective domain for each attribute defining possible values for the attribute, and restrictions, each comprising an attribute, a respective value and a logical operator, the partial description comprising an attribute and a value thereof; computing a similarity measurement between the partial description and each model in a corpus, based on similarity of attributes and domains, and not on a test space defined by compared models, whereby similarity between models is indifferent to a size of the test space of the models; selecting models from the corpus according to the computed similarity measurements; determining elements from the selected models to be suggested for modifying the partial description; and outputting the suggestions to modify the partial description useful for completing the combinatorial model.

Abstract: A method, computer product and computerized system, the method comprising: receiving computer code to be protected, wherein the computer code comprising of code blocks arranged consecutively; modifying the computer code, whereby a modified computer code is created, wherein said modifying comprises: introducing a padding area inbetween every two code blocks, wherein each padding area comprises one or more computer instructions; and storing the modified computer code in a computer readable medium.