Abstract: To make validity of a prediction model created by machine learning be able to be verified with appropriate accuracy and efficiency. A software test apparatus includes a storage device configured to store a prediction model, and an arithmetic device. The arithmetic device is configured to accept inputs of a precondition, a constraint condition, and an approximation threshold value, convert the prediction model into a logical expression, analyze an approximation range based on the approximation threshold value with respect to the logical expression to simplify the logical expression, generate an inspection expression by combining the simplified logical expression with the precondition and negation of the constraint condition, search for, as a counterexample, a value satisfying the inspection expression, input the value to the prediction model to evaluate inspection accuracy when the counterexample exists, and output a result of the evaluation.

Abstract: To make validity of a prediction model created by machine learning be able to be verified with appropriate accuracy and efficiency. A software test apparatus includes a storage device configured to store a prediction model, and an arithmetic device. The arithmetic device is configured to accept inputs of a precondition, a constraint condition, and an approximation threshold value, convert the prediction model into a logical expression, analyze an approximation range based on the approximation threshold value with respect to the logical expression to simplify the logical expression, generate an inspection expression by combining the simplified logical expression with the precondition and negation of the constraint condition, search for, as a counterexample, a value satisfying the inspection expression, input the value to the prediction model to evaluate inspection accuracy when the counterexample exists, and output a result of the evaluation.

Abstract: A learning support device stores, as execution information, a set of an input value, an execution output value as an execution result and an expected output value, for a machine learning program. An arithmetic device generates an output value different from the expected output value as a teacher output value in accordance with a predetermined rule when the execution output value and the expected output value for a predetermined input value match each other, generates a loss function based on a difference between the teacher output value and the execution output value, generates a change value indicating a change in the loss function for each parameter in the neural network, calculates an influence degree of predetermined learning by calculating influence of an update value of the parameter obtained by the learning on the change value, and determines whether or not the update value is adopted based on the influence degree.

Abstract: A validity of a prediction model can be evaluated comprehensively. A machine learning program verification apparatus 100 includes a calculation device 104. The calculation device 104 obtains a decision tree logical expression by logically combining path logical expressions indicating decision tree paths indecision trees for a program created by machine learning, creates a combined logical expression by logically combining a verification property logical expression and an objective variable calculation logical expression with the decision tree logical expression, performs satisfiability determination by inputting the combined logical expression to a satisfiability determiner, and when a result of the determination indicates satisfaction, obtains, from a satisfaction solution of the satisfiability determination, a violation input value that is a value of an explanatory variable that violates a verification property and a violation output value that is a value of an objective variable.

Abstract: To efficiently verify and improve a robustness of a learning model for supervised machine learning. A data-creation assistance apparatus 100 includes: a storage device 101 configured to store a neural network model 110 and test data 120; and a computing device 104 configured to specify an uncertainty of an inference result acquired by the neural network model 110; acquire gradient information of the test data 120 by a back propagation process using the uncertainty as a loss; apply various minute changes to the test data 120 to generate a plurality of minutely changed test data, and calculate deviations between each of the plurality of pieces minutely changed test data and the test data 120; and specify, based on the uncertainty information, the gradient information, and the deviations, a minute change that increases or decreases the uncertainty.

Abstract: A learning dataset generation support device 100 is configured to include: a storage device 101 that is configured to store a plurality of pieces of learning data used for supervised machine learning along with correct answer labels; and a computing device 104 that is configured to perform a process of sequentially acquiring the pieces of learning data from the storage device to extract feature vectors, an editing process of adding and/or deleting a feature vector according to a predetermined algorithm, and a process of generating learning data from the edited feature vectors.

Abstract: The boundary search test support device includes: a storage device that holds a plurality of input vectors; and an arithmetic device that executes a test by sequentially inputting the input vectors to a program generated by a neural network and acquiring output vectors which are test results, respectively generates, in a coordinate system which takes each of a predetermined plurality of elements among elements constituting the output vectors as a coordinate axis, a straight line in which the plurality of elements has a same value and a hyperplane in which a sum of values of the plurality of elements is taken as a predetermined value, and arranges a most antagonistic point and boundary vectors whose values of the elements rank higher than or equal to a predetermined ranking among the output vectors in the coordinate system, and outputs the coordinate system together with input vectors corresponding to the boundary vectors.

Abstract: Software test equipment including a data conversion unit receives test input data to convert the test input data into software input data to be provided to software and model input data, a software execution unit receives the software input data, executes the test target software based on the software input data, and outputs an execution result, a model execution unit receives a reference model for the software to generate a model allowable output value range of the execution result obtained by executing the software, based on the model input data and the reference model, a difference analysis unit generates difference information based on the execution result output by the software execution unit and the model allowable output value range generated by the model execution unit, and an evaluation unit receives evaluation criteria and evaluates the behavior of the software based on the difference information and the evaluation criteria.

Abstract: A coverage test support device includes a memory device that stores a test case and specification content of each of a plurality of coverage indexes, and an arithmetic device that sequentially gives a test input value of each pair in the test case to a program created by a neural network, executes a predetermined number of tests, and acquires a test result of the tests and neuron information at the time of test execution, applies the acquired neuron information to the specification content of each coverage indexes and calculates a value for each coverage index, and identifies, among the coverage indexes, a coverage index in which an elongation rate of the calculated value shows a predetermined tendency, as a preferential coverage index that is to be used preferentially, when either the number of executions of the tests or the number of bugs in the test result exceeds a predetermined standard.

Abstract: A validity of a prediction model can be evaluated comprehensively. A machine learning program verification apparatus 100 includes a calculation device 104. The calculation device 104 obtains a decision tree logical expression by logically combining path logical expressions indicating decision tree paths indecision trees for a program created by machine learning, creates a combined logical expression by logically combining a verification property logical expression and an objective variable calculation logical expression with the decision tree logical expression, performs satisfiability determination by inputting the combined logical expression to a satisfiability determiner, and when a result of the determination indicates satisfaction, obtains, from a satisfaction solution of the satisfiability determination, a violation input value that is a value of an explanatory variable that violates a verification property and a violation output value that is a value of an objective variable.

Abstract: A learning support device stores, as execution information, a set of an input value, an execution output value as an execution result and an expected output value, for a machine learning program. An arithmetic device generates an output value different from the expected output value as a teacher output value in accordance with a predetermined rule when the execution output value and the expected output value for a predetermined input value match each other, generates a loss function based on a difference between the teacher output value and the execution output value, generates a change value indicating a change in the loss function for each parameter in the neural network, calculates an influence degree of predetermined learning by calculating influence of an update value of the parameter obtained by the learning on the change value, and determines whether or not the update value is adopted based on the influence degree.

Abstract: The boundary search test support device includes: a storage device that holds a plurality of input vectors; and an arithmetic device that executes a test by sequentially inputting the input vectors to a program generated by a neural network and acquiring output vectors which are test results, respectively generates, in a coordinate system which takes each of a predetermined plurality of elements among elements constituting the output vectors as a coordinate axis, a straight line in which the plurality of elements has a same value and a hyperplane in which a sum of values of the plurality of elements is taken as a predetermined value, and arranges a most antagonistic point and boundary vectors whose values of the elements rank higher than or equal to a predetermined ranking among the output vectors in the coordinate system, and outputs the coordinate system together with input vectors corresponding to the boundary vectors.

Abstract: A coverage test support device includes a memory device that stores a test case and specification content of each of a plurality of coverage indexes, and an arithmetic device that sequentially gives a test input value of each pair in the test case to a program created by a neural network, executes a predetermined number of tests, and acquires a test result of the tests and neuron information at the time of test execution, applies the acquired neuron information to the specification content of each coverage indexes and calculates a value for each coverage index, and identifies, among the coverage indexes, a coverage index in which an elongation rate of the calculated value shows a predetermined tendency, as a preferential coverage index that is to be used preferentially, when either the number of executions of the tests or the number of bugs in the test result exceeds a predetermined standard.

Abstract: Software test equipment including a data conversion unit receives test input data to convert the test input data into software input data to be provided to software and model input data, a software execution unit receives the software input data, executes the test target software based on the software input data, and outputs an execution result, a model execution unit receives a reference model for the software to generate a model allowable output value range of the execution result obtained by executing the software, based on the model input data and the reference model, a difference analysis unit generates difference information based on the execution result output by the software execution unit and the model allowable output value range generated by the model execution unit, and an evaluation unit receives evaluation criteria and evaluates the behavior of the software based on the difference information and the evaluation criteria.

Abstract: In transformation from a symbol string to a term, transformation rules received describe structures of input symbol strings in the form of a context-free grammar, and include structures of output terms as arguments of terminal symbols and non-terminal symbols. An inputted symbol string is analyzed by reduction processing based on the structures of input symbol strings described in the transformation rules, and an intermediate tree is formed. A term for output is produced in accordance with the structures of output terms shown in the arguments of the terminal symbols and the non-terminal symbols corresponding to the structure of the inputted symbol string. Transformation of structured data is performed in like manner using transformation rules which describe structures of input data in terms of relations between classes of partial structures, and includes structures of output data as arguments of class identifiers.