Verification of Signal Processing Using Intelligent Points

Abstract

A technique for debugging of a signal processing program or model (or the debugging and reconciliation of two independent software instantiations). Timelines (Vectors) of signal values for verification are condensed to a small amount of information, easy to understand and manage. The information is stored in “Intelligent Points”, which can be graphically specified by the test engineering. The Intelligent Points can include one or more attributes that describe how to generate the appropriate signals. The information so stored can then be automatically expanded to complete timelines.

Description

RELATED APPLICATION(S)

This application claims the benefit of U.S. Provisional Application No. 61/416,152, filed on Nov. 22, 2010, U.S. Provisional Application No. 61/416,153, filed on Nov. 22, 2010, U.S. Provisional Application No. 61/458,406, filed on Nov. 23, 2010, U.S. Provisional Application No. 61/458,410, filed on Nov. 23, 2010 and U.S. Provisional Application No. 61/481,973, filed on May 3, 2011. The entire contents of the referenced applications are hereby incorporated by reference.

TECHNICAL FIELD

This application relates to verification of signal processing software, and in particular to using graphical Intelligent Points to produce verification timelines.

PROBLEMS WITH THE EXISTING ART

Several categories of software require very careful verification. In some cases the verification is mandated by government agencies, with strict requirements and procedures. This is typical, but not limited, to aircraft flight code and critical medical equipment code. For instance, the Federal Aviation Administration requires that software used in aeronautical applications be tested according with DO 178B standards.

Software that is critical to flight operations (Class A) is tested with extreme care. Usually this software belongs to a special category of coding, which can be generally described as signal processing. In other words, the software receives input signals from a variety of real-world sources, such as a pilot adjusting a control yoke, rudder pedals, and/or throttle controls, altimeter, attitude indicator, airspeed indicator, magnetic compass, or other flight instrumentation, analog to digital converters (A/D), etc., and continuously processes such signals at a steady rate, fixed frames or whenever necessary. The software produces output signals in the form of commands or warnings, via Digital to Analog converters (D/A).

The user, by himself or herself or possibly jointly with the manufacturer of the is actual hardware, often referred to as a Black Box, that runs the software prepares a set of clearly defined and understandable specifications and requirements, which define the processes inside the Black Box. In the verification process, a set of test cases and test procedures are independently prepared in order to verify that the Black Box performs properly. That is, the Black Box does everything it is required to do, and only what it is required to do.

In the past, a typical approach to verification has been to list the requirements to an “atomic” level, the smallest possible testable unit, such as an AND or OR logic gate, write test cases that apply combinations of Boolean values to the input of the element under test, and state what results should be expected. Such test cases are translated into test procedures. The test procedures are then run to collect actual results, and compare the actual results with the expected results. A common methodology for such procedures is to force values into the code, referred to as stabbing, and extract output values from some intermediate place in the code process.

Regardless of the methodology or completeness of testing, the method discussed above only proves that the Black Box has the correct collection of elements, not that the elements are correctly connected. In other words, the verification process discussed above does not prove that the Black Box works properly end-to-end. It, therefore, does not prove that the Black Box, as a whole, will behave as intended.

Another shortcoming of this verification testing approach occurs everywhere that timing is involved, such as the verification that a timer expires as required. A simple verification at the desired expiration time does not preclude the Black Box from generating unexpected values at any or all of the intermediate, unverified times. Other issues of partial verification occur in the verification process of certain components or groups of components. These partial verification problems, which can be potentially very dangerous, can include: overlapping of data structures stored in memory but cannot be accessed; or “memory leaks”; incorrect connections between components, known as “stitching”; incorrect order of execution, such as loop-backs. None of these types of faults, and potentially others, would be detected.

Currently, verification has started to move from test cases that test component-by-component, or function-by-function to Black Box level verification. This level includes verification by test procedures that enter “real world” inputs into the Black Box which then generates real world outputs. The inputs are generated for each “run” of the program, or for each “frame”, to exactly mimicking the real application. In other words, the inputs are presented to the Black Box during the verification process exactly as they are presented in reality. Expected outputs are then produced for each run or frame, which are then compared to the collected actual outputs. This “continuous” method of verification is far superior to any other method, provided that the test procedures are able to exercise and verify each component and requirement of the Black Box.

However, the generation of such test procedures at the Black Box level is difficult for several reasons. Mainly, the variables used for inputs to a component, or block, may not be the same as the Black Box level real world inputs, and the outputs from the component, or block, may not be real world outputs from the Black Box. Therefore, the inputs to the element under test must be “demoted” to real world inputs, and the outputs from the element under test must be “promoted” to real world outputs. In other words, the verification process requires the test engineer to generate one test procedure for each targeted component, diagram, or block of components. The test procedure must be of a duration that addresses all timing issues, achieves the desired inputs using a combination of real world input values, and also generates real world outputs that are representative, in a detectable, unique way, of the targeted component outputs. The test procedure is complete when all the aspects of the targeted component are tested under the specified guidelines, such as modified condition/decision coverage, exhaustive, etc.

In the above environment, one way to generate independent expected outputs is to provide and run an alternative code in the Black Box. This is often referred as a “model,” but it could be just an alternative instantiation of the code, such as code independently produced by another team. Of course, it is desirable that the expected results are produced using a different compiler, a different hardware platform, and a different operating system than the Black Box.

Usually, one of the first problems encountered is reconcile differences between the expected results and the actual results. In a complex system, with thousand of components, this is a difficult task. A currently available method of performing reconciliation use the debugging features of two compilers in an integrated design environment (IDE), for example, compiling the model code side-by-side with the Black Box code. This method allows only “forward” debugging, meaning that both compilers must be stopped at the same point in the respective codes so that the user can look at values held by variables and data structures. The user then move to the next break point and again looks at values held by variables and data structures, and so on. The reconciliation process is difficult and may take very long time. Reconciliation is particularly long and difficult if the model is written using visual techniques, such as Simulink®, and/or if the input data to the Black Box cannot be provided faster than real time—it is normally provided much slower than real time. (Simulink® is a registered trademark of MathWorks, Inc., of 3 Apple Hill Drive, Natick, Mass. 01760.) The user may wait 10 minutes between break points, the point at which he may be able to investigate mismatching values, just to learn that the process must be started anew.

SUMMARY OF AN EMBODIMENT

In a preferred example embodiment, signal timelines (vectors) for verification of signal processing software and/or model code are condensed to provide a small amount of information that is easy to understand and manage. The information is stored as “Intelligent Points,” and contain attributes that describe how to generate appropriate signals. The condensed information can then be automatically expanded to complete timelines. The Intelligent Points can be specified by the test engineer graphically, or in other ways.

An environment in which it is useful, but not necessary, to apply an example embodied technique is where a layer of code is wrapped around the executable signal processing code, so that at each execution frame, the values assumed by each variable at each frame are saved. Such an environment allows the test engineer to apply changes to the Intelligent Points, select a point or points in time, for example, by moving the mouse on a timeline by using a slider, and inspect the values assumed by one or more selected variables in response to the changes.

More particularly, in an example signal processing verification environment, a test engineer must verify test code against the operations of a Black Box. One way to generate independent expected outputs is to provide an alternative code for the code running in the Black Box. This alternative code is often referred as a model, but it can simply be an alternative instantiation of the code, independently produced by another team. It is useful for the expected results to be produced using a different compiler, a different hardware platform, and a different operating system than those used by the Black Box.

In the verification environment, a complete and correct set of Test Procedures must be generated according to some agreed-upon testing standards. Generation is typically achieved by forcing values for the Real World input variables that correctly exercise, and thus verify, the components targeted by the test procedure. The number of test procedures used for verification purposes can be one or many. One, or a few, test procedures may be complex and long in duration, while many others may be simple and short in duration. In either scenario, generation of a large number of files is difficult to manage. Regardless of the number of test procedures, a preferred example embodiment easily optimizes the information of a set of test procedures required to verify a set of requirements, and, therefore, the file size, and ease of understanding.

The following scenario is offered only as a descriptive example and it should be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention. Consider a Black Box with a multitude of input, intermediate, and output variables running at fixed intervals of 10 milliseconds. Intermediate variables are those within the box that are not accessible from outside. That is the Black Box performs 100 runs per second, where each run processes inputs and produces outputs. Assuming a set of standard requirements for timer verification, and a time constant of 10 seconds, the complete test procedure has a duration of approximately 40 seconds, and, therefore, needs to produce 4,000 steps (100×40 steps) or input variables.

An example embodiment of the present disclosure obviates the need to create values for all the input variables, 4,000 input values in the above example. Instead, what input variable values are necessary to toggle the timer, and at what time they need to be toggled are determined. The determined information, which can include a time and value for each variable that changes to generate the desired input to the element under test is only a few bytes. While the choice of such “points” is not trivial, and may require complex analysis, if done correctly it can perform complete verification akin to creating all the input steps. The choice of points can be computer-aided.

By adding attributes to a selected point of change for input variable values, an “Intelligent Point” is created. The additional attributes can instruct the system on how is to handle input variable values between two selected points, which may include instructing the system to linearly interpolate from two points, or hold the value between two points, or inject a frequency via a periodic waveform such as, a sine wave, square wave, triangular wave, or a variable frequency with varying frequency and amplitude parameters, and/or Boolean variables for an exhaustive logic test, to fulfill any signal processing requirement. The resulting information needed to describe an Intelligent

Point is may include value, time, point type, and a few parameters for the type as necessary, is very small as compared to information needed to specify the signal at every point along a timeline.

As such, an example embodied method for storing a complex Test Procedure that stores a list of the Real World input variables, which require modifications in order to achieve the verification of the target component, and the Intelligent Points associated with them. Even if the list of variables includes information on how to display such variables, the condensed Test Procedure so is still small and much more understandable to a user or test engineer than the complete timeline offered for verification.

A test engineer may use any number of techniques, including graphical user interfaces, to select the appropriate variables and points for specific component verification.

An example embodied method of process verification can proceed as follows:

1) Select appropriate variables and appropriate points (and point attributes) to verify the target component(s).

2) According to the intelligent points, automatically generate a complete timeline for the selected variables, and, eventually, all of the input variables that require explicit values, i.e., those that are not nominal.

3) Evaluate expected results for the output variables that unequivocally represent the outputs of the targeted components, using the independent instantiation of the software or model.

4) Feed the generated timeline into the “Black Box” and collect actual results.

5) Compare the expected results and actual results for each variable at each step (10 milliseconds, in the above example).

Once the meaningful input and output variables are selected and the Intelligent Points are appropriately set, the resulting small amount of information needed to specify the combination can be stored in a simple, easy to understand file. The rest of the verification process can easily and completely be automated, up to the determination of which results match, and therefore pass, and which results do not match, or within a threshold tolerance, and therefore fail.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

The foregoing will be apparent from the following more particular description of example embodiments, as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the embodiments.

FIG. 1 is a high level operations flow diagram.

FIG. 2 illustrates a main window of the user interface.

FIGS. 3-8 illustrate several example diagrams of signal processing components is of a Black Box used as part of a Demo project.

FIGS. 9-14 illustrate various ways to verify certain diagrams.

FIG. 15 illustrates how to set IntelliPoints.

FIG. 16 is another verification view.

FIG. 17 illustrates verification of a timer.

FIG. 18 is a state diagram for timer verification.

FIGS. 19-21 are other timer verification diagrams.

DETAILED DESCRIPTION OF EMBODIMENTS

A description of example embodiments follows.

High Level Description of the Model-Based Verification Application

FIG. 1 is a high level operations flow diagram of an example embodiment of a Model-Based Verification Application 100.

The Model-Based Verification Application 100 is made of several components.

Model Core: The Model Core 105 provides many of the features of the Model-Based Verification Application 100, such as generating input signal timelines using Intelligent Points, all graphical rendering and displaying of signals and corresponding signal processing components or blocks of connected components (also referred to hereafter as the “diagrams”), debugging capabilities, file name and file input/output (I/O) services.

UNITSCD: The UNITSCD (Specification Control Drawings) 110 is a code unit where all the information pertaining to the diagrams is stored. The information can include the devices, ASCII names of variables, the positions of flags, and constants on the diagram visual representations.

VARSANDBODIES: The VARSANDBODIES 115 enables the functionality of each diagram. It also enables the functionality of the complete Black Box under test by connecting together or “stitching” two or more of the diagrams that make up the Black Box. Typically, the VARSANDBODIES 115 is made up of calls to a library of components with standard behavior, or in some cases custom behavior. The library components can include, but are not limited to, Boolean logic gates, such as, OR gates, AND gates, and NOT gates, and other signal processing functions, such as delay timers, and various filters. The VARSANDBODIES 115 also provides all the variables of the Black Box in static memory.

This structure allows this same Model Core 105 to be re-used in different projects. New custom components can be added to the library of components as necessary. Using minimal additional resources, the custom components can be carried on as legacy elements in projects that do not use them.

The UNITSCD 110 and VARSANDBODIES 115 code units are typically new and unique for each project. Thus, they typically are developed for a specific project, and cannot be re-used in a different project. These code units can be considered to belong to a customer of the user. The VARSANDBODIES 115 needs to be compiled for performance, while the UNITSCD 110 is a database-like set of information.

Other (External) Model Components:

ALIAS Table: Alias Table 120 is a computer file, such as an Excel® file that contains information regarding names of variables used in the actual code, i.e., the code under test. (Excel is a registered trademark of Microsoft Corporation, One Microsoft Way, Redmond, Wash. 98052.) Such information can further include minimum and/or maximum limitations for values, granularity or decimation/digitalization properties, diagram stitching information, and other names in use. Information regarding other variable names in use is especially useful when there is redundancy, and/or inconsistency in the signal naming scheme among and between model code and actual code.

It should be understood by those of skill in the art that the mention of Excel® files throughout this document is due only to the convenience of using the Excel® format as a presentation layer, and not for any other features of Excel®. All specific file formats such as Excel, Visio, Simulink, Subversion, etc. referred to in this document can be in other formats, such as Comma Separated Values (.csv) or any other suitable form.

Diagram Bitmaps: Diagram Bitmaps 125 are a visual representation of the diagrams. The Diagram Bitmaps 125 can be a screen shot, or scan, of a diagram. While visual representations can be generated from the file that stores the diagram in native format, such as Visio®, Simulink®, or similar tool, it is preferred to use a bitmap file, because there are no possible errors in the interpretation of the native-mode files. In other words, what you see is what you get with bitmap files. (Visio is a registered trademark of Assistance Technique Et Etude De Materiels Electroniques Societe Anonyme, France, Burospace Bâtiment 26 Route de Gizy; F-91570 Bievres, France; and Simulink is a registered trademark of MathWorks Inc., 3 Apple Hill Drive, Natick, Mass. 01760.) This preferred visual representation allows for an excellent independent interpretation and analysis of the diagram.

CMOD Files: CMOD Files 130 are computer files, such as Excel® files, that contain visual information, and environment or harness parameters about a specific test. The CMOD Files 130 contain only the “Intelligent Points” that allow the creation of complete test files in conjunction with the Model 101a/b.

Test Cases: The Test Cases 135 are a collection of computer files, such as Excel® files, which are related to a single diagram. In an example embodiment the columns represent all the input variables and output variables of the related diagram, and each row represents the inputs, and expected outputs, for each and every processing step. In other words, the Test Case 135 files contain all of the I/O variables, input values, and expected output values for each step of time along a timeline.

Test Procedures: Test Procedures 140 are a collection of files, such as Excel® files, which have columns representing Real World input values and output values. The input values contain all the initial condition values for variables. That is, Test Procedure 140 files contain all the Real World values that set the stage for the test, including the values for all the variables that have non-nominal values—even if they do not change during the test. The Real World output variables represented are a set of manually selected variables pertinent to a specific test. Each row represents input values and expected output values, for every time step. Each Test Procedure 140 is a companion to a Test Case 135, in the sense that the Test Procedure 140 will faithfully generate the Test Case 135 data for the related diagram, when the Real World input values traverse the diagram. In other words, when the Real World input values are completely processed according to a Test Procedure 140, the Test Case 135 data will be generated for the associated diagram.

Hardware (Rig): The Hardware (or Rig) 145 is the actual physical equipment against which the Model 101 is verified. Test Procedures 140 are submitted to the Hardware 145 to produce Actual Results, which are then fed back to the Model 101 and/or stored in record-keeping or version control systems 155, such as Subversion® or ClearCase®. (Subversion is a registered trademark of CollabNet, Inc., 8000 Marina Blvd., Suite 600, Brisbane, Calif. 94005; and ClearCase is a registered trademark of Atria Software, Inc., 24 Prime Park Way, Natick, Mass. 01760.)

Intelligent Points Setup and Manual Review Process:

The purpose of the Model-Based Verification Application 100 is to ascertain that each block in the Back Box, such as one or more selected components or diagrams, produces the intended functionality. The verification process is a complex function that depends on device type, signal type, required verification procedures, and several other parameters. Verification must sometimes also comply with applicable standards 170, such as DO178 as applied to flight code, and perhaps other custom requirements.

In a preferred example embodiment, each device can be verified by specifying a combination of Intelligent Points 175 (also referred to herein as “IntelliPoints”). An Intelligent Point is a command for a variable to assume a value, or a signal type, from one point in time until the point in time of the next Intelligent Point 175. In other words, an Intelligent Point 175 dictates the signal type for an input variable, and therefore input values, for a period of processing time along a time line. The simplest form of input variable signal is a linear interpolation between the two points, but other signal can be created in accordance with the selected attributes of the Intelligent Point 175 .

While the number of Intelligent Points 175 in a particular Model 101 may vary depending on many considedrations, normally a very small set of Intelligent Points 175 is sufficient for a particular verification. For example, a method for verification of a DelayOn device requires only four Intelligent Points 175 to generate the Boolean input values necessary to verify Persistency and Expiration tests. However, the Intelligent Points 175 must be carefully and appropriately set in time. For other variables more points may be required to promote the output signal to a visible Real World signal, using a promotion process.

The selecting or setting of Intelligent Points 175 can be achieved via a trial-and-error manual process, aided by the Model's 101 recalculation of all signals when a point is moved. Useful to selecting of Intelligent Points 175 is a real-time recalculation, which allows the user to graphically see the implications on all signals when an Intelligent Point 175 is modified in time, value or signal type, or another Intelligent Point 175 is added.

The combination of Intelligent Points 175 and real-time recalculation with graphical-aided feedback enables the Model-Based Verification Application 100 to verify the Black Box components. Without this combination, the Black Box verification of a complex system would exceed reasonable human resources.

User Interface:

FIG. 2 illustrates a user interface 200 for specifying various parts of the Model 101. The main window of the user interface 200 is organized in tabs 205. All options are preferably immediately visible with no hidden drop-down menus. All folder names 210 are always visible, and can be typed or browsed. Subsets of file names 215 can be selected by using a standard Windows mask. All parameters are saved in a local file for each PC, or user. All visible lists of files and produced tables can be saved in Excel® format for re-use.

Virtual Input Variable

The stitched Model 101 consumes all the Real World input variables and produces all the Real World output variables. However, the list of all the input variables can be impractical to maintain in some projects, therefore the Model 101 resorts to Intelligent Points 175 for Virtual Real World input variables. Here, for Virtual Real World input variables, it is intended as a variable that does not exist as a real world input variable, that may have a similar name to a Real World input variable, and has the purpose of setting a number of Real World input variables to the desired value or values, according to rules.

For example, the setting of a variable carried by a triple-redundant ARINC bus typically requires setting the value of 12 input variables; 3 values for each of the following: Data, Activity, Parity, and Sign/Status Matrix (SSM). A Virtual Real World input variable would set all 12 variables so that the internal variable selected after the redundancy process assumes the same value as the Virtual Real World input variable.

In another example, an input variable can assume a value in degrees inside the Black Box, or even inside a particular diagram, but has a real world input variable value in radians, or Sines and Cosines, the Virtual Real World input variable is set in degrees, and the Model 101 adjusts the input values to Sines and Cosines accordingly.

The Virtual Variable can be set to any value, but the Model 101 limits the Real World input values according to the specified minimum and maximum values in the Alias table 120 (shown in FIG. 1) and only produces values that match the actual granularity of the input values, i.e., the digitized or decimated values, so that the variables internal to the Black Box can only assume values that can be carried through the physical hardware. This is useful so that model-computed expected results can exactly match Black-Box computed actual results.

Ranges for Variables

In the real world, physical hardware, variable values are carried to the Black Box model via communications channels or buses. Usually, a bus or channel has a width in bits.

For example, if the variable is generated by a 10-bit analog-to-digital converter (A/D), the bus may also have a 10-bit width. Assuming, as an example, a 10-bit A/D is connected to a sensor that measures Vertical Speed Accelerations (VSA) in gravitational-force (Gs). Further assuming that the aircraft in which the VSA is installed is designed to operate between −3 and +7 Gs, and that the device will disintegrate if the acceleration exceeds the range between −6 and +14 Gs. The sensor is capable of providing data values between -20.48 and +20.44 Gs.

The example assumed will be able to accomodate 1024 discrete values. Further assuming that one bit is used for the sign, and that −20.48 G would correspond to −512 and +20.44 G would correspond to 511. It is useful to introduce a scale to convert from the raw input data signal to a more meaningful, and floating point, value. In this case, the scale would be 1/25. (Range of values÷Number of discrete values for the range=(20.44−(−20.84))/(1024−1)=1/25.) This would also implicitly dictate that the signal is received in minimum increments (granules, decimation, or digitization units) of 0.04 Gs.

In the above example, the Alias table 120 would carry the information about is range and granularity for this variable. The Model 101 would use the Alias table 120 to create input values only in accordance with the range and granularity. Although the variable input can assume values from −20.48 to +20.44 Gs, in 0.04 increments, the values outside of the range between −6 and +14 Gs are meaningless, because, in the real world the VSA sensor would be destroyed. It would useful that a project have a limiting layer of code that ensures that all input variable values are meaningful operational values.

Demo Project

In the following discussion of several drawings, it is assumed that the specifications are for a Demo project.

FIG. 3 is an example of an analog limiting and combining diagram 301a, for which the input variable ranges are specified, contained within the Black Box. The tabs 370 indicate different diagrams, or sets of interconnected signal processing components, contained within the Black Box. For example, VAR_S_A 302 can range from a MAX_SA 303 value of +21.0 down to a MIN_SA 305 of −8.0. Similarly, VAR_S_B 312 ranges from a MAX_SB 313 value of +7.0 down to a MIN_SB 315 of −7.0 and VAR_S_C ranges from a MAX_SC 323 value of +10.0 down to a MIN_SC 325 of −10.0. The limiters 350 produce the Real World output variables, VAR_S_A_LIM 307, VAR_S_B_LIM 317, and VAR_S_C_LIM 327. The combiner 360 use Real World output variables VAR_S_A_LIM 307, VAR_S_B_LIM 317, and VAR_S_C_LIM 327 to produce VAR_S_S_AB 319 and VAR_S_AC 329. The input variable values ranges, MAX_SA 303, MIN_SA 305, MAX_SB 313, MIN_SB 315, MAX_SC 323, and MIN_SC 325 are shown in green. Window pane 301b illustrates a listing of the input variables and output variables and the attributes corresponding to each, such as input/internal/output (I/N/O), visibility, decimals, color, line width (LW), scale/bias, actual value, and variable name, for the diagram 301a.

FIG. 4 illustrates another diagram 401 a inside the Black Box containing different types of components, such as DelayOn 405, Boolean gates 406, 407, and a decision making comparator 408. The tabs 470 indicate different diagrams, or sets of interconnected signal processing components, contained within the Black Box. The flag-shaped rectangles, with a numeric or Boolean value inside, or flags 402, are set up by the Model 101 to show the value of variables and other components at a given time, and are not part of the specifications or diagrams. The components DelayOn 405, Boolean gates 406, 407, and decision making comparator 408, operate in accordance with their respective function and connections, or stitching, on the values associated with VAR_B_Z 411, VAR_B_Y 412, VAR_B_X 413, VAR_B_W 419, VAR_S_U 415, and VAR_S_T 416 to produce output variables VAR_B_ZOR 421, VAR_B_WXY 422, and VAR_B_TU 423. Window pane 401b illustrates a listing of the input variables and output variables and the attributes corresponding to each, such as I/N/O, visibility, decimals, color, LW, scale/bias, actual value, and variable name, for the diagram 401a.

FIG.5 illustrates yet another example diagram 501a containing a Differentiator (S) 551, Integrator (1/S) 552, Limited Integrator 553 and a couple of First-Order filter components 554. The tabs 570 indicate different diagrams, or sets of interconnected signal processing components, contained within the Black Box. The flags 502, are set up by the Model 101 to show the value of variables and other components at a given time, and are not part of the specifications or diagrams. The components Differentiator (S) 551, Integrator (1/S) 552, Limited Integrator 553 and couple of First-Order filter components 554 operate in accordance with their respective functions and connections on the values associated with input variables VAR_S_A_LIM 307, VAR_S_B_LIM 317, and VAR_S_C_LIM 327 to produce output variables VAR_S_A_DIFF 521, VAR_S_B_INTEG 522, VAR_S_B_INTLIM 523, VAR_S_C FILT 524 and VAR_S_C_FOFILT 525. Window pane 501b illustrates a listing of the input variables and output variables and the attributes corresponding to each, such as I/N/O, visibility, decimals, color, LW, scale/bias, actual value, and variable name, for the diagram 501a.

FIG. 6 exemplifies yet another example diagram 601 a that has four different Second-Order filters 650. The tabs 670 indicate different diagrams, or sets of interconnected signal processing components, contained within the Black Box. The six constants 653, the numbers shown in green, define the response characteristics of each Second-Order filter 450. The parameters of the Second-Order filters 450 are shown in the S-domain, but Z-domain parameters can also be used. The flags 602, are set up by the Model 101 to show the value of variables and other components at a given time, and are not part of the specifications or diagrams. The Second-Order filter components 650 operate in accordance with their respective functions, including their respective response characteristics, and their connections on the values associated with input variable VAR_S_C_LIM 327 to produce output variables VAR_S_C_HPFILT 621, VAR_S_C_BPFILT 622, VAR_S_C_BSFILT 623, and VAR_S_C_SOFILT 624. Window pane 601b illustrates a listing of the input variables and output variables and the attributes corresponding to each, such as I/N/O, visibility, decimals, color, LW, scale/bias, actual value, and variable name, for the diagram 601a.

s FIG. 7 shows another example diagram 701a a special type of Second-Order filter, a Notch Filter 750. The tabs 770 indicate different diagrams, or sets of interconnected signal processing components, contained within the Black Box. The six constants 753, the numbers shown in green, define the response characteristics of the Notch filter 750. The parameters of the Notch filter 750 are shown in the S-domain, but Z-domain parameters can also be used. The Notch filter component 750 operates in accordance with its respective function, that is filtering according to its response characteristics, and connections, on the values associated with input variable VAR_S_C_LIM 327 to produce output variables VAR_S_C_NOTCH 721. Window pane 701b illustrates a listing of the input variables and output variables and the is attributes corresponding to each, such as I/N/O, visibility, decimals, color, LW, scale/bias, actual value, and variable name, for the diagram 701a.

FIG. 8 illustrates a diagram 801a in which analog and Boolean (digital) variables are processed for presentation to a Real World output layer. The purpose of this diagram is to exercise the stitching between diagrams and components and the promotion process of moving virtual variables to the Real World. The tabs 870 indicate different diagrams, or sets of interconnected signal processing components, contained within the Black Box. The flags 802, are set up by the Model 101 to show the value of variables and other components at a given time, and are not part of the specifications or diagrams. The components, switches 850 and mixer 851, operate in accordance with their respective functions and connections on the values associated with input variables VAR_S_A_DIFF 521, VAR_B_ZOR 421, VAR_S_S_B_INTEG 522, VAR_S_B_INTLIM 523, VAR_S_C_FILT 524, VAR_B_WXY 422, VAR_S_C_FILT 524 VAR_S_C_FOFILT 525 and VAR_B_TU 423 to produce output variables VAR_S_AB_OUT 821, VAR_S_BC_OUT 822, and VAR_S_C_OUT 823. Window pane 801b illustrates a listing of the input variables and output variables and the attributes corresponding to each, such as I/N/O, visibility, decimals, color, LW, scale/bias, actual value, and variable name, for the diagram 801a.

FIG. 9 is an example of a timeline plot of signals used for verification of the diagram with limiters 301a (shown in FIG. 3). The canvas 901a depicts a 30 second timeline of signals 910, 920, and 930. The window pane 901b on the left contains a list of all variable inputs (here, all Virtual) and all variables output to the Real World. The three waveforms of signals 910, 920, and 930 each have an amplitude that exceeds the limits of the values for the input variables corresponding to the respective limiters 350 (shown in FIG. 3). Therefore, the signals 910, 920, and 930 are useful to verify, at least partially, the performance of the limiters 350.

FIG. 10 is another diagram illustrating an alternate example approach to verification of the diagram 301a with limiters (FIG. 3). Canvas 1001a depicts a 30 second timeline of signals 1010, 1020, and 1030. Here, the three waveforms, signals 1010, 1020, and 1030, are obtained suing a few IntelliPoints 1050. A single IntelliPoint 1050 defines each of the periodic sinusoidal waveforms 1010 (colored purple) and 1030 (red) by setting their respective frequencies in Hertz, and amplitudes. The waveform signal 1020 (green) is generated by linear interpolation between the adjacent IntelliPoints 1050 in a set of four IntelliPoints 1050.

FIG. 11 illustrates an example verification of the limiter diagram 301a (FIG. 3) in another way. The values for the input variables and output variables are plotted as a function of time on the canvas 1101a. Here, in window pane 110ab, the user can inspect the variables local to the diagram 301a at the same time as he or she sets the Real World input values. The effect of the limiters 350 is obvious from the chopping of peaks and troughs shown in the output signals 1114, 1115, 1116, 1117, and 1118 generated from the sinusoidal waveform input signals 1111, 1112, and 1113. Dragging an IntelliPoint to reposition it along the canvas has an immediate effect on both windows, allowing the user to gauge the verification effort.

FIG. 12 illustrates an example verification of the limiter diagram 301a (FIG. 3). Canvas 1201a depicts a 30 second timeline of signals 1210, 1220, and 1230. Here, the three waveforms, signals 1210, 1220, and 1230, are obtained suing a few IntelliPoints 1050. A single IntelliPoint 1250 defines each of the periodic sinusoidal waveforms 1210 (colored purple) and 1230 (red) by setting their respective frequencies in Hertz, and amplitudes. The waveform signal 1220 (green) is generated by linear interpolation between the adjacent IntelliPoints 1250 in a set of four IntelliPoints 1050. In FIG. 12 linearly interpolation between each of four IntelliPoints 1250 (green) shows that the signal is sufficient for the verification of one of the limiters 350 (FIG. 3). The other limiters 350 are verified by waveform signals created by setting one single Intelligent Point for each signal, where each generated a sinusoidal signal has amplitude exceeding each respective limiter's threshold.

FIG. 13 depicts another example diagram 1301a, which show the values of all the variables at that time in the flags 1302. The flags 1302, containing variable values at the specified time step, are over-imposed to the bitmap of the diagram 1301a. In a preferred embodiment, example diagram 1300 is displayed at the same time as the verification display 1200 (FIG. 12). Other features of the figure have been previously described.

FIG. 12 in combination with FIG. 13 illustrates a useful feature in canvas 1201a (FIG. 12) is that the crosshair can be moved back and forth in time along the timeline, and the variables values can be immediately shown to the user in the flags 1302 (FIG. 13) without re-computing. Re-computing is unnecessary because all variable values, at all time points within the canvas, have already been computed and saved. Thus, when the user selects a different point on the time line, such as by moving the cross-hair, the variable values at the newly selected time are determined simply by looking them up in an array, the DataSim table, and displayed in the flags. Re-computation occurs only when an IntelliPoint is changed.

Therefore, the model therefore enables the selective visualization of output variables, allowing a user to immediately determine if a Real World output is a viable function, such as a linear transfer function, or the best possible approximation, of the selected diagram output.

FIG. 14 illustrates another example embodiment of a step in a verification method. In order to verify end-to-end operation of a Black Box, it is useful to select a is Real World output variable that unequivocally represents the diagram output variable.

This can be achieved through a process called promotion, in which a signal is promoted towards the output. It is useful to unequivocally represented a signal that is the output variable of a diagram, but is an intermediate variable, i.e., a non-Real World output of the Black Box. This can be accomplished by navigating to diagrams downstream of the diagram under test (i.e., signal processing functions that occur later in the Model 101 process) and selecting suitable variables that represent the original signal as the output. Here, the diagram under test is diagram 301a (shown in FIG. 3), and an intermediate variable is VAR_S_AB_OUT 1410. Canvas 1401a depicts a 30 second timeline of signals 1010, 1020, and 1030. Window pane 1402 displays a data for intermediate variable is VAR_S_AB_OUT 1410, including values and time increments for each processing frame number and the frame number.

FIG. 15 illustrates a preferred graphical user interface (GUI) 1501 for selecting or setting IntelliPoints. The choice of the parameters for the IntelliPoints can be simple and intuitive. The following commands are preferably used in a graphical view of input variable values as a function of time, such as illustrated in FIG. 14: a left-click of a mouse allows a user to drag-and-drop the IntelliPoint to a new location; a center-click deletes the IntelliPoint; and, a right-click opens a window, such as for the GUI 1501, where the user can set a variety of parameters. As shown in GUI 1501, a variety of attributes can be selected to create functions for input variable values, such as normal (i.e., linear interpolation) 1510, frequency 1511, pulses 1512, square wave 1513, saw wave 1514, hold value (i.e., set to a constant) 1515, and frequency increment 1516. Depending upon the function selected, such as frequency increment 1516, for example, certain other IntelliPoint parameters will become available to the user.

An example embodiment of a disclosed method can ultimately result in a CMOD file 130 (FIG. 1). FIG. 16 illustrates the CMOD files contents 1600 for the verification of a relatively complex diagram. While the CMOD file is shown in Excel®, the user never has to manually operate within Excel® program. The CMOD file 130 and 1600 represents a complete Test Procedure.

Verification of a Diagram with Timer on Boolean Logic

FIG. 17 illustrates a typical diagram 1701a for timer verification containing a set of logical rules. It should be understood that such rules can vary depending upon project requirements. In this case, the Boolean logic is normally tested using Modified Condition/Decision Coverage (MCDC). Other aspects and features of the figure have been previously described in discussions regarding FIG. 4, the difference being that in FIG. 17 the variable values of the diagram 1701a are shown for time 7.76 seconds.

FIG. 18 show a state event diagram 1800 for timer logic, such as that illustrated in 1701a (FIG. 17). In this case, according to the State-Event diagram 1800, the requirements mandate the timer, having Boolean input values and output values, to be verified for persistency and expiration. The State-Event diagram 1800 illustrates that only four Intelligent Points, when properly allocated in time along the timeline, satisfy the requirement.

FIG. 19 illustrates an example method of verifying the timer on Boolean logic diagram 1701a (FIG. 17). The setting of four IntelliPoints 1950, between 4 seconds and 14 seconds on the canvas 1901a, is sufficient to verify the timer. Window pane 1901b illustrates a listing of the input variables and output variables and the attributes corresponding to each, such as I/N/O, visibility, decimals, color, LW, scale/bias, actual value, and variable name, for the diagram 1901a.

FIG. 20 illustrates an example verification method of the diagram 1701a (FIG. 17) in another way. The canvas 2001 a in window 2000 shows the variables related to the diagram under test only, such as VAR_BZ 2010, VAR_BY 2020, VAR_BX 2030, VAR_BW 2040, VxB1 2050, VAR_B_ZOR 2060, VAR_B_WXY 2070, and VAR_B_TU 2080.

FIG. 21 illustrates verification of the diagram 700 with Notch Filter (FIG. 7). The notch filter is verified by appropriately selecting a few IntelliPoints. One IntelliPoint sets a variable-frequency waveform 2110, another IntelliPoint sets a square wave (impulse response), and another IntelliPoint sets a frequency 2115 tuned to the expected “notch.” The amplitude of the response characterizes the filter parameters. The Test Case results in an Excel® file with 1500 rows (30 frames at 20 millisecond per frame).

Advantages of an Embodiment

In an example embodiment, the Test Cases generated using appropriately selected IntelliPoint verify all the relevant and necessary variables at all signal processing frames. The Test Procedures compare expected results, those produces by the Model, and actual results, those produced by the Black Box, for all the variables at each and every frame, leaving no uncertainties regarding random, unexpected variable values or wrong stitching between diagrams.

The IntelliPoints are similar to fractals in that IntelliPoints expand to a lengthy and complex Test Case and Test Procedure files. This expansion process is relatively simple and the Test Cases and Test Procedures are automatically generated by the Model. While the Test Case and Test Procedure files are lengthy and complex, at the same time, the IntelliPoints are humanly understandable and settable, with the aid of the backward and forward timeline traversal debugging and visualization features of the Model.

An example embodiment of the method has been proven successful in a project with 200 complex diagrams and 6,000 variables, while still providing reasonably approximate real time feedback when moving IntelliPoints. In the test case, the 500

CMOD files, having an average of 20 IntelliPoints each, produced four giga bytes (4 GB) of Excel® Test Cases and Test Procedures in about 6 hours of batch processing on a medium-powered PC. The complete verification process produces only IntelliPoints that verify the desired specifications for all components. With this method, each Intelligent Point only requires a few minutes of user attention. This is particularly useful to save time in the event that the requirements change; the user needs only to add or modify a few Intelligent Points, after which the system can re-compute all expected results for all diagrams.

The batch process that re-computes output variable values at all times is particularly useful because a change in requirements, or a change in a diagram, can significantly alter the expected results, and is very difficult to predict. Then, the batch process may produce then correct verification, which perhaps is incomplete. The verification of a component is complete only when the component is fully exercised according to specification. However, an incomplete verification is still useful if the expected results match the actual results. The completion of the verification may be achieved in steps, for example, by adding Intelligent Points, to the existing Test Cases and Test Procedures.

The above described forward and reverse timeline traversal debugging technique is more than simply moving a mouse back and forth on a timeline. It is a complex balance of programming resources allocation. It provides the user with the perception that all the variables in all the diagrams are immediately recalculated for any change to is the Intelligent Points and other parameters, and that all the values are immediately available at the time frame where the mouse is positioned on the timeline. This perception of real-time availability is useful to enable the interactive interface allowing the user to easily and quickly set IntelliPoints for verification. Test Case is manually generated, and the generation process is aided by the real-time feedback of the expected results, immediately available to the user to explore for each device and each variable at any time interval along the timeline.

Consider the typical software code to be verified, such as a control law for avionics, or medical applications, while appearing very complex, it is usually designed to run at a duty cycle of 40 percent or less on an inexpensive microchip, which perform digital signal processing in computational “frames” of a few milliseconds. Rarely there are complex, non linear, mathematical algorithms needed. And if such complex algorithms are needed, they are highly optimized. Furthermore, rarely does a single test case required more than a 30 second timeline to prove that one aspect of the design is verified. However, many test cases may be required—normally a few per diagram. In other words, if the model of the program under test, produced by an independent interpretation of the requirements diagrams, is compiled on a relatively fast PC in “C” or Pascal, with virtually no memory constraints, the Test Case that would take microchip CPU time of 12 seconds (40% of 30 seconds), takes only a few seconds to run on the PC. This is an acceptable real time-response for the test engineer that changes a meaningful parameter—but it is not acceptable for looking at the changing values of variables when a mouse is moved on the timeline. When the engineer explores the possibilities of changing signals, the response time of the program must be much quicker. The above described forward and reverse timeline traversal debugging technique stores all the values for a complete timeline run within a few seconds, but allows the user to select a time, for example by using mouse movement or using arrow keys, and immediately—within milliseconds—see the values for all the variables.

The manual generation of Test Cases, aided through the use of timeline views of the signals, is not possible if the model program is written in an interpretative or semi-interpretative language. Experience with a large project provided that an environment such as PYTHON® has a greater than 10x response time, and SIMULINK® has a greater than 100x response time compared to a streamlined, compiled PASCAL rendition in which all variables are in static memory. (PYTHON is a registered trademark of Python Software Foundation Corporation, P.O. Box 37 Wolfeboro Falls, N.H. 03896-0037.)

After establishing that the Model code runs in a few seconds or less, two other features are useful in providing the perception of immediacy in displaying of data to the Test Case designer.

The first feature is the fact that the program tracks which diagram the user is viewing. The user can only look at one diagram at a time. Thus, when the user changes a diagram tab, a two second time for recalculation is acceptable. When the program tracks the diagram under test and the information associated with it, such as the position the flags on the diagram, the Model Core only needs to lift the values of the variables the user can see from static memory, at end of each computational frame. This can be done without disturbing the model code execution or adding cycles to it. The data is quickly accessed and copied from the variable's values, at each computational frame, are stored sequentially in memory, for immediate access, with access via frame number. This allows the corresponding values to be immediately displayed when the engineer selects any time corresponding to a computational frame. While it is true that all the data could be saved at each frame, it would require many gigabytes of memory for a large project, and the program would be slow for a variety of other reasons, such as memory paging.

Therefore, all the data related to all the I/O variables that the user can actually “see” on the timeline, and only those I/O variables, appropriately organized in meaningful groups, are stored in a memory array for each frame of execution along the timeline. The data related to the I/O variables that the user can see may include either the flag over the variable name in the diagram bitmap or the variable plotted with appropriate bias and colors.

After the data has been stored, when the user moves the mouse on the timeline plot, the model core only displays all the variables that the user can see on the diagram under being viewed. Since this process typically takes a very short time for a reasonable diagram, for example, less than 100 milliseconds after the mouse settles, the user perceives that the program responds immediately.

Also useful is the fact that the user does not have to type the variable names; the values of all the variables that can be seen are immediately reported. The user needs only to select the time (i.e. processing frame) with a single mouse motion—for any frame in the timeline—moving the cross-hair on the canvas.

Sometimes diagrams pre-consume variables that are produced later during the computational chain. In other words, a variable assumes two different values during a single computational frame, such as when an output signal is fed back to some input. In an example embodiment, pre-consumption is solved by storing the value of the intermediate, non-Real World I/O variables at the time when the diagram under test is executed, in the order of execution chain, and not at the end of the computational frame. This feature can also be bypassed, if the need arises.

The teachings of all patents, published applications and references cited herein are incorporated by reference in their entirety.

While this invention has been particularly shown and described with references to example embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention encompassed by the appended claims.

Claims

1. A method for verifying a target component in a program model of a signal processing system comprising:

determining a list of one or more Real World input variables that require modifications to achieve verification of the target component in the program model;

accepting an input specifying at least one point of change in the value of at least one of the input variables along a timeline; and

adding attributes to the point of change to determine an Intelligent Point that specifies how the input variable changes at a later point in time along the timeline, the attributes specifying at least one of

a linearly interpolation from two points;

holding the variable value between two points;

inject a frequency as a sine wave, a square wave, a triangular wave, or a variable frequency with varying frequency and amplitude parameters;

an Exhaustive test for Boolean variables.

2. The method of claim 1 wherein the attributes further comprise:

information on how to graphically display the variable including at least one of a color, a number of decimals formatting criteria, position on a canvas, or scale.

3. The method of claim 1 wherein at least one of the attributes of the Intelligent Point are specified via a graphical user interfaces.

4. The method of claim 1 additionally comprising:

a. automatically generating a timeline for one or more input variables according to the associated Intelligent Points;

b. evaluating Expected Results for output variables resulting from the program model that represent outputs of the targeted component.

5. The method of claim 1 additionally comprising:

outputting the timeline generated in step a. to a real world system;

collecting Actual Results from the same; and

comparing Expected Results and Actual Results.

6. The method of claim 1 additionally comprising:

for each increment of time between a start time and an end time along the time line, determining values for at least one variable at each such increment of the time line; and

saving the values so determined;

at a later time,

retrieving at least a portion of the saved values;

displaying a graphical representation of the values of the variables over at least a portion of the time line, along with at least one numerical representation of at least one variable at a selected time; and

providing a user-manipulable element of the graphical representation that is used to specify the selected time, and such that the user-manipulable element can be visually moved backward and forward along the graphical representation.

7. The method of claim 6 additionally comprising:

accepting an input specifying one or more changes in an attribute of an Intelligent Point;

re-determining values of variables affected by the changed Intelligent Points; and,

thus providing visualization of any change in the Intelligent Point attributes.