Availability

Child pages
  • Feature - automatic comparisons

Skip to end of metadata
Go to start of metadata

Description

GINT provides considerable help for validating data produced by testcases matches expected results. A testcase simply needs to declare data expected in the output. Another useful technique is for a testcase to declare data that, if found in the output, indicates that the test has failed.

Capabilities

  1. Data found in output of command - list of strings and/or regular expression patterns
  2. Data that must NOT be found in output of command - list of strings and/or regular expression patterns
  3. Output file that needs to be produced by the command or inline code
  4. Output data that needs to be found in the file produced by the command or inline code - list of strings and/or regular expression patterns
  5. Output data that must NOT be found in the file produced by the command or inline code - list of strings and/or regular expression patterns
  6. File differences between current and expected output files are within tolerance condition
  7. User defined closure that can do additional comparisons

Ordered data

In some cases, the order of the data to be verified is important. Therefore, the various data verifications have been expanded to also accept a map with ordered and unordered data. Example - data: [ordered: ['aaa', 'bbb']]

 

Using

These are the Testcase parameters that control automatic comparisons:

  1. data - a list of strings or regular expression patterns that must be found in the output produced by running the command or inline code
  2. failData - a list of strings or regular expression patterns what will cause the testcase to fail if any of them are found in the output produced by running the command or inline code
  3. patternFlag - used for string matching - use Pattern.CASE_INSENSITIVE so ignore case on string comparisons
  4. output - a map that defines comparisons for an output file produced by running the command or inline code
    • output.file - the file name that is expected to be produced
    • output.data - similar to data above, except will be used for matching on the output file data
    • output.failData - similar to failData above, except will be used for matching on the output file data
    • output.patternFlag - similar to patternFlag above, except will be used for matching strings on the output file data
  5. compare - true or a map that requests running a full file difference command on files. This capability requires an appropriate diff command be available on the system - most operating systems except Windows have a standard diff command available. Window's users may need to do an install - see Dependencies. Normally, this is used to compare the output of the testcase, but, can also be used in a separate testcase that is just comparing the output of other testcases or operations. Diff output explains the output when differences are detected.
    • compare: true - compare is done using standard defaults
    • compare.fileName - file name for comparison if either file1 or file2 is not specified explicitly, defaults to the testcase name
    • compare.asDir - true to treat a fileName with no extension as a directory name to allow for the entire directory to be compared. Otherwise the fileName will get extension '.txt' by default.
    • compare.file1 - usually the expected file output, defaults to the fileName in the the compare directory
    • compare.file2 - usually the file output just created, defaults to the fileName in the output directory
    • compare.expected - expected number of line differences between the files, default is 0. Comparison is considered successful if the number of differences is less than or equal to this number.
    • compare.fileDiff - either true or a file name. If specified, the output from the diff command will be redirected to a file. Default file name is the testcase name with '.diff' extension in the output directory. If not specified, the diff output is logged when verbose logging is requested.
    • compare.ignoreCase - true to ignore case in the comparison (case insensitive)
    • compare.ignoreWhiteSpaceChanges - true to ignore changes in white space in the comparison
    • compare.ignoreAllWhiteSpace - true to ignore all white space in the comparison
    • compare.ignoreBlankLines - true to ignore blank lines in the comparison
    • compare.options - other diff command line options for advanced users - diff reference

Regular expressions

  • Regular expression patterns provide powerful matching capabilities when simple string matching is not sufficient
  • Regular Expression Test Page is useful for testing regular expressions
  • While patternFlag can be used for setting special matching characteristics, an alternative is to use embedded actions. A case insensitive pattern string example is: (?i)find this. Here are some:

    (?i) - case insensitive
    (?m) - multiline 
    (?s) - dot all, . matches line terminators
    (?d) - unix (\x) line endings 
    (?u) - unicode
    

Examples

Data and FailData

Logged output can be compared directly using the data or failData parameters.

gint.add(
    [name: 'dir', 
        cmd: 'dir c:\\.', 
        data: ['Program Files', 'Windows'], 
        failData: ['usr', 'bin'],
    ]
])

File Output

The output file in this case is named after the testcase in the standard output location. This means means the output parameter will default the file parameter correctly without it being explicitly set.

gint.add(
    [name: 'dirToFile', description: 'Note compare uses a case insensitive regex pattern',
        cmd: "dir c:\\. > ${gint.getOutputFile('dirToFile')}",
        output: [data: [~/(?i)program files/, 'Windows'], failData: ['usr', 'bin']],
    ]
])

Avoid hardcoded directory references

GINT provides functions and capabilities to help testcases avoid making hardcoded directory references. Using these functions together with the defaulting provided by GINT, testcases can be written more concisely and be more maintainable. For instance, functions are provided for:

  1. getInputFile, getInputDirectory
  2. getOutputFile, getOutputDirectory
  3. getResourceFile, getResourceDirectory
  4. getCompareFile, getCompareDirectory

File Compare

The support allows for any files to be compared. However, this example will show a specific use case supported by GINT that automatically compares the current run with results from a previous release. By default, GINT makes it very easy to save results for a software release that can later be used to verify a new release.

The following test run will direct all output from the test run to a compare directory: compare/myTest/R1.0

gant -f myTest -Dlabel=R1.0

The following test run will enable testcases that use the compare parameter to automatically compare output files to R1.0

gant -f myTest -Dlabel=R1.1 -DcompareLabel=R1.0

In myTest, at least one of the testcases uses the compare parameter.

gint.add(
    [name: 'dirToFile',
        cmd: "dir c:\\. > ${gint.getOutputFile('dirToFile')}",
        output: [data: ['Program Files', 'Windows'], failData: ['usr', 'bin']],
        compare: true,
    ]
])





Default compare

compare: true is the easiest way to get the default behavior for this use case. This works because default names and locations are used. The output file is named after the testcase name and is put in the default output location. If a few line differences are expected in the file compare, simply use compare: [expected: 2]

  • No labels