Test Lab Guidelines - DRAFT 0.45

I. General Information

II. Unpacking and setting up the test system

• 1. Instructions for installing the test suite
• 2. Instructions for rendering the suite for a particular test run
• 3. Instructions for canonicalizing and comparing output
• 4. Instructions particular to each "operation" scenario
• 5. Guidance on running single test cases that check the setup
• 6. Guidance on the uses of individual data items in the catalog
• 7. Guidance on the directory structure (e.g., read-only parts)

• 1. Information about discretionary items and the questionnaire
• 2. Information on the W3C specs and errata we deal with
• 3. How to submit test cases
• 4. The OASIS generic testing framework and this package as an instance of it

• 1. File tree of test cases (read-only)
• 2. Documents in top-level DOCS directory
• 3. Utility stylesheets in TOOLS
• 4. Reference output in two trees, raw and InfoSet (read-only); will need 3rd for canonical
• 5. Generate the canonical version of the reference output (one-time process on your system)
• a. Known conformant canonical processor available at http://www.elcel.com/products/xmlcanon.html
• b. Checking that it ran correctly
• 6. Suggestion: directory trees for each processor being tested
• 7. May need extra trees for different invocation options, run dates, etc.
• 8. Remarks about relative paths

• 1. You will need to know how to invoke each processor (see next section)
• 2. Operating environment: command shell? JRE? other?
• 3. Do you want to create output in its storage place or move it when done?
• 4. Master script that runs all test cases, invoking single-test script
• 5. Byte-wise file comparison tool
• 6. Capturing console output
• 7. Lack of generated output, planned vs. unplanned
• 8. If multiple processors under test, need to render suite for each to account for discretionary items
• 9. If a processor "compiles" the stylesheet first, plan for storage of compiled versions

III. Getting an XSLT processor ready for testing

• 1. Specifications of the necessary operating system (or JRE)
• 2. Specifications of other software needed to run the processor
• 3. Instructions for installing the processor
• 4. Command-line and/or API arguments; invocation sequence
• 5. Information about how errors and xsl:message are handled
• 6. Their answers to our questionnaire about discretionary items (xsltDevQ.htm)
• 7. Information about languages and locales supported
• 8. (For later use) How to set parameters "externally"

• 1. Vendor's setup tests
• 2. Try a couple tests from this suite
• 3. Devise a single-case script, batch file, or invocation program
• 4. Try pertinent invocation options

• 1. The sufficiency test (MkFinder.xsl) is also an example
• 2. Trial generation of a master script for running tests
• 3. How you can manipulate fixed strings and name parts to make other scripts

IV. Running the tests and evaluating a processor

• 1.Script to check that all the input files needed by a test case exist in the correct directories
• a. The "cd issue" arises because the principal stylesheet can refer to files in other roles
• 1) supplemental-data: document('filename')
• 2) supplemental-stylesheet: xsl:import or xsl:include files
• b. Supplemental files may have relative paths, not just names.
• c. The current directory should be the one containing the stylesheet.
• d. The supplemental inputs can be tested for existence relative to that directory.
• 2. The test lab uses catalog data to generate a script with all the necessary "cd" (or equivalent) commands intermixed among the commands to invoke the tests.
• 3. The following naming parts are supplied to the test lab
• a. test-catalog/@submitter names the directory for one submission
• b. file-path gives the intermediary directories to locate a case
• c. test-case/@id is the name of a case for display purposes
• d. input-file and output-file elements contain names of all files used in a test case
• e. Note that (a) and (b) are only separate for Committee purposes, and that all the test lab uses need to use the concatenation of the two (with a / or appropriate separator between).
• 4. Using the above name parts plus naming conventions suggested by the testing guide, a test lab should be able to create all the following in an automated way. In these items, "script" means a batch file, shell script, or whatever. These would be generated off the catalog data by an XSLT transformation with text-method output
• a. Script to canonicalize all InfoSetized reference output
• b. Script that wraps the process of rendering the test suite for a given processor and errata level
• c. Script to check that all the input files needed by a test case exist in the correct directories (MkFinder.xsl does this)
• d. Line in a script to run a test from the command line, feeding in the name of the principal output file, and ensure that the output is either generated in or moved to the correct directory
• e. Variation of (d) where console output must be captured and moved to an appropriate directory.
• f. Script to run all tests using a mixture of (d) and (e), and also make a log file that names the cases being run (could also display "purpose" strings or spec-citations if desired)
• g. Programs similar to (d) and (e) that use API calls
• h. Scripts to InfoSetize and then canonicalize the output from the test run, using different directory trees for each
• i. Script to compare all canonicalized test-run outputs against the corresponding reference outputs
• j. Script to delete files no longer needed after a successful run
• k. HREF-clean version of fully-qualified name (change / to _)
• 5. Each "operation" has its own details of invocation
• a. standard: provide both XML and stylesheet as arguments
• b. embedded: provide XML as only input
• c. external-param [TBD later]
• d. execution-error: need to capture console, transformation output is a mistake
• e. result-analysis: human review of results needed
• f. (should xsl:message have its own, combining a and d?)
• 6. Certain "compare" values require special attention
• a. ignore: presence of this file is a mistake
• b. manual/generate-id: look at generated IDs, ensure that they are in correct format
• c. manual/system-property: look at actual values, ensure they are accurate

• 1. Gray areas and discretionary items filter out cases - need data in XML
• 2. Specify the errata level
• 3. Rendering the processor-specific suite to exclude inapplicable test cases creates a trimmed catalog

• 1. The XSLT/XPath Conformance Committee will supply the test lab with the expected results and the test lab will apply an Information Set Analysis mechanism to produce the Users Results Description of the actual test results run on a particular processor.
• 2. The XML results can be serialized in canonical or other proprietary consistent way and then compared byte wise.
• 3. Alternatively, HTML or XML can be compared directly if a suitable comparitor exists
• 4. The raw results from XSLT Stylesheet (Test Case) can be XML, HTML or text format.
• 5. The user's actual results after running the test can be XML, HTML or text format.

• 1. XML Output
• a. The XML results will be converted to an XML document using a subset of the XML InfoSet
• b. Canonical XML or any consistent serialization can be used for byte-wise comparison
• 2. Text Output
• a. If the results are text, the test lab or user can be supplied with a form of the text coded in an XML InfoSet document.
• 1) Encoded in UTF-8
• 2) Separate text node for each line
• 3) Actual encoding specified as an attribute of the root
• b. An alternate approach is supplying the output as the combination of all text nodes from XML output in document order.
• 3. HTML Output - This one is hard due to:
• 1) Different treatment for tags and attributes between HTML and XML
• a. HTML allows attributes without values. Ex: INPUT disabled /
• 2) Different treatment of white spaces between HTML and XML
• b. Tools to solve:
• 1) W3C Tidy tool to solve 1st problem
• 2) Tool that produce consistent output regarding whitespace (IE libraries has one)
• 3) If you have money: HTML Match? DiffDoc? CS-HTMLdiff? HTML Compare? more?