WO2003043255A9 - Method and system for computer based testing using an amalgamated resource file - Google Patents

Method and system for computer based testing using an amalgamated resource file

Info

Publication number
WO2003043255A9
WO2003043255A9 PCT/US2002/036220 US0236220W WO03043255A9 WO 2003043255 A9 WO2003043255 A9 WO 2003043255A9 US 0236220 W US0236220 W US 0236220W WO 03043255 A9 WO03043255 A9 WO 03043255A9
Authority
WO
WIPO (PCT)
Prior art keywords
segment
test
storage location
expansion module
storage
Prior art date
Application number
PCT/US2002/036220
Other languages
French (fr)
Other versions
WO2003043255A2 (en
WO2003043255A3 (en
Inventor
Clarke D Bowers
Original Assignee
Prometric Inc
Clarke D Bowers
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Prometric Inc, Clarke D Bowers filed Critical Prometric Inc
Priority to AU2002360371A priority Critical patent/AU2002360371A1/en
Publication of WO2003043255A2 publication Critical patent/WO2003043255A2/en
Publication of WO2003043255A9 publication Critical patent/WO2003043255A9/en
Publication of WO2003043255A3 publication Critical patent/WO2003043255A3/en

Links

Classifications

    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09BEDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B5/00Electrically-operated educational appliances
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/36Preventing errors by testing or debugging software
    • G06F11/3668Software testing
    • G06F11/3672Test management
    • G06F11/3684Test management for test design, e.g. generating new test cases
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00Handling natural language data
    • G06F40/10Text processing
    • G06F40/12Use of codes for handling textual entities
    • G06F40/14Tree-structured documents
    • G06F40/143Markup, e.g. Standard Generalized Markup Language [SGML] or Document Type Definition [DTD]
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00Handling natural language data
    • G06F40/10Text processing
    • G06F40/166Editing, e.g. inserting or deleting
    • G06F40/174Form filling; Merging
    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09BEDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B19/00Teaching not covered by other main groups of this subclass
    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09BEDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B5/00Electrically-operated educational appliances
    • G09B5/06Electrically-operated educational appliances with both visual and audible presentation of the material to be studied
    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09BEDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B7/00Electrically-operated teaching apparatus or devices working with questions and answers

Definitions

  • the present invention generally relates to the field of computer-based testing, and in particular, the present invention relates to amalgamating an exam resource file to combine first defined and later defined test content and specification of a computer-based test to reduce the amount of authored XXL and to reduce the size of the size of the exam resource file during delivery of the computer-based test to an examinee.
  • a BACKGROUND OF THE RELATED ART ' BACKGROUND OF THE RELATED ART
  • FIG. 1 illustrates a prior art process for computerized test customization, denoted generally by reference numeral 10.
  • a client details the desired test requirements and specifications, step 12.
  • the computerized test publisher then creates the tools that allow the test publisher to author the items, presentations, etc., required to fulfill the requirements, step 14.
  • the test publisher then writes an item viewer, which allows the test publisher to preview what is being authored, step 16.
  • An item presenter is then written to present the new item, for example, to the test driver, step
  • test driver Presenting the new item to the test driver requires a modification of the test driver's executable code.
  • the test driver must be modified so that it is aware of the new item and can communicate with the new item presenter, step 20.
  • the test packager must then also be modified, step 22.
  • the test packager which may also be a compiler, takes what the test publisher has created and writes the result as new object codes for the new syntax.
  • the scoring engine must also be modified to be able to score the new item type, step 24.
  • results processor must be modified to be able to accept the new results from the new item, step 26. This process requires no less than seven software creations or modifications to existing software.
  • test specification based on the order of delivery and the hierarchy may be: a two-section exam of math followed by English. All math questions have a minimum response value of 2 and all English questions minimum response value of 3. An example that will not support repetition of test definition is a two-section exam of math and English. Where half the math questions have a minimum response of 2 and the other half of the math questions have a minimum response of 3. In this situation, the hierarchy does not match the pattern of repetition.
  • the test development system comprises a test document creation system for specifying the test contents, an item preparation system for computerizing each of the items in the test, a test preparation system for preparing a computerized test, and a test packaging system for combining all of the items and test components into a computerized test package.
  • the computerized test package is then delivered to authorized examinees on a workstation by the test delivery system.
  • Figure 2 illustrates the relationship among session scripts 30, test scripts 32, and units.
  • a script consists of a series of files and further specifies the option settings and configuration data, which the
  • Test Delivery Application needs for operation.
  • scripts are prepared and combined with the items prepared during item preparation. Scripts control the sequence of events during a testing session.
  • Two types of scripts are preferably used, for example: the session script 30 and one or more test scripts 32.
  • the session script 30 controls the order in which units within the testing session are presented. Units provide specific services to the examinee, such as delivering a test or presenting a score report.
  • the test script controls what is presented to the examinee during the testing unit.
  • Each testing unit may include one or more delivery units, which are separately timed and scored subdivisions of a test.
  • the system can dynamically select, or spiral, scripts and other test components so that examinees are given what appear to be different tests.
  • FIG. 24 shows the relationship among session scripts 30, test scripts 32, and units.
  • the session script is the second-level component of the testing package. It performs two primary functions: First, it specifies the Session Control Information, which defines the default options that are in effect for the entire examinee testing session. Second, it controls the order in which units within the testing session are presented and the options used to present them.
  • the units that can be presented within a session script are: General information screen units, tutorial units, Break units, Data collection units, Scoring and Reporting units, and Testing units.
  • the session control information contains the default options in effect for the entire session. Control information can be provided at multiple levels within the testing session. Thus, the control information provided at the session level can be overridden by information that occurs later in the session.
  • the information provided at the session level would generally include the following: Name-- the session script name to be used by administrators in selecting a specific session script from Administrative Application menus; Input device—the input device to be used during the session (e.g., mouse or keyboard); Color— the colors to be used during the session; Messages— program-specific messages to override default messages during the session; Demo Script— indicates whether the script presents a demonstration or operational test; Research Indicator— indicates whether the script presents a research pilot test; Special Timing— indicates whether the script is standard or specially timed version.
  • the testing unit presents a test, based on the contents of a test script that may have been selected at runtime.
  • the following units can be included within a testing unit: general information screen unit; tutorial unit; break unit; delivery unit, which delivers items to the examinee. This permits testing programs to interleave general information screens, tutorials, and breaks with sections of a test.
  • the testing unit contains the following information: script selection mode indicates whether dynamic runtime selection is to be used to select the test script; reference to a test script which controls the sequence of events and options used during the testing unit. If dynamic runtime selection is to be used, the reference is to a set of test scripts.
  • the test script performs two primary functions. First, it specifies the test and delivery unit control information.
  • Test control information defines the options that are in effect for the testing unit.
  • Delivery unit control information defines the options that are in effect for a particular delivery unit within a testing unit. It controls the order in which units are presented within the testing unit and the options used to present them. The rules for presentation of units are the same as those for the session script, except that an additional unit, the delivery unit, can be included within a test script.
  • U.S. Patent No. 5,513,994 (Kershaw et al.), which is incorporated herein by reference, discloses a centralized administrative system and method of administering standardized test to a plurality of examinees.
  • the administrative system is implemented on a central administration workstation and at least one test workstation located in different rooms at a test center.
  • the administrative system software which provides substantially administrative functions, is executed from the central administration workstation.
  • the administrative system software which provides function carried out in connection with a test session, is executed from the testing workstations.
  • test specification and content to reduce the size of a resource file that contains the test specification and content, where a test driver and an expansion module uses the test specification and content to deliver a test to an examinee.
  • a test publisher uses the test specification and content to deliver a test to an examinee.
  • a test publisher uses the test specification and content to deliver a test to an examinee.
  • a test publisher uses the test specification and content to deliver a test to an examinee.
  • test publisher to define different aspects of test specification and content for a particular feature of the test in multiple locations and to amalgamate those multiple locations at the time of delivery of the test such that the test publisher does not have to repeat the same aspects if the test specification and content for the particular feature at the multiple locations.
  • later defined elements of a particular feature to override previously defined elements that exist at a higher level of the test specification and contents.
  • the test has a presentation format that determines the visual presentation of the test and data content that determines the functional properties of the test.
  • the system includes a storage device that has a first storage location and a second storage location. The first storage location stores a first segment of a test definition language and the second storage location stores a second segment of the test definition language.
  • the first segment and the second segment define information comprising at least one of the data content, the presentation format, progression, scoring, printing, timing, and/or results reporting of the test.
  • the first segment and the second segment of the test definition language further comprise the same category of information, where the category is at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and/or the results reporting of the test.
  • the system also includes a validation expansion module that validates the first segment and the second segment of the test definition language by determining whether the first segment and the second segment are correctly formatted.
  • the validation expansion module also stores the first segment to one of the first storage location and the second storage location and the second segment to another one of the first storage location and the second storage location in the storage device.
  • the validation expansion module is a plugin.
  • the system further includes a test packager that transmits the first segment and the second segment of the test definition language to the validation expansion module during delivery of the test.
  • the test packager determines to which of the first storage location and the second storage location in the storage device the first segment and the second segment are stored by the validation expansion module.
  • the test packager also amalgamates the first storage location and the second storage location and stores an amalgamated segment of the test definition language in a first virtual storage location and transmits the amalgamated segment to the validation expansion module such that the validation expansion module can determine whether the amalgamated segment forms a complete and valid set of the first segment and second segment of the test definition language.
  • the test packager is a compiler.
  • the system further includes a test driver that has an executable code that controls functionality that enables the test driver to deliver the test to an examinee using a display device, manage the test, control the progression of the test, control the scoring of the test, control the printing of the test, control the timing of the test, and control the results reporting of the test based on the test definition language.
  • the test driver amalgamates the first storage location and the second storage location into a second virtual storage location such that the validation expansion module can retrieve the amalgamated segment from the second virtual storage location to enable the functionality of the test driver.
  • a system for computer-based testing includes a storage device that has a first storage location, a second storage location, and a third storage location.
  • the first storage location stores a first segment of a test definition language
  • the second storage location stores a second segment of the test definition language
  • the third storage location stores a third segment of the test definition language.
  • the first segment, the second segment, and the third segment comprise at least one of the data content, the presentation format, progression, scoring, printing, timing, and/or results reporting of the test.
  • the first segment, the second segment, and the third segment of the test definition language further comprise the same category of information, where the category is at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and the results reporting of the test.
  • the system also includes a validation expansion module that validates the first segment, the second segment, and the third segment of the test definition language, and stores the first segment to one of the first storage location, the second storage location, and the third storage location, the second segment to another one of the first storage location, the second storage location, and the third storage location, and the third segment to another one of the first storage location, the second storage location, and the third storage location.
  • the validation expansion module is a plugin.
  • the system further includes a test packager that transmits the first segment, the second segment, and the third segment of the test definition language to the validation expansion module during production of the test such that the validation expansion module is capable of validating the first segment, the second segment, and the third segment.
  • the test packager determines to which of the first storage location, the second storage location, and the third storage location in the storage device the first segment, the second segment, and the third segment are stored by the validation expansion module.
  • the test packager also amalgamates the first storage location, the second storage location, and the third storage location and stores an amalgamated segment of the test definition language in a first virtual storage location, and transmits the amalgamated segment to the validation expansion module such that the validation expansion module is capable of determining a complete and valid set of the first segment, the second segment, and the third segment of the test definition language.
  • the test packager is a compiler.
  • the system also includes a test driver that has an executable code that controls functionality performed by the test driver that enables the test driver to deliver the test to an examinee using a display device, manage the test, control progression of the test, control scoring of the test, control printing of the test, control timing of the test, and control reporting of test results based on the test definition language.
  • the test driver amalgamates the first storage location, the second storage location, and the third storage location into a second virtual storage location such that the validation expansion module is capable of retrieving the amalgamated segment from the second virtual storage location to enable the functionality of the test driver.
  • a method of computer-based testing for a test is provided, where the test has a presentation format that determines the visual presentation of the test and data content that determines the functional properties of the test. Delivery of the test is controlled by a test driver that has an executable code that enables the test driver to deliver the test to an examinee using a display device, manage the test, control progression of the test, control scoring of the test, control printing of the test, control timing of the test, and control results reporting of the test.
  • the method includes the sequential, non-sequential, and/or sequence independent steps of validating a first segment of the test definition language during a test production cycle and validating a second segment of the test definition language during the test production cycle.
  • the first segment and the second segment define information comprising at least one of the data content, the presentation format, progression, scoring, printing, timing, and/or results reporting of the test.
  • the first segment and the second segment further define the same category of information, which is at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and the results reporting of the test.
  • the method also include amalgamating the first segment and the second segment of the test definition language during the test production cycle such that an amalgamated segment is formed, validating the amalgamated segment during the test production cycle, such that a validated amalgamated segment is created and such that the amalgamated segment is valid if the amalgamated segment forms a complete and valid set.
  • the method further includes amalgamating the first segment and the second segment of the test definition language during a test delivery cycle. The validated amalgamated segment is reformed and retrieved by a validation expansion module to enable the functionality of the test driver.
  • a method for computer-based testing includes validating a first segment of the test definition language during a test production cycle, validating a second segment of the test definition language during the test production cycle, and validating a third segment of the test definition language.
  • the first segment, the second segment, and the third segment define information comprising at least one of the data content, the presentation format, progression, scoring, printing, timing, and/or results reporting of the test.
  • the first segment, the second segment, and the third segment of the test definition language further define the same category of information, the same category of information being at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and the results reporting of the test.
  • the method also includes amalgamating the first segment, the second segment, and the third segment of the test definition language during the test production cycle, such that an amalgamated segment is formed and validating the amalgamated segment during the test production cycle, such that a validated amalgamated segment is created and such that the amalgamated segment is valid if the amalgamated segment forms a complete and valid set.
  • the method further includes amalgamating the first segment, the second segment, and the third segment of the test definition language during a test delivery cycle, such that the validated amalgamated segment is reformed and retrieved by a validation expansion module to enable the functionality of the test driver.
  • a method for computer-based testing includes defining the presentation format and the data content in at least two locations comprising a plugin element and actual usage of the plugin element on at least one unit of the test, the at least one unit comprising form, section and group associated with the test and amalgamating the presentation format and the data content defined in the two locations by at least one test driver to deliver the test to an examinee.
  • a method for computer-based testing includes validating by a plugin at least partial exam source information that is received and amalgamating exam resource data associated with the test. The method also includes validating the exam resource data that has been amalgamated to provide a substantially complete amalgamated exam specification and content and delivering the substantially complete amalgamated exam specification and content validated by the validating step.
  • a method for computer-based testing for a test that has a first presentation format, a second presentation format, a first data content, and a second data content.
  • the method includes defining the first presentation format and the second presentation format in at least two locations comprising a plugin element and actual usage of the plugin element on at least one unit of the test, the at least one unit comprising form, section and group associated with the test.
  • the method further includes defining the first data content and the second data content in at least two locations comprising the plugin element and the actual usage of the plugin element on at least one unit of the test, the at least one unit comprising form, section and group associated with the test.
  • the method also includes amalgamating at least one of the first and second presentation format and the first and second data content defined in the two locations by at least one test driver to deliver the test to an examinee.
  • Figure 1 is a flow diagram of a prior art method for computerized test customization
  • Figure 2 is a block diagram of a prior art testing script
  • Figure 3 is a schematic diagram of a computer-based testing system according to the present invention
  • Figure 4 is a block diagram illustrating different types of plugins that are used with the computer-based testing system according to the current invention
  • Figure 5 illustrates various components that comprise an exam source file
  • Figures 6A and 6B are schematics illustrating the components, classes, and interfaces that comprise a test definition language compiler according to the present invention
  • Figure 7 is a schematic illustrating the components that comprise a test driver and a test administration system according to the present invention.
  • Figures 8A and 8B are schematic illustrating the classes and interfaces that comprise the test driver
  • Figure 9 illustrates the interfaces that comprise a structured storage according to the present invention.
  • Figures 10A and 10B are schematics illustrating the classes and interfaces that comprise the structure storage and associated operations;
  • Figure 1 1 is a block diagram of main storage branches of an exam resource file according to the present invention.
  • Figure 12 is a block diagram illustrating an exams branch of the exam resource file
  • Figure 13 is a block diagram illustrating a forms branch of the exam resource file
  • Figure 14 is a block diagram illustrating an items branch of the exam resource file
  • Figure 15 is a block diagram illustrating a categories branch of the exam resource file
  • Figure 16 is a block diagram illustrating a templates branch of the exam resource file
  • Figure 17 is a block diagram illustrating a sections branch of the exam resource file
  • Figure 18 is a block diagram illustrating a groups branch of the exam resource file
  • Figures 19A, 19B, 19C, and 19D are block diagrams illustrating an events sub-branch of the groups branch of the exam resource file
  • Figure 20 is a block diagram illustrating a plugins branch of the exam resource file
  • Figure 21 is a block diagram illustrating a data branch of the exam resource file
  • Figure 22 is a block diagram illustrating a formGroups branch of the exam resource file
  • Figure 23 is a block diagram illustrating an attributes branch of the exam resource file
  • Figure 24 is a block diagram illustrating a scripts branch of the exam resource file
  • Figure 25 is a block diagram illustrating a message box branch of the exam resource file
  • Figures 26A, 26B, 26C, and 26D are block diagrams of an exam instance file according to the present invention
  • Figure 27 is a flow diagram of a method for computerized test customization according to the present invention.
  • Figure 28 is a diagram of a life cycle of a plugin according to the present invention
  • Figure 29 is a flow diagram of a process for compiling plugins according to the present invention
  • Figures 30A, 30B, 30C, and 30D are flow diagrams of a process for delivering plugins to an examinee during a computer-based test
  • Figure 31 is a flow chart illustrating a process for amalgamation of invisible plugins according to the present invention.
  • Figure 32 is a flow chart illustrating a process for amalgamation of visible plugins according to the present invention.
  • Figure 33 is a flow chart of a process for amalgamation according to the present invention.
  • Figure 34 is a flow chart for a process for validating exam source according to the present invention.
  • Figure 35 is a flow chart for a process for validating amalgamated test specification and content using a plugin according to the present invention
  • Figure 36 is a flow chart for a process of delivering amalgamated test specification and content using a test driver and a plugin according to the present invention
  • Figure 37 is a flow diagram for an example of amalgamation of test specification and content relating to items according to the present invention.
  • the present invention discloses a system and method of computer-based testing using a test driver that is, for example, object-oriented and is architected to dynamically add functionality through, for example, the use of an expansion module, and preferably through the use of plugins.
  • the test driver preferably references component object model servers using standard interfaces, and uses, for example, class names (that can be an Active Document) defined in a custom test definition language entitled extensible eXam Language (“XXL”) based on extensible Markup Language (“XML”) format to interact with existing applications while offering the flexibility of allowing development of new plugins.
  • XXL extensible eXam Language
  • XML extensible Markup Language
  • the plugins advantageously enable the test driver to support, for example, new item types, navigation algorithms, information displays, scoring algorithms, timing algorithms, test unit selection algorithms, results persistence reporting, printed score reporting, and/or helm types without change to the test driver's executable.
  • Plugins also allow expansion of the test driver's functionality without requiring the test driver to be recompiled or re-linked, and without requiring the test publisher to learn to program. Since plugins are written independently of the test driver, plugins can be written long after the test driver is built.
  • the client and the software developer can design and test the plugins and distribute the plugins to each test site. By using this method, large-scale regression testing of other examinations will not usually be necessary unless changes are made to the plugins that may be used by many examinations.
  • a test publisher defines common plugin test specification and common plugin test content in early XXL elements. These previous elements can be referenced by later XXL elements. If the later element defines nothing more then it will receive just the common specification and content. Optionally, the later elements can define omitted non-common plugin specification and contents.
  • later elements can override previously defined common plugin specification and contents with exception plugin specification and contents.
  • This process of new definition and overriding of specification and contents by later XXL elements is not limited to two stages. Multiple later stages are allowed.
  • Test driver 110 is responsible for controlling all aspects of the computer-based test.
  • Test driver 1 10 identifies examinees scheduled to take the computer-based test and identifies and creates the appropriate test.
  • Test driver 110 then presents all of the test components to examinees using a display device (not shown), such as a computer monitor, and enables examinees to enter responses to test questions through the use of an input device (not shown), such as a keyboard, a mouse, etc.
  • Test driver 110 also monitors the security of the test. For example, test driver 110 can prevent access to the Internet and can validate examinees, although, these functions are preferably performed by the test center administration system.
  • Test driver 110 also monitors the timing of the test, providing relevant warnings to examinee regarding the elapsed time of the test and the time remaining for a particular section of the test or for the entire test. Test driver 110 is also responsible for scoring the test, once the test is completed or while the test is in progress, and for reporting the results of the test by physical printout using printer 182 or in a file format using candidate exam results file 180. If the test is interrupted while in progress, for example, due to a power failure, test driver 110 restarts the test, preferably at the point at which the test was interrupted, as will be described subsequently in more detail. Finally, if the test is left incomplete, test driver 110 cleans up the incomplete test.
  • test driver 1 10 picks up where the event was interrupted and invisibly deliveries the rest of the units of the test.
  • a test specification is authored by a test publisher according to the specifications of the client and stored in exam source files 130.
  • Exam source files 130 include data files 132, XXL files 134, multimedia files 136, and hypertext markup language (“HTML”) files 138.
  • XXL files 134 include the test specification, which contains the client's requirements for the test, a bank of test items or questions, templates that determine the physical appearance of the test, plugins, and any additional data necessary to implement the test. Additional data is also stored in data files 132. For example an adaptive selection plugin may need a, b &c theta values. These values are stored in a binary file created by a statistical package.
  • HTML files 130 include, for example, any visual components of the test, such as the appearance of test items or questions, the appearance of presentations on the display device, the appearance of any client specified customizations, and/or the appearance of score reports. HTML files
  • HTML files 130 are preferably authored using Microsoft's FrontPage 2000. FrontPage 2000 is preferably also used to manage the source files in a hierarchy that is chosen by the test publisher.
  • Multimedia files 136 include, for example, any images (.jpg, .gif, etc.) and/or sound files (.mp3, .wav, .au, etc.) that are used during the test.
  • XXL compiler 140 retrieves XXL files 134 from exam source files 130 using interface 190 and compiles the XXL test content stored in XXL files 134. XXL compiler 140 stores the compiled test files in exam resource file 120. In another embodiment, exam source files 130 do not contain XXL files 134 and contains, for example, only multi-media files. In this embodiment, XXL compiler 140 is merely a test packager that writes the data directly to exam resource file 120 without modification or validation. The data appears in a stream under the "data" branch of exam resource file 120. The name of the stream is specified by the test author.
  • XXL files 134 also include XXL language that defines plugins 150, in which case, plugins 150 assist XXL compiler 140 in compiling XXL files 134.
  • Test driver 110 preferably supports, for example, nine different types of plugins 150, including, for example: display plugin 152; helm plugin 154; item plugin 156; timer plugin 158; selection plugin 160; navigation plugin 162; scoring plugin 164; results plugin 166; and report plugin 168.
  • Plugins 150 which are also included in XXL files 134, are the first XML files compiled into exam resource file 120.
  • Plugins 150 allow a test designer to customize the behavior of test driver 110 and are divided into two types, for example: visible plugins and invisible plugins, as shown in Figure 4.
  • the visible plugins which include display plugin 152, helm plugin 154, and item plugin 156, enable the test driver to control what is presented visually to an examinee on the display device.
  • the invisible plugins which include timer plugin 158, selection plugin 160, navigation plugin 162, scoring plugin 164, results plugin 166, and report plugin 168, enable the test driver to control more functional aspects of the test.
  • Plugins 150 are used to validate data stored in exam source files 130 that is to be used by one of plugins 150 during delivery of the test to the examinee, as is described below in greater detail.
  • Plugins 150 are, preferably, component object model ("COM") objects, as described below. Plugins 150, may also utilize Java implementation. Plugins 150 are preferably written using Microsoft Visual C++ or Visual Basic 6.0 or any fully COM enabled language. Plugins 150 may be in or out-of-process, and, therefore, can exist as executable (“.EXE”) files or as dynamic link library (“.DLL”) files.
  • COM component object model
  • Plugins 150 may also utilize Java implementation. Plugins 150 are preferably written using Microsoft Visual C++ or Visual Basic 6.0 or any fully COM enabled language. Plugins 150 may be in or out-of-process, and, therefore, can exist as executable (“.EXE”) files or as dynamic link library (“.DLL”) files.
  • An application or component that uses objects provided by another component is called a client.
  • Components are characterized by their location relative to clients.
  • An out-of process component is an .exe file that runs in its own process, with its own thread of execution. Communication between a client and an out-of-process component is therefore called cross-process or out-of-process communication.
  • An in-process component such as a .dll or .oxc file, runs in the same process as the client, it provides the fastest way of accessing objects, because property and method calls don't have to be marshaled across process boundaries.
  • an in-process component must use the client's thread of execution.
  • Exam resource file 120 receives the compiled test content from XXL compiler 140 and plugins
  • OLE object-linking and embedding
  • POLESS object-linking and embedding
  • Other storage formats may optionally be used.
  • OLE allows different objects to write information into the same file, for example, embedding an Excel spreadsheet inside a Word document.
  • OLE supports two types of structures, embedding and linking.
  • OLE embedding the Word document of the example is a container application and the Excel spreadsheet is an embedded object.
  • the container application contains a copy of the embedded object and changes made to the embedded object affect only the container application.
  • OLE linking the Word document of the example is the container application and the Excel spreadsheet is a linked object.
  • Test driver 110 comprises Active Document container application 112 for the visible plugins, display plugin 152, helm plugin 154, and item plugin 156, which function as embedded objects, preferably COM objects.
  • FIG. 3-1 shows an example of Active Document container application 112 being used with several item plugins 156.
  • Each item plugin 156 in this case representing a multiple choice ("multi- choice") item, a hot area item, and a fill in the blank item, are linked to Active Document container application ] 12 through the Iltem and IPlugin COM interfaces 169.
  • Both XXL compiler 140 and plugins 150 are involved in storing the compiled test content into exam resource file 120, if any of plugins 150 are being used.
  • Exam resource file 120 comprises, for example, a hierarchical storage structure, as will be described in further detail below. Other storage structures may optionally be used.
  • XXL compiler 140 determines to which storage location a specific segment of the compiled test content is to be stored. However, if any of plugins 150 are used to validate the portion of any of the data from exam source files 130, then the plugins 150 store the data directly to the exam resource file, based upon directions from XXL compiler 140.
  • XXL compiler uses IPersistResource interface 192, co-located with I-Plugin interface 167 in Figure 3, to control the persistence of the data to exam resource file 10.
  • XXL compiler 140 and plugins 150 write the data to exam resource file 120 using POLESS interfaces 191.
  • Figure 5 illustrates the contents of exam source file 130, which are compiled into exam resource file 120 by XXL compiler 140 and plugins 150.
  • FrontPage 2000 Web 200 is used, for example, to author the test.
  • Exam source files 130 contain media files 210, visual files 220, and logic files 230.
  • Media files 210 are multimedia files used to enhance the presentation of the test, including, for example, XML data files 212, sound files 214, image files 216, and binary files 218.
  • XML data files 212 include the XXL test definition language and the XXL extensions from the plugins 150 that use XML.
  • the test specification, presentation, scoring and other information is specified in the XML files.
  • Sound files 214 include any sounds that are to be used during the test, such as .mp3 files, .au files, etc.
  • Image files 216 include any images to be used during the test, such as jpg files, .gif files, etc.
  • Binary files 218 include any data needed by a plugin 150 that is not in XXL format.
  • Visual files 220 are HTML files that specify the visual presentation of the test as presented to the examine on the display device, including items files 222, presentation files 224, score report files 226, and custom look files 228.
  • Items files 222 include HTML files that are used to specify the visual component of test questions, e.g., stems and distractors. Items files 222 are capable also of referencing external exhibits. An exhibit could be a chart, diagram or photograph. Formats of exhibits include, for example: jpg, .png, etc. Presentation files 224 define what is seen by the examinee on the display device at a particular instant during the test. Score report files 226 include is typically an HTML file with embedded script that includes, for example candidate demographics, appointment information, and candidate performance. The performance might include pass/fail, achievement in different content areas, etc. Custom look files 228 include are typically HTML files with embedded script to layout, for example, the title bar and information contained therein.
  • Logic files 230 are XML files that specify the functional aspects of the test, including test specification files 232, plugin files 234, item bank files 236, and template files 238.
  • Test specification files 232 specify the content and progression of the test as provided by the client.
  • Plugin files 234 define plugins 150 and contain any data necessary to implement plugins 150.
  • Item bank files 236 include the data content and properties of the items, or test questions, that are to be presented to the examinee during the test. Properties of an item include the correct answer for the item, the weight given to the item, etc.
  • Template files 238 define visual layouts that are used with the display screen during the test.
  • test driver 110 accesses exam resource file 120 for the instructions and files needed to implement the test, using POLESS interfaces 193.
  • Test driver 110 also accesses plugins 150 for additional data that expands the functionality of test driver 110 in the areas of items, navigation algorithms, information displays, scoring algorithms, timing algorithms, test unit selection algorithms, results persistence reporting, printed score reporting, and/or helm types.
  • Test driver 110 communicates with plugins 150 using various COM interfaces 169.
  • COM interfaces facilitate OLE linking.
  • test driver 110 is an Active Document container application and plugins 150 are embedded objects.
  • the COM interfaces function as communications paths between the container application and the objects. There are, for example, ten COM interfaces utilized in computer-based test delivery system 100.
  • IPlugin interface 167 which is also a COM interface, is supported by all of plugins 150.
  • COM interfaces 169 therefore, includes the IPlugin interface.
  • the IPlugin interface contains generic operations such as loading and unloading required of all plugins 150.
  • each plugin 150 also uses, for example, a second, individual COM interface 169 to communicate with test driver 110.
  • Alternative structures of the IPlugin interface may also be used. Table 1 shows the relationship between each plugin 150 and the COM interface 169 used with that particular plugin 150.
  • Exam instance file 170 is used to restart a test if the test has been interrupted, for example, because of a power failure.
  • exam instance file 170 receives examination state information from test driver 110 and plugins 150 regarding the state of all running objects being used to deliver the test.
  • the examination state information includes the presentation that was being delivered on the display device before the interruption, the responses the examinee had entered in that presentation, etc.
  • the exam instance file 170 loads the state information back to test driver 110 and plugins 150, allowing the test to return to operation at the point where the test had been interrupted.
  • the running state of all objects is saved to exam instance file 170 rather than of only some of the objects.
  • Exam instance file 170 may also store additional information relating to the test, including, for example: the timing utilized and time remaining on units of the exam, the current unit of delivery, candidate score, etc.
  • Test driver 110 and plugins 150 communicate with exam instance file
  • Test driver 110 controls communications between test driver 110 and plugins 150 using IPersistlnstance interface 196, which is co-located with COM interfaces 169 in
  • TCM Test Center Manager
  • ETS Electronic Transaction Service
  • UAS Unified Administration System
  • Administrative functions include, for example: checking-in an examinee, starting the test, aborting the test, pausing the test, resuming the test, and transmitting results.
  • Test driver 110 There are preferably two ways to run Test driver 110. The first is through a series of command line options and the second is using COM interfaces describing appointment information.
  • the command line option exists for backwards compatibility in a standard ETS environment and a TCM environment.
  • Table 2 shows a list of command line options test driver 110 supports.
  • LaunchTest.exe for test production and client review
  • UAS for test production and client review
  • UTD2ETS.dll an internal compatibility module for use with the ETS administration environment
  • UTD2TCM for the Test Center Manger environment
  • the administration environments use several interfaces to communicate with test driver 110.
  • -Appointment interface 176 is part of UAS 174 and allows access by test driver 110 to examinee information for the examinee taking the test, such as demographics.
  • the examinee information is included in candidate exam results file 180, which is created by the test driver.
  • ILaunch2 interface 177 functions as the primary control interface for UAS 174 and allows UAS 174 to control various components such as test driver 110, screen resolution change, accommodations for disabled candidates, examinee check-in, etc., in a test center, which is the physical location where the examinee is taking the test.
  • ITransfer interface 199 transfers candidate exam results file 180 and other files back to UAS 174.
  • IPrint interface 198 sends information regarding any reports to printer 182.
  • Figures 6A and 6B illustrate the main diagram for XXL compiler 140.
  • XXL compiler 140 comprises the following classes, for example: cCompile 2000; cData 2004; cArea 2006; cTemplate 2008; cCategory 2010; cltem 2012; cPresentation 2014; cGroup 2016; cSection 2018; cForm 2020; cFromGroup 2022; cExam 2024; cMsgBox 2026; cChecksum 2028; cEvent 2030; cResult 2032; cReport 2024; cPlugin 2036; and cXXL 2038.
  • cCompile 2000 cData 2004; cArea 2006; cTemplate 2008; cCategory 2010; cltem 2012; cPresentation 2014; cGroup 2016; cSection 2018; cForm 2020; cFromGroup 2022; cExam 2024; cMsgBox 2026; cChecksum 2028;
  • ICo pile interface 2002 The main interface to XXL compiler 140 is ICo pile interface 2002.
  • ICompile interface 2002 is implemented by cCompiler class 2000. All control and initiation of compilation of exam source files 130 into exam resource file 120 occurs by way of this single public interface.
  • the core, non-plugin related elements of the XXL test definition language, as stored in XXL files 134, are compiled by classes in XXL compiler 140. For example, cSection class 2018, compiles the section element, and cGroup class 2016 compiles the group element.
  • ICompile interface 2002 supports the following operations, for example: createResource(); addSourceO; addData(); closeResource(); about(); linkResource(); openResource() and getCryptoObject().
  • CreateResource() creates a resource file, for example, an XXL based resource file such as exam resource file 120.
  • AddSource() compiles an XXL file into the resource file.
  • AddData() adds a file directly to a data branch of the resource file.
  • CloseResource() closes the resource file.
  • LinkResourceO links a resource in the resource file and is performed after all compiling of the source files are completed.
  • GetCryptoObject() returns an ICrypto object containing the current encryption setting of POLESS, as described below.
  • the classes of XXL compiler 1040 e.g., cForm 2020 and cltem 2012, handle individual XXL core language elements. All of these classes compile the specific XXL source element into exam resource file 120. All of these class language elements are also symbols used in later references. Therefore, the classes all derive from cSymbol class 2040.
  • cSymbol class 2040 allows the classes of XXL compiler 140 to reside in a symbol table.
  • This XXL call causes an instance of cPlugin class 2036 to be created, compiles the source, and writes the compiled result to exam resource file 120.
  • the name and ID of Plugin 150 is also added to the symbol table for later reference.
  • XXL compiler 140 also contains the following token classes, for example: cToken 2042; cTokenCreatorNoRef 2044; cTokenCreator 2046; CtokenCreatorRef 2048; cTokenCreatorBase 2050; and cTokenFactory 2054.
  • token classes are involved in the identification of tokens. Tokens turn into symbols after identification. Symbols are any class derived from cSymbol, e.g., cTemplate, cSection, etc.
  • XXL compiler 140 also contains the following symbol table classes, for example: cPluginSymbolTable 2058; cTemplateSymbolTable 2060; cSymbolTable 2062; cFFGSymbolTable 2064; cSGPSymbolTable 2066; and cSymbolTableBase 2068. These classes are varieties of symbol tables. There are different symbol tables for different groups of symbols. A group of symbols define a name space for the symbol. Common symbol table functions are located in the base symbol table classes and templates.
  • the item element is handled by a cltem class 2012 object.
  • the data element in the XXL definition is handled by a cData class 2004 object.
  • Item plugin 156 Plugin 150 will receive the source to compile from the cData class 2004 object, in this example, a multiChoice element.
  • cWrapXML class 2052 a wrapper class for XML DOM nodes, supports error handling.
  • cCustomAttributes class 2056 compiles the custom attributes XXL element.
  • cWrapPropertySet class 2070 is a wrapper class for a POLESS property storage.
  • FIG. 7 shows test driver 110, UAS 174, and the interfaces used by and between the test driver 110 and UAS 174 to deliver the test.
  • UAS 174 defines LLaunch2 interface 177, which is used by UAS 174 to initiate test events.
  • ILaunch2 interface 177 is an extension of ILaunch interface 178, which, in other embodiments of the present invention, is also used by UAS 174 to initiate test events.
  • UAS 174 also defines and implements additional interfaces, for example: LAppointment interface 176; IPrint interface 198; and ITransfer interface 199.
  • LAppointment interface 176 transfers examinee candidate information and appointment details from UAS 174 to test driver 110, as is illustrated by the dashed arrow connecting LAppointment interface 176 to test driver 110.
  • IPrint interface 198 allows UAS 174 to send print requests to printer 198 regarding reports, for example, score reports.
  • ITransfer interface 199 allows UAS 174 to request the transfer of information from candidate exam results file 180
  • Test driver 110 defines various interfaces to allow test driver 110 to communicate with different parts of computer-based test delivery system 100.
  • Test driver 110 includes, for example, ten COM interfaces 169 to communicate and transfer data with plugins 150.
  • the COM interfaces 169 are denoted in Figure 7 as follows, for example: LDisplay interface 169a; IHelm interface 169b; Iltem interface 169c; IUnitTimer interface 169d; ISelection interface 169e; .Navigate interface 169f; IScore interface 169g; IResults interface 169h; IReport interface 169i; and IPlugin interface 169j.
  • Test driver 110 and plugins 150 communicate and transfer data with exam resource file 120 using, for example, three LPersistResource interfaces 192: LPersistResourceStream interface 192a; IPersistResourceSet interface 192b; and LPersistResourceStore interface 192.
  • LPersistResource interfaces 192 are used by plugins 150 during compilation of exam source files 130 and are used by both test driver 110 and plugins 150 during delivery of the test.
  • XXL compiler 140 directs plugins 150 in which storage location of exam resource file 120 to store any information that plugins 150 have validated. Plugins 150 can then retrieve the stored information from exam resource file 150 during delivery of the test.
  • Other number of interfaces and different combination of functionality may alternatively be used.
  • IPersistResourceStream interface 192a saves the information, for example, as a stream of data.
  • a stream of data is simply a stream of bytes stored as a linear sequence.
  • IPersistResourceSet interface 192b saves the information, for example, as a set of data.
  • a set of data is preferably a name-value property pair. For example, the name of a particular property for an item is distractors and the value is the number of distractors required for that item.
  • IPersistResourceSet interface 192 allows the name-value property pair to be saved together in exam resource file 120.
  • LPersistResourceStore interface 192c saves the information, for example, in a directory format with storage areas.
  • the directory format allows other streams of data to be saved within the storage area and for sub-storages to be saved under the storage area.
  • LPersistlnstance interface 196 likewise, comprises, for example, three, different interfaces, for example: IPersistlnstanceStream interface 196a; LPersistlnstanceSet interface 196b; and
  • LPersistlnstanceStore interface 196c Examination state information is saved to exam instance file 170 as, for example, a stream of data, as a set of data, or as a storage element, depending on which of the three LPersistResource interfaces 192 is implemented.
  • IContainerNotify interface 200 allows a visible plugin to inform test driver 110, for example, that the plugin is displayed and ready for examinee interaction.
  • IContainerNotifyHelm interface 206 allows helm plugin 154 to request navigation from test driver 110 after receiving an input from the examinee to move to another section of the test.
  • EVIore interface 202 is used to convey whether the examinee has seen all content in a presentation. For example, a "more” button appears in place of the next button when the content exceeds the window length. When the examinee scrolls to the bottom, the "more” button disappears and is replaced with the "next" button.
  • Collection interface 204 is used by test driver 110 to hold any group entities, for example, categories and sections of the test. The remaining interfaces are, for example, Microsoft defined Active Document interfaces, used to implement OLE linking functions of test driver 110 and the visible plugins, display plugin 152, helm plugin 154, and item plugin 156.
  • IOlelnPlaceFrame interface 210 controls the container's top-level frame window, which involves allowing the container to insert its menu group into the composite menu, install the composite menu into the appropriate window frame, and remove the container's menu elements from the composite menu.
  • IOlelnPlaceFrame interface 210 sets and displays status text relevant to the end-place object.
  • IOlelnPlaceFrame interface 210 also enables or disables the frames modeless dialogue boxes, and translates accelerator key strokes intended for the container's frame.
  • IOlelnPlaceUI window interface 21 1 is implemented by container applications and used by object applications to negotiate boarder space on the document or frame window.
  • the container provides a RECT structure in which the object can place toolbars and other similar controls, determine if tools can in fact be installed around the objects' window frame, allocates space for the boarder, and establishes a communication channel between the object and each frame and document window.
  • IAdviseSync interface 212 enables containers and other objects to receive notifications of data changes, view changes, and compound-document changes occurring in objects of interest. Container applications, for example, require such notifications to keep cached presentations of their linked and embedded objects up-to-date. Calls to IAdviseSync interface 212 methods are a synchronous, so the call is sent and then the next instruction is executed without waiting for the calls return.
  • lOleWindow interface 213 provides methods that allow an application to obtain the handle to the various windows that participate in-place activation, and also to enter and exit context-sensitive help mode.
  • IOlelnPlaceSite interface 214 manages interaction between the container and the objects in-place client site.
  • the client site is the display site for embedded objects, and provides position and conceptual information about the object.
  • IOleClientSite interface 215 is the primary means by which an embedded object obtains information about the location and extent of its display site, its moniker, its user interface, and other resources provided by its container.
  • Test driver 110 called IOleClientSite interface 215 to request services from the container.
  • a container must provide one instance of IOleClientSite interface 215 for every compound-document it contains.
  • IOleDocumentSite interface 216 enables a document that has been implemented as a document object to bypass the normal activation sequence for in-place-active objects and to directly instruct its client site to activate it as a document object.
  • a client site with this ability is called a "document site”.
  • Figures 8A and 8B illustrate the main classes of test driver 110 and the interfaces between test driver 110 and plugins 150. Also shown are the classes that interface to UAS 174. ITransfer interface 199, IPrint interface 198, LLaunch2 interface 177, and lAppointment interface 176 represent the connections from test driver 110 to UAS 174, as described previously. Some of the lines depicted in Figure 8 are solid and some are dashed. The solid lines, for example, between IcResults interface 240 and cEvent class 252, represent inheritance. The dashed lines, for example, between IExam interface 222 and IPlugin interface 169j, represent instantiation.
  • Inheritance, or generalization relates to a generalized relationship between classes that shows that the subclass shares the structure or behavior defined in one or more superclasses.
  • a generalized relationship is a solid line with an arrowhead pointing to the superclass.
  • Instantiation, or dependency represents a relationship between two classes, or between a class and an interface, to show that the client class depends on the supplier class/interface to provide certain services.
  • the arrowhead points to the supplier class/interface.
  • Some services from a supplier class to a client class include, for example: the client class access a value (constant or variable) defined in the supplier class/interface; methods of the cline class invoke methods of the supplier class/interface; and methods of the client class have signatures whose return class or arguments are instances of the supplier class/interface.
  • the cardinality of the relationship is illustrated in Figure 8 if the relationship represents containment. Cardinality specifies how many instances of one class may be associated with a single instance of another class. Cardinality can be shown for relationships to indicate the number of links allowed between one instance of a class and the instances of another class.
  • Test driver 110 also has several interfaces and implementing classes.
  • Test driver 1 10 interfaces include, for example: LExam interface 222; LMsgBox interface 224; ICategory interface 232; LForm interface 238; IcResults interface 240; IcReport interface 242; IScript interface 246; ISection interface
  • Icltem interface 256 The classes that implement the main interfaces include, for example: cScreenMinimum class 226; cFormGroup class 228; cPlugin class 230; cArea class 234; cTemplate class 236; cActivePlugin class 250; and cEvent class 252.
  • the interfaces that are prefaced by "lc" have names that already exist for plugins 150 to enact, for example, item plugin 156 implements LLtem interface 169c.
  • Icltem interface 256 is the interface implemented by test driver 110 class cltem (not shown). Of course, any number of interfaces may be used, depending on the necessary functionality.
  • the core class cExam (not shown) implements LLaunch2 interface 177 so that UAS 174 can control test driver 110.
  • the appointment object which implements LAppointment interface 176, is the main object UAS 174 supplies to test driver 110.
  • the appointment object is available to plugins 150 by way of IPlugin interface 169j. Furthermore, all plugins 150 get access to the test (Iexam) using the IPlugin interface 169, also.
  • the cExa class selects and delivers the form, using cFormGroup class 228 and LForm interface 238.
  • the form delivers results using IcResults interface 240, reports using IcReport interface 242, and sections contained with in the test using ISection interface 250.
  • Classes that are in the test delivery chain preferably derive from cEvent class 252.
  • the cResults class (not shown) delivers a results plugin 166 that implements IResult interface 169i.
  • the cReport class (not shown) delivers a report plugin 168 that implements IReport interface 169h.
  • the cSection, cGroup, and cForm classes (not shown) use several invisible plugins 150 to control the delivery of the test. These plugins 150 are timer plugins 158, which implement IUnitTimer interface 169d, selection plugins 160, which implement ISelection interface 169e, scoring plugins 164, which implement IScore interface 169g, and navigation plugins 162, which implement LNavigate interface 169f.
  • the cPresentation class (not shown) supplies data to its template for the display of the presentation.
  • the three visible plugins 150 are created and controlled through cTemplate class 236 and child objects cArea class 234.
  • Item plugins 156 have an extension class in the cltem class (not shown) that wraps the item plugin 156 and provides generic extended services that all item plugins 156 implements.
  • the cltem class in test driver 110 is a wrapper class.
  • the cltem class provides two base services, for example: generic item functionality and access to item plugin 156, which is the wrapping function.
  • Item generic functionality includes, for example: having an item name, having an item title, determining if the item is scored or un-scored, determining whether the item has been presented to the examinee, etc. These services are generic to all items and are provided by test driver 110.
  • Item plugins 156 perform the actual scoring of the item, which is unique to each item type. Item plugins 156 present the content of the item and allow the examinee to interact with the item. These services are unique to each item type.
  • test driver 1 10 implements IRegistry interface 220, which allows VB code to access the Windows registry.
  • Test driver 1 10 also implements LLegacyltem interface 258 and LLegacyScore interface 260, which are defined by test driver 1 10 and are implements by certain item plugins 156 and scoring plugins 164.
  • LLegacyScore interface 260 allow old item types that existed in previous test drivers to report results like the previous test drivers. For some tests, test driver 110 must report results for old item types, which had very specific ways of reporting results. ILegacyltem interface 258 and ILegacyScore interface 260 allow the new item plugins 156 that represent old item types to report this legacy format of information to result plugins 166 trying to imitate previous test drivers.
  • test driver 110 classes and interfaces are included in Appendix A.
  • V. POLESS All persistent storages, exam resource file 120 and exam instance file 170, preferably utilize
  • POLESS allows data to be embedded, linked, or references as external files from the persistent storage to test driver 110 and Active Document container application 112 ( Figure 3).
  • POLESS supports a hierarchical tree structure with node or branch level additions, replacements, and deletions.
  • POLESS also supports optional data encryption at the node level. The type of encryption employed depends on the destination of the information in the persistent storage. For example, different encryption keys may optionally be used for data being routed to test centers, data being routed to administrative data centers, and data being routed for client use (e.g., client review).
  • Microsoft Crypto-API is preferably used to perform encryption of data in the persistent storage.
  • POLESS also supports optional compression at the node level, preferably using Lempal-Zev compression.
  • POLESS is an extension of OLE structured storage compound document implementation.
  • a compound document is a single document that contains a combination of data structures such as text, graphics, spreadsheets, sound and video clips. The document may embed the additional data types or reference external files by pointers of some kind.
  • Structured storage provides file and data persistence by treating a single file as a structured collection of objects known as storage elements and streams. Another benefit is incremental access. If test driver 110 or plugins 150 need access to an object within a compound file, only that particular object need be loaded and saved, rather than the entire file. Additionally, structure storage supports transaction processing. Test driver 1 10 or plugins 150 can read or write to compound files in transacted mode, where changes made can subsequently be committed or reverted.
  • FIG 9 shows the major components that support POLESS and the interfaces that connect the components.
  • POLESS 300 may be either exam resource file 120 or exam instance file 170.
  • POLESS 300 utilizes PKware library component 330 for storage compression and decompression.
  • POLESS 300 uses Crypto API component 332, a Microsoft application, for storage encryption and decryption.
  • Crypto API component 332 relies on a crypto service provided ("CSP") 334 to perform the actual encryption algorithms. Access to the services of these components is facilitated by standard (API) interfaces exposed by these components.
  • CSP crypto service provided
  • OLE2SS component 310 contains all the interface definition that makeup structure storage.
  • interfaces can be realized by any structured storage implementation, such as compound document implementation OLE2 320 and POLESS 300.
  • the interfaces include, for example: IStream interface 340; ISequential Stream interface 342; IStorage interface 344; and IRootstorage interface 346.
  • POLESS 300 additionally implements IStreamVB interface 348 and IStorageVB interface 350.
  • IStreamVB interface 348 supports several functions, for example: ReadVB(); WriteVB();
  • Clear(); Reset(); get_sName(); get_oStream(); and CopyTo() ReadVB() reads a specified number of bytes to a data array. WriteVB() writes the byte data to the stream. Clear() clears the stream of all data. Reset() sets position to the beginning of the stream. get_sName() is a read-only function that returns the name of the stream. get_oStream() is a read-only function that returns the IStream interface 348. CopyTo() copies a source stream to a destination stream.
  • IStorageVB interface 350 supports several functions, for example: Clear(); CommittVB(); RevertVBO; sElementName(); bStorage(); oElement(); CreateStream(); OpenStream();
  • Clear() clears the storage of all elements.
  • CommittVB() causes transacted mode changes to be reflected in the parent.
  • RevertVB() discards changes made since the last commit.
  • sElementName() returns the name of the element.
  • bStorage() returns TRUE if the element is a sub-storage.
  • oElement() returs IStreamVB interface 348 or IStorage interface VB 350 for the element.
  • CreateStream() creates and opens a stream and returns IStreamVB interface 348.
  • OpenStreamO opens a stream and returns IStreamVB interface 348.
  • CreateStorage() creates and opens a nested storage and returns IStreamVB interface 348.
  • OpenStorage() opens an existing storage and returns IStreamVB interface 348.
  • get_sName() is a read-only function that returns the name of the storage.
  • get_oStorage() is a read-only function that returns IStorage interface 350.
  • get_nCount() is a read-only function that returns a count of the elements.
  • GetCompression() returns the status of file compression.
  • GetEncryptionO returns the status of file encryption.
  • GetCRC() returns the status of file CRC checking.
  • CreateStreamLinked() creates and opens a linked stream and returns IStreamVB interface 348.
  • CreatePropertyStgO creates and opens a property storage and returns IpropertyStorageVB interface 414.
  • OpenPropertyStgO opens a property storage and returns IpropertyStorageVB interface 414.
  • SetClassO sets the CLSLD for the storage.
  • RegisterAlias() registers a pluggable protocol.
  • DestroyO destroys the specified elements.
  • getJElementType() is a read-only function that returns the type of the element.
  • FIGS 10A and 10B illustrate the main class of POLESS 300, the interfaces used to implement the classes, and the flow of the creation of streams 424 and storages 426.
  • cFileRoot class 400 is the first object instantiated and is used to create a new or open an existing a POLESS file.
  • cStorageRoot class 406 is returned, which is a slightly overloaded version of cStorage class 410. From cStorageRoot class 406 creates or opens cStream class 408 and cStorage class 410, from which any streams or storages and sub-storages of those can be created or opened, respectively.
  • cStorage class 410 creates cPropertyStorage class 412, which creates storage for property sets.
  • the classes implement interfaces that perform operations and/or define attributes that further define the function or properties of the class.
  • a complete description of POLESS 300 classes and interfaces is included in Appendix B.
  • cFileRoot Class 400 is the root POLESS class and controls the creation and opening of all POLESS files. cFileRoot class 400 is generally instantiated first before any other POLESS objects can be created, although other sequences are possible. cFileRoot class 400 implements LFileRoot interface 401 , which is collocated in Figure 10 with cFileRoot class 400. LFileRoot interface 401 is used to open one file at a time and is not released until all other storage object 426, stream object 424, and property storage interfaces are released and the file is ready to be closed.
  • cFileRoot class 400 and IRoot interface support the following operations, for example: StorageFileCreate(); StorageFileOpen(); CryptoGetQ; bStorageFile(); StorageAmalgamatedGet(); DeltaFileCreate();DeltaFileApply();
  • StorageFileCreateO creates a new storage file, returns the root storage to interface, marks the new structured storage file as a POLESS file by storing the class ID ("CLSID") of this class in a stream in the root storage.
  • StorageFileOpen() opens an existing storage file and returns the root storage interface.
  • CryptoGet() gets a default configured crypto class and should be set and used on the open or create of the storage file.
  • bStorageFile() returns true if the file provided is an OLLE structured storage file and not a POLESS storage file.
  • StorageAmalgamatedGet() gets an empty small cStorageAmalgamated class 404.
  • DeltaFileCreate() creates a POLESS difference file by comparing the original POLESS file to the updated POLESS file.
  • DeltaFileApplyO applies a POLESS delta file and applies the original POLESS file to the delta file to create an updated POLESS file.
  • GetObjectFromPathO uses monikers to retrieve the object named by the path and returns a pointer to the object retrieved.
  • CreateStreamFromFile() creates a structured storage stream and populates it with the contents of the file.
  • CreateStreamFromBSTR() creates a structures storage stream and fills it with the specified string.
  • MemoryStreamFromStream() is used to copy a stream to a newly created memory stream object.
  • GetPicture() loads a picture from stream object 424. SavePicture() saves the picture into the stream 426.
  • cCrypto Class cCrypto class 402 controls the configuration of the encryption/decryption of POLESS 300.
  • cCrypto class 402 has the following attributes, for example: sProviderName; eProviderType; sContainerName; and sPassword.
  • SProviderName represents the name of CSP 334 being used to perform the encryption/decryption services.
  • eProviderType is the type of CSP 334.
  • the field of cryptography is large and growing. There are many different standard data formats and protocols. These are generally organized into groups or families, each of which has its own set of data formats and way of doing things.
  • Crypto API is designed so that a CSP provider type represents a particular family.
  • sContainerName is the key name and must be provided by cCrypto class 402.
  • sPassword is an optional password on the public/private key pair and can only be entered by a human operator. The password can be used for review disks and their resource files.
  • cCrypto class 402 implements ICrypto interface 401 and they support the following properties and method, for example: ProviderName; Password; FileType; Algorithm; EnumProviders(); and EnumAIgorithms().
  • Get_ProviderName() returns the name of the Crypto provider. Put_ProviderName() sets the name of the Crypto provider. Get_Password() and Put_Password() are only used for sponsor resource files. Get_FileType() gets the file type and put_FileType() sets the file type. Get_Algorithm() gets the encryption algorithm and put_Algorithm() sets the encryption algorithm. EnumProviders() returns an enumerator for the list of installed providers. EnumAlgorithmsO enumerate a list of algorithms for the current provider.
  • cStorageAmalgamated class 404 is an implementation of IStorage interface 344.
  • cStorageAmalgamated class 404 holds references to an ordered collection of IStorage objects.
  • cStorageAmagalmated class 404 searches the collection of storage objects in order to find the first storage object that has the requested stream and returns this stream.
  • cStorageAmalgamated class 404 handles compound storage resolution and delegates all other work to cStorage class 410.
  • cStorageAmalgamated class 404 is, for example, read-only. cStorageAmalgamated class 404 will not allow stream or storages to be created but is primarily for reading exam resource file 120.
  • cStorageAmalgamated class 404 implements IStorage Amalgamated interface 405.
  • cStorageAmalgamated class 404 and IStorageAmalgamated interface 405 support the following operations, for example: StorageAdd(); Clearstorage(); OpenStorageAmalgamated(); and OpenPropertyStgAmalgamated().
  • StorageAdd() adds a new storage to the collection of storages.
  • ClearstorageQ clears all the storage objects from the collection.
  • OpenStorageAmalgamated() opens a sub-storage of the current amalgamated storages in an amalgamated fashion.
  • OpenPropertyStgAmalgamatedO opens a property storage of the current amalgamated storages in an amalgamated fashion. Amalgamation is described in greater detail, in the co-pending application filed on the same date, entitled "EXTENSLBLE EXAM LANGUAGE (XXL) PROTOCOL FOR COMPUTER BASED TESTING,” incorporated herein by reference.
  • cStorageRoot Class cStorageRoot class 406 is the POLESS implementation of IStorage interface 344 and IRootstorage interface 346. cStorageRoot class 406 handles any storage object 426 that is POLESS specific and then delegates work to the cStorage class 410. IRootstorage interface 346 supports the SwitchToFile() operation, which copies the current file associated with the storage object to a new file, which is then used for the storage object and any uncommitted changes. cStorageRoot class 406 also implements IPersistFile interface 418, which provides methods that permit an object to be loaded from or saved to a disk file, rather than a storage object or stream.
  • IPersistFile :Load on the object preferably also open its disk file.
  • IPersistFile interface 418 inherits its definition from IPersist, so all implementations must also include the GetClassID() method of IPersist interface 418.
  • cStream Class cStream class 408 is the POLESS implementation of IStream interface 340.
  • cStream class 408 handles any storage object 426 that is POLESS specific and then delegates work to compound document implementation OLE2 320. The specific work includes compression/decompression and encryption/decryption of stream object 424.
  • IStream interface 340 supports the following operations, for example: Seek(); SetSize();CopyTo(); Committ(); Revert(); LockRegion(); UnlockRegion(); Stat(); and Clone().
  • Seek() changes the seek pointer to a new location relative to the beginning of stream object 424, the end of stream object 424, or the current seek pointer.
  • SetSize() changes the size of stream object 424.
  • CopyTo() Copies a specified number of bytes from the current seek pointer in stream object 424 to the current seek pointer in another stream object 424.
  • Commit() ensures that any changes made to a stream object 424 open in transacted mode are reflected in the parent storage object.
  • Revert() discards all changes that have been made to a transacted stream since the last call to IStream: .Commit.
  • LockRegionO restricts access to a specified range of bytes in stream object 424. Supporting this functionality is optional since some file systems do not provide this operation.
  • UnlockRegion() removes the access restriction on a range of bytes previously restricted with IStream: :LockRegion.
  • Stat() retrieves the STATSTG structure for the stream object 424.
  • Clone() creates a new stream object that references the same bytes as the original stream but provides a separate seek pointer to those bytes.
  • IStreamVB interface 348 is an automation friendly version of IStream interface 340.
  • IStreamVB interface 348 supports the following operations, for example: Read(); WriteQ; ClearQ; Reset(); get_sName(); get_oStream; and CopyTo().
  • Read() reads data from stream object 424.
  • Write() writes data, including the entire byte array, to stream object 424.
  • Clear() clears stream object 424 of all data.
  • Reset() resets the position in stream object 424 to the beginning of stream object 424.
  • Get_sName() returns the name of the stream.
  • Get_oStream() returns the IDispatch interface.
  • CopyTo() copies the contents of a source stream to a destination stream.
  • cStorage Class cStorage class 410 is the POLESS implementation of IStorage interface 344 and IcStorage interface 41 1. cStorage class 410 handles any storage object 426 that is POLESS specific and then delegates work to compound document implementation OLE2 320. IStorage interface 344 supports the following operations, for example: CreateStream();
  • CreateStream() creates and opens a stream object 424 with the specified name contained in a storage object.
  • OpenStreamO opens an existing stream object 424 within a storage object using specified access permissions.
  • CreateStorage() creates and opens a new stream object 424 within a storage object.
  • OpenStorage() opens an existing storage object 426 with the specified name according to the specified access mode.
  • CopyTo() copies the entire contents of an open storage object 426 into another storage object.
  • the layout of the destination storage object may differ from the layout of the source storage object. MoveElementTo() copies or moves a sub-storage or stream object 424 from one storage object 426 to another storage object.
  • CommitO reflects changes for a transacted storage object 426 to the parent level.
  • Revert() discards all changes that have been made to the storage object 426 since the last IStorage::Commit operation.
  • EnumElementsO returns an enumerator object that can be used to enumerate storage objects 426 and stream objects 424 contained within a storage object.
  • DestroyElement() removes the specified storage object 426 or stream object 424 from a storage object.
  • RenameElement() renames the specified storage object 426 or stream object 424 in a storage object.
  • SetElementTimes() sets the modification, access, and creation times of the indicated storage element, if supported by the underlying file system.
  • SetClassO assigns the specified CLSID to a storage object.
  • SetStateBits() stores state information in a storage object, for example up to 32 bits.
  • Stat() returns the STATSTG structure for an open storage object.
  • IStorageVB interface 350 is an automation friendly version of IStorage interface 344.
  • IStorageVB interface 350 supports the following operations, for example: Clear(); Commit(); Revert(); sElementNameO; bstorage(); bElement(); CreateStream(); OpenStreamO; Createstorage();Openstorage(); get_sName(); getoStorage(); get_nCount(); GetCompression(); GetEncryptionO; GetCRC(); CreateStreamLinked(); CreatePropertyStg(); OpenPropertyStgO; SetClassO; Register Alias(); DestroyO; and get_ElementType().
  • Clear clears the storage of all elements, e.g. sub-storages and streams.
  • Commit() ensures that any changes made to a storage object opened in transacted mode are reflected in the parent storage. For non-root storage objects in direct mode, this method has no effect. For a root storage, it reflects the changes in the actual device, for example, a file on disk. For a root storage object open in direct mode, the commit() method is always called prior to releasing the object. Commit() flushes all memory buffers to the disk for a root storage in direct mode and will return an error code upon failure. Although releasing the object also flushes memory buffers to disk, it has no capacity to return any error codes upon failure. Therefore, calling releasing without first calling commit() causes indeterminate results. Revert() discards all changes that have been made to the storage object since the last Commit() operation.
  • sElement() returns the name of the element.
  • bstorage() returns true if the element is a sub- storage.
  • bElement() returns either iStreamVB interface 412 or iStreamVB interface 414 or IStorageVB interface 412 for the selected element.
  • CreateStream() creates and opens a stream object with the specified name contained in the storage object. None is returned if the stream cannot be created.
  • OpenStreamO opens an existing stream object within this storage object in the specified access mode. Nothing is returned if the stream cannot be opened.
  • Createstorage() creates and opens a new storage object nested within the storage object. Nothing is returned if the storage cannot be created.
  • Openstorage() opens an existing storage object with a specified name in the specified access mode. None is returned if the storage cannot be opened.
  • Get_sName() returns the name of the storage.
  • Get_oStorage() returns the IDispatch interface, which exposes objects, methods and properties to programming tools and other applications that support Automation. COM components implement the IDispatch interface to enable access by Automation clients, such as Visual Basic.
  • Get_nCount() returns the count of elements in the storage.
  • GetCompression() determines if streams may be compressed in the file and if enabled streams may optionally be compressed when created.
  • GetCRC() indicates whether a cyclic-redundancy-check ("CRC"), or a digital signature, check is to be performed on the file.
  • CreateStreamLinked() creates a link to a stream in another POLESS file.
  • CreatePropertyStgO creates a property storage. OpenPropertyStgO opens a property storage.
  • SetClassO assigns the specified CLSID to a storage object.
  • Register Alias() registers an alias to a storage in the POLESS file for access by the pluggable protocol.
  • DestroyO destroys the specified element.
  • Get_ElementType() is a read-only command that returns the type of the element.
  • cPropertyStorage class 412 implements LPropertyStorage interface 413, which supports the following operations, for example: ReadMultiple(); WriteMultiple(); DeleteMultiple();
  • ReadMultiple() reads property values in a property set.
  • WriteMultipleO writes property values in a property set.
  • DeleteMultiple() deletes property values in a property set.
  • ReadPropertyNamesO gets corresponding strung names fro given property identifiers.
  • WritePropertyNamesO creates or changes string names corresponding to given property identifiers.
  • DeletePropertyNamesO deletes string names for given property identifiers.
  • SetClassO assigns a CLSID to a property set.
  • Commit() flushes or commits changes to a property storage object, as is done with the command IStorage: .Commit, described previously.
  • Revert() discards all changes made since the last commit call when a property storage is opened in transacted mode.
  • Enum() creates and gets a pointer to an enumerator for properties within a property set.
  • Stat() receives statistics about a property set.
  • SetTimes() sets modification, creation, and access times for a property set.
  • LPropertyStorageVB interface 414 is an automation friendly version of LPropertyStorage interface 413 that manages the persistent properties of a single property set.
  • IPropertyStrorageVB interface 414 supports the following operations, for example: ReadVB(); Write VB(); Delete();
  • ReadVB() reads the value of a specified property from the property set.
  • Write VB() writes a value for a specified property to the property set. If the property does not exist the property/value pair will be created. If the property already exists, the value will be updated if opened in eAccess_Write mode.
  • Delete() removes a property from the property set.
  • SetClassO assigns the specified CLSLD to a property storage object.
  • Get_nCount() returns the count of properties in the property set. CopyTo() copies the contents of the source property set to a destination property set.
  • GetName() returns the name of the specified property.
  • WriteMultiple() writes property values in a property set.
  • ReadMultiple() reads property values in a property set.
  • cPropertyStorageAmalgamated class 416 implements IPropertyStorageAmalgamated interface 417, which supports the following operations, for example: PropertyStorageAdd() and ClearStorage().
  • PropertyStorageAdd() adds a property set to the collection of property sets.
  • ClearStorage() clears the collection of property sets.
  • FIGS G and H1-H14 illustrate the POLESS layout of exam resource file 120 according to the present invention.
  • Exam resource file 120 stores the various pieces of compiled information from exam source files 130, as shown in Figure 5.
  • Exam resource file 120 contains all of the content required to deliver the test. However, where the test is media-intense, exam resource file 120 will contain the core elements for the test with "links" to the external content.
  • XXL compiler 140 and plugins 150 store the compiled information to exam instance file 120 using one of LPersistResourceStream interface 192a, IPersistResourceSet interface 192b, or LPersistResourceStore interface 192 to store the compiled information as a stream of data, a set of data, or a storage element, respectively.
  • the layout of exam resource file 120 is in a hierarchical POLESS format that directly implements the format of the XXL test definition language.
  • the test developer uses the XXL test definition language to create the logic files 230 and data files 212 ( Figure 5) of exam source file 130.
  • XXL compiler 140 determines the storage location in exam resource file 120 that stores a particular piece of compiled information, even information stored into exam resource file 120 by one of plugins 150.
  • Figure 11 illustrates the main storage branches of exam resource file 120, which corresponds to the top-level elements of the XXL test definition language, denoted by reference numeral 500.
  • the main storage branches of exam resource file 120 are, for example: exams branch 550; forms branch
  • Exam branch 550 stores, for example, the primary attributes, properties, and data that govern the test.
  • Exam branch 550 can store information for various tests, as is denoted by the three, vertical ellipses.
  • a specific test is identified by the data stored in name attribute storage 552.
  • the various tests may each be identified, for example, by a different name, as denoted by the solid border around name attribute storage 552 or other identification scheme.
  • Attributes storage 554 stores, for example, version information 555, and title information 556 of the test as a stream of data.
  • Title information 556 is optional, as is denoted by the broken border. Any optional, customized information regarding the test is stored in custom properties 558 as a property storage.
  • Information relating to the forms of the test are optionally stored in forms property storage 560.
  • a form is a fixed or substantially fixed order of testing events. Many different forms can be stored in forms storage 560, giving flexibility to test driver 110 in controlling progression of the test.
  • FormGroups storage 562 optionally stores information relating to a collection of exam forms as a stream of data. Preferably, a single form from the formGroup is chosen to deliver to an examinee. The slection of the form from the group is performed by a selection plugin 160.
  • Exam branch 550 preferably contains at least one forms storage 560 either independently or within formGroups storage 562. Other information relating to the test may be stored under exam branch 550. Other storage formats may optionally be used.
  • Forms branch 600 stores, for example, the primary attributes, properties, and data that govern the progress of the test.
  • Forms branch 600 can store information for various forms, as is denoted by the three, vertical ellipses. As described previously, a form is a fixed or substantially fixed order of testing events. A single form is identified by the data stored in name attribute storage 602. Other identification formats may optionally be used. Again, the various forms may each be identified by a different name, as denoted by the solid border around name attribute storage 602.
  • Attribute storage 604 stores, for example, begin section information 605, end section information 606, event information 607, and optionally stores version information 608, title information 609, skip allowed information 610, restartable information 611, with information 612, height information 613, and bit depth information 614. All information stored in attribute storage 604 is stored as a stream of data or other data storage format. Begin section information 605 and end section information 606 indicates, for example, respectively which section of the test begins and ends the test.
  • Event information 607 indicates, for example, the order of events of the test for that form. Each event has a name and is prefixed with an event type and a colon. Other formats are optional. The event type includes "section”, “report”, and “results”. Version information 608 and title information
  • Skip allowed information 610 indicates, for example, whether or not by default skipping of sections is allowed.
  • Restartable information 61 1 indicates, for example, whether the form can be restarted.
  • Any optional, customized information regarding the form is stored in custom storage 616 as a property set or other data storage format.
  • Timer storage 628 stores, for example, information relating to how the form is to be timed as a storage element.
  • Attributes storage 630 stores, for example, the names of Timer Plugin 158 to be used with the form.
  • Plugin data storage 632 and plugin data storage 633 store any data necessary for timer plugin 158 as a storage element and a stream of data, respectively.
  • Plugin data storage 632 and plug in data storage 633 are optional.
  • Scoring storage 634 stores, for example, information relating to the scoring of the form. Attributes storage 636 stores, for example, the name of scoring plugin 164 to be used with the form. Plugin data 638 and plugin data 639 optionally store any data needed for scoring Plugin 164 as a storage element and a stream of data respectively.
  • Items Branch 650 stores, for example, the primary attributes, properties, and data that govern the items, or test questions, to be delivered to the examinee during the test.
  • Items branch 650 can store information for various items, as is denoted by the three, vertical ellipses.
  • a single item is identified by the data stored in name attributes storage 652.
  • the various items may each be identified by a different name, as denoted by the solid border around name attributes storage 652.
  • Attributes storage 654 stores, for example, weight information 654, scored information 655, and optionally stores, for example, skip allowed information 656, title information 657, start information 658, finish information 659, and condition information 660.
  • Weight information 654 indicates, for example, a value used for judging and scoring the item.
  • Scored information 655 indicates, for example, whether or not the item is scored as opposed to whether the item is being used as an example. The default of scored information 655 is true.
  • Skip allowed information 656 indicates, for example, whether the examinee can skip the item without answering.
  • Start information 658 indicates, for example, script execution at the beginning of the item and finish information 659 indicates, for example, script execution at the end of the item.
  • Condition information 660 indicates, for example, whether or not there is a condition on the item being delivered to the examinee.
  • the information stored in attributes storage 654 is stored as a stream of data or other data storage format.
  • Data storage 662 and data stream 664 store any information regarding the properties of the item. For example, data storage 662 or data stream 664 can store the correct answer of a multiple choice item.
  • Data storage 662 and data stream 664 stored the information as a storage element and a stream of data respectively. Any optional, customized information regarding the item is stored in customs storage 666 as a stream of data or other data storage format.
  • Category storage 668 stores, for example, information relating to each category to which the item belongs.
  • the information stored in category storage 668 preferably and optionally is redundant, as category branch 700 stores, for example, all the items within the specific categories. The reason for the optional redundancy is so that test driver 110 can quickly look up the category of any item.
  • Category branch 700 stores, for example, the primary attributes, properties, and data that govern the test categories.
  • a test category provides a grouping mechanism, which is independent of delivery of the test, allowing for exotic reporting and scoring if necessary.
  • Category branch 700 is optional as denoted by the broken border.
  • Category branch 700 can store information for various categories, as is denoted by the three, vertical ellipses.
  • a single category is identified by the data stored in name attributes storage 702. Again, the various categories may each be identified by a different name, as denoted by the solid border around name attributes storage 702.
  • Attributes storage 704 stores, for example, complete information 705, duplicates information 706, contents information 707, and optionally stores, for example, description information 708.
  • Complete information 705 indicates, for example, whether or not every item in the category must appear within the category or within its subcategories.
  • Duplicates information 706 indicates, for example, whether the item can appear more than once within the category or within the subcategories.
  • Contents information 707 determines what can exist within a category.
  • Description information 708 is used within the category to contain a description of the category's contents.
  • Category storage 710 stores, for example, information relating to any subcategories under the category identified in name attribute storage 702.
  • Items storage 712 indicates, for example, any items that exist within the category.
  • Sections storage 714 contains information indicating what any sections that exist within the category.
  • Scoring storage 716 contains information relating to the scoring of the items within the category.
  • Attributes storage 718 stores, for example, the name of the scoring plugin to be used with the item.
  • Data storage 720 and data stream 722 contain the information needed to initialize scoring plugin 164.
  • Data storage 720 and data stream 722 store the information as a storage element and a stream of data respectively.
  • Templates branch 750 stores, for example, the primary attributes, properties, and data that govern the templates used in the test.
  • Template branch 750 can store information for various main templates, as is denoted by the three, vertical ellipses.
  • a single main template is identified by the data stored in name attributes storage 752. Again, the various templates may each be identified by a different name, as denoted by the solid border around name attributes storage 752.
  • Attributes storage 754 stores, for example, split information 756, order information 757, and optionally stores, for example, size information 759.
  • Split information 656 defines how a specific area within the template is to be split or separated, for example, either by rows or columns.
  • Size information 759 indicates, for example, possible values for describing the size of the template, for example, pixels, percentages, or html syntax.
  • Template storage 760 stores, for example, information relating to any sub-templates to be used under the templates specified by the information in name attributes storage 752. Sub-templates are identified by the information in name attributes storage 762. Many sub-templates 760 can exist as denoted by the three vertical ellipses.
  • Areas storage 764 indicates, for example, information relating to the areas used within the template denoted by the information in name attributes storage 752. Many areas may exist within a template as denoted by the three vertical ellipses. Each area is identified by the information stored in name attribute storage 766. Attribute storage 768 stores, for example, visible plugin name information 760, size information 770, and allow more information 771. Plugin name information 760 indicates, for example, the name of the visible plugin to be used with the area. Size information 770 indicates, for example, the size of the area, as for example a pixel value, a percentage value, or HTML syntax. Plugin data 772 and plugin data 774 store information relating to the visible plugin to be used in the area.
  • plugin data storage 772 or plugin data stream 774 stores, for example, the information as either a storage element or a stream of data, respectively. Other information may optionally be stored.
  • Section branch 800 stores, for example, the primary attributes, properties, and data that govern test sections. Test sections dictate the navigation and timing of groups of items as well as displays within the test. Sections branch 800 can store information for various sections, as is denoted by the three, vertical ellipses. A single section is identified by the data stored in name attribute storage 802. Again, the various sections may each be identified by a different name, as noted by the solid border around name attributes storage 802. Attributes storage 804 stores, for example, group information 805 and optionally stores, for example, title information 806, skip allowed information 807, start information 808, finish information 809, and condition information 810. Group information 805 indicates, for example, to which group of the test the section belongs.
  • Skip allowed information 807 indicates, for example, whether or not the items within the section may be skipped.
  • Start information 808 indicates, for example, script execution at the beginning of the section and finish information 809 indicates, for example, script execution at the end of the section.
  • Condition information 810 indicates, for example, any conditions that exist regarding the section. Any optional, customized information regarding this section is stored in custom property storage 812 as a stream of data or other data storage format. Custom attributes will be stored as a property set. The "key" for each attribute will be a string.
  • Timer storage 814 stores, for example, information regarding the timing of the section.
  • Attribute storage 816 stores, for example, information identifying timer plugin 158, which is to be used with a section.
  • Plugin data storage 818 and plugin data storage 820 stores, for example, data needed for timer plugin 158.
  • Plugin data storage 818 and plugin data storage 820 stores, for example, information as a storage element and a string of data or other acceptable format respectively.
  • Navigation storage 822 stores, for example, information relating to the delivery of presentations and groups within the section.
  • Attributes storage 824 stores, for example, information indicating which navigation plugin 162 is to be used with this section.
  • Plugin data storage 826 and plugin data stream 828 store information needed for the navigation plugin 162.
  • Plugin data storage 826 and plugin data stream 828 store the information as a storage element and a stream of data respectively.
  • Groups branch 850 as seen in
  • Figure 18 stores, for example, the primary attributes, properties, and data that govern the groups within the test.
  • a group determines the order of events within the test.
  • Groups branch 850 can store information for various groups, as is denoted by the three, vertical ellipses.
  • a single group is identified by the data store in name attributes storage 852.
  • the various groups may each be identified by a different name, as noted by the solid border around name attributes storage 852.
  • Attributes storage 854 stores, for example, type information 855, event information 856, title information 857, and reviewed name information 858.
  • Type information 855 indicates, for example, whether the group is either a "group holder" (group of presentations), or a "section holder” (group of sub-sections). These are mutually exclusive.
  • Event information 856 indicates, for example, the order of events within the test.
  • Review name information 858 indicates, for example, whether or not a presentation within the group is to be used as a review screen. Any optional, customized information regarding the group is stored in custom storage 860 as a stream of data or other data storage format.
  • Events storage 862 stores, for example, event information as is described in further detail in Figure 19.
  • Scoring storage 864 stores, for example, information relating to the scoring of items within the group.
  • Attributes storage 866 stores, for example, information indicating which scoring plugin 164 is to be used with the group.
  • Selection storage 872 stores, for example, information relating to the selection of items within the group.
  • Attributes storage 874 indicates, for example, which selection plugin 160 is to be used with the group.
  • FIGS 19A, 19B, 19C, and 19D illustrate the events sub-branch of groups branch 850 in greater detail in accordance with one embodiment of the invention.
  • events sub-branch 862 can store information for various events.
  • events sub-branch 862 is storing information in events name sub-branch 880, event name sub-branch 890, and event name sub-branch 897.
  • Attributes storage 881, in Figure 19B, under events name storage 880 stores, for example, type information 882, template information 883, and optionally stores title information 884, counted information 885, start information 886, finish information 887, and condition information 888.
  • Type information 882 indicates, for example, whether the event is an item or a display.
  • Template information 883 indicates, for example, which template is being used with the event.
  • Counted information 885 indicates, for example, whether a presentation should be included in the totals of presentations presented to the examinee in a section. Generally, presentations with items, or questions, are counted and introductory presentations are not counted.
  • Start information 886, finish information 887, and condition information 888 indicates, for example, start, finish, and conditional scripts respectively. Any optional, customized information regarding the event is stored in custom storage 889.
  • the "key" for each custom attribute will be a string.
  • event name storage 890 indicates, for example, a different event, which contains different attributes.
  • area information 891, in Figure 19B indicates, for example, which area is rendering the presentations content and item information 892 indicates, for example, the name of the associated item if the event is of the item type.
  • data storage indicates, for example, start, finish, and conditional scripts respectively. Any optional, customized information regarding the event is stored in custom storage 889.
  • the "key" for each custom attribute will be a string.
  • event name storage 890 indicates, for example, a different event, which contains different attributes.
  • area information 891, in Figure 19B indicates, for example, which area is rendering the presentations content and item information 892 indicates, for example, the name of the associated item if the event is of the
  • Plugins branch 900 stores, for example, the primary attributes, properties, and data that govern any plugins 150 used for the test.
  • Plugins branch 900 can store information for various plugins, as is denoted by the three, vertical ellipses.
  • a single plugin is identified by the data stored in name attribute storage 902.
  • a CLSID is stamped with the name of the plugin 150.
  • Attributes storage 904 stores, for example, information identifying the plugin 150 by a program LD.
  • Data storage 906 stores, for example, the data, for example, as either a stream, set of data, or as a storage element if plugin 150, respectively.
  • Data branch 950 stores, for example, any global data needed for the test.
  • Data stored optionally under data branch 950 may be stored as either a storage element or a stream of data as indicated by data storage 952 and data storage 954.
  • Data stored under data branch 950 may be directly used by a plugin 150 or the data may be resources (.gif, jpeg, .wab, .mpeg, etc.) used internally by a plugin 150.
  • FormGroups branch 1000 stores, for example, the primary attributes properties and data that govern the formGroups of the test.
  • FormGroups branch 1000 can store information for various formGroups, as is denoted by the three, vertical ellipses.
  • a single formGroup is identified by the data stored in name attributes storage 1002.
  • the various formGroups may each be identified by a different name, as denoted by the solid border around name attributes storage 1002.
  • Attributes storage 1004 stores, for example, information indicating which forms are to be used within the formGroup.
  • Selections storage 1006 stores, for example, information relating to the selection of items within the formGroup.
  • Attributes storage 1008 indicates, for example, which selection plugin 160 is to be used with the formGroup.
  • Plugin data storage 1010 and plugin data storage 1012 store any information needed for the selection plugin 160.
  • Attributes storage branch 1050 stores, for example, attribute information that is global to exam resource file 120. This includes the last execution state of XXL compiler 140 [sMode], the major [iXXLMaj or Version] and the minor version [iXXLMinorVersion] of the XXL language.
  • Scripts branch 1100 stores, for example, information relating to scripts used within the test.
  • Attributes storage 1 102 stores, for example, type information that specifies which type of language the script is in. For example, VB script of J script.
  • Scripts storage 1104 stores, for example, global scripts used within the test that may be referenced by the test driver.
  • MsgBox branch 1150 stores, for example, information relating to the size and content of any message boxes that may be delivered to the examinee during the test. Message boxes may be triggered by plugins 150 during the exam.
  • Exam instance file 170 stores, for example, information regarding the current examinee' s test.
  • Exam instance file 170 is created when a test starts for an examinee.
  • Exam instance file 170 is destroyed when the test successfully completes. If the examinee must restart her test due to some interruption, for example, a power failure, the state of the test is restored from Exam instance file 170.
  • the layout of exam instance file 170 is in a hierarchical POLESS format.
  • the top-level storage branches of exam instance file 170 from root 1200 are, for example: running branch 1202; contents branch 1310; and history branch 1320.
  • Root 1200 relates to POLESS cStorageRoot class 406 ( Figure 26), which instantiates exam instance file 170.
  • Running branch 1202 stores, for example, the state information of all running objects in test driver 110 and plugins 150.
  • Plugins 150 use one of LPersistlnstanceStream interface 196a,
  • IPersistlnstanceSet interface 196b or LPersisflnstanceStore interface 196c to store information to exam instance file 170 as a stream of data, a set of data, or a store of data, respectively.
  • Test driver 110 determines the storage location in exam instance file 170 that stores, for example, a particular piece of examination state information.
  • Exam sub-branch 1204 contains examination state information relating to the exam.
  • Contents storage 1206 stores, for example, exam status information 1207 and version information 1208.
  • Exam status information 1207 indicates, for example, the status of the exam, for example, initializing or terminating.
  • Template storage branch 1210 stores, for example, examination state information relating to templates running in the exam.
  • Name attribute storage 1212 stores, for example, count information 1214 and observed ever information 1215. Observed ever information 1215 indicates, for example, whether or not the template's content has ever been fully seen by the examinee.
  • Form storage branch 1216 contains information relating to the forms used within the exam.
  • Contents storage branch 1218 stores, for example, seconds information 1219, date start information 1220, date finish information 1221 , current section information 1222, and version information 1223.
  • Current section information 1222 indicates, for example, the current section being delivered to the examinee in the form.
  • Version information 1223 indicates, for example, the identification of the form.
  • Sections chosen storage branch 1224 stores, for example, information relating to sections in the form being delivered to the examinee.
  • Contents storage 1226 stores, for example, the names of the sections that have been or will be delivered to the examinee.
  • Name attribute storage 1228 indicates, for example, the name of a particular section.
  • Contents storage 1230 stores, for example, current child information 1231, seconds information 1232, date start information 1233, and date finish information 1234.
  • Navigation storage 1236 and navigation storage stores, for example, current child information 1231, seconds information 1232, date start information 1233, and date finish information 1234.
  • Navigation storage 1237 store the state information of navigation plugin 162.
  • Navigation storage 1236 stores, for example, the examination state information from navigation plugin 162 if navigation plugin 162 implements the LPersistlnterfaceSet 196b or LPersistlnterfaceStore 196c.
  • Navigation storage 1237 stores, for example, the information from navigation plugin 162 if navigation plugin 162 implements
  • IPersistlnterfaceStream 196a IPersistlnterfaceStream 196a.
  • Timers storage 1238 and timers storage 1239 store information from timer plugin 158.
  • Timer storage 1238 is used if timer plugin 158 implements IPersistlnterfaceSet 196b or LPersistlnterfaceStore 196c.
  • Timers storage 1239 is used if timer plugin 158 uses IPersistlnterfaceStream 196a.
  • Items chosen sub-branch storage 1240 stores, for example, information relating to items that have been or will be delivered to the examinee.
  • Contents storage branch 1242 stores, for example, the names and order of all the items that have been or will be delivered to the examinee.
  • Name attributes storage 1244 indicates, for example, the identification of a particular item.
  • Contents storage branch 1246 stores, for example, presented information 1244, complete information 1248, skipped information 1249, seconds information 1250, dehydrated information 1251, and observed ever information 1252.
  • Presented information 1247 indicates, for example, whether the item has ever been delivered to the examinee.
  • Completed information 1248 indicates, for example, whether or not the item has been completed.
  • Skipped information 1249 indicates, for example, whether the item has been skipped.
  • Item plugin storage 1254 and item plugin storage 1255 stores, for example, examination state information from item plugin 156.
  • Item plugin storage 1254 is used if item plugin 156 uses IPersistlnterfaceSet 196b or LPersistlnterfaceStore 196c.
  • Item plugin storage 1255 is used if item plugin 156 uses iPersist interface 196A.
  • item light storage 1256 exists only if the item was dehydrated (to save memory or when a section ends). The dehydrated item stores the data but actions on the data are no longer available until the item is re-hydrated.
  • Item light storage 1256 stores, for example, score candidate information 1257. Score minimum information 1258, score nominal information 1259, score maximum information 1260, complete information 1261, skipped information 1262, correct answer display 1263, response results 1264, and correct answer results 1266.
  • Timers storage 1268 and timers storage 1269 store information from timer plugin 158. Timer storage 1268, as seen in Figure 26B, is used if timer plugin 158 implements IPersistlnterfaceSet 196b or LPersistlnterfaceStore 196c.
  • Timers storage 1269 is used if timer plugin 158 uses IPersistlnterfaceStream 196a.
  • Score storage 1270 and Score storage 1271 store information from timer plugin 158.
  • Timer storage 1270 is used if timer plugin 158 implements LPersistlnterfaceSet 196b or LPersistlnterfaceStore 196c.
  • Score storage 1271 is used if timer plugin 158 uses IPersistlnterfaceStream 196a.
  • groups chosen sub-branch storage 1272 indicates, for example, which groups have been or will be delivered to the examinee.
  • Contents storage 1274 stores, for example, the names of the groups.
  • Name attributes storage 1276 indicates, for example, the name of a particular group.
  • Contents storage 1278 stores, for example, names of groups and the order of groups. Scoring storage
  • Scoring storage 1280 and scoring storage 1281 store examination state information from score plugin 164.
  • Scoring storage 1280 is used if score plugin 164 implements IPersistlnterfaceSet 196b or LPersistlnterfaceStore 196c.
  • Scoring storage information 1281 is used if score plugin 164 implements IPersistlnterfaceStream
  • Selection storage 1282 and selection storage 1283 store information from selection plugin 160.
  • Selection storage 1282 is used if selection plugin 160 implements IPersistlnterfaceSet 196b or
  • Selection storage 1283 is used if selection plugin 160 implements
  • Delivered storage 1284 stores, for example, an ordered list of groups chosen for delivery.
  • Delivered storage 1285 stores an ordered list of the sub-classes of the form, for example: sections, reports and results.
  • Presentations chosen storage sub-branch 1286 indicates, for example, any presentations that have been or will be delivered to the examinee.
  • Contents storage 1288 stores, for example, the names of the presentations.
  • Names storage sub-branch 1290 stores, for example, the name of the presentation.
  • Names storage 1290 also stores, for example, comment information 1291, marked information 1292, count information 1293, name information 1294, observed ever information 1295, name information 1296, and observed ever information 1297.
  • Name information 1294 and observed information 1295 relate to the name of the first presentation area stored under presentations chosen sub-branch 1286 and whether or not the presentation has ever been observed, and name information 1296 indicates, for example, the last presentation area that was delivered to the examinee and whether or not the presentation was ever observed.
  • Contents storage 1298 stores, for example, information leading to events.
  • Contents storage 1298 stores ready information 1299 ever checked information 1300, ever started information 1301, and ever finished information 1302.
  • Ready information 1299 indicates, for example, whether the event is ready to be delivered to the examinee.
  • Ever checked information 1300 indicates, for example, whether an event's conditional delivery script ever been checked. Preferably, the conditional delivery script is only checked once.
  • Ever started information 1301 indicates, for example, whether the event was ever started by the examinee.
  • Ever finished information 1302 indicates, for example, whether the event was completed by the examinee.
  • contents branch 1310 stores, for example, a property set containing information to identify the examination instance and the examination start count 1312.
  • the identifying information used is the examinee appointment identification 1311, the name 1313 of exam resource file 120, and the name 1314 of the specified form or group.
  • History branch 1320 is a single stream of chronological text messages that logs the history of the test. These text messages are used by staff at system headquarters to diagnose problems that occurred in the field. Each text message is prefixed with the date, time, and a level of severity, for example: information, warning, or error. Test driver 110 will filter the text messages to a level of diagnostics desired for test driver 110, such as determining errors in test driver 110 or detail history tracking, including general information.
  • Figure 27 illustrates the process for customizing test based on specific requirement from the client using plugins 150, denoted generally by reference numeral 1400.
  • the client presents the new requirements, for example, a new item type, to the test developer, step 1402.
  • the test developer then writes and XML schema to define the XXL test specification, step 1404.
  • the schema is subsequently used to validate the XXL test specification.
  • An example of the XXL schema is as follows:
  • Attribute type "initialReview”/>
  • ⁇ attribute type "markAllowed” />
  • ⁇ attribute type "incompleteEndAllowed”/>
  • the test developer next writes the appropriate plugin 150, in this example, item plugin 156.
  • the test developer also implements the IPlugin interface 167 and IPlugin interface and Iltem interfaces 169. Additionally, the test developer implements IPersistResource interface 192 ( Figure 3) to enable persistence of compiled test information from item plugin 156 to exam resource file 120. The test developer can optionally implement fPersistlnstance interface 196 ( Figure 3), step 1408, to enable persistence of examination state information from item plugin 156 to exam instance file 170. After the appropriate interfaces have been implemented, item plugin 156 is valid and operating. Finally, after the test is delivered to the examinee, the result processor accumulates results from the examinee, 1410.
  • Customization process 1400 only required the test developer to write one piece of software, item plugin
  • Figure 28 illustrates the life cycle of plugin 150 from test production to test delivery, denoted generally by reference numeral 1420.
  • Dashed vertical line 1422 divides the plugin life cycle 1420 into a test production cycle, to the left of dashed vertical line 1422, and a test delivery cycle, to the right of dashed vertical line 1422.
  • the test production cycle occurs only occasionally when new plugins 150 are developed to satisfy the requirements of a client.
  • the test delivery cycle occurs whenever the test is delivered to the examinee, for example, daily.
  • Exam source files 130 contain every aspect of the test as written by the test publisher.
  • XXL compiler 140 reads from XXL files 134 and interprets instructions that call for the use of a plugin 150.
  • Plugin 150 is identified in the XXL test definition language by both a name and a program identification ("prog ID").
  • program ID program identification
  • XXL compiler receives the prog ID from XXL files 134, XXL compiler knows that a plugin 150 is required to complete the compilation of exam source files 130.
  • An example of plugin 150 calls in XXL is as follows:
  • the ten plugins defined in the previous example represent eight different types of plugins 150. Not all of the possible types of plugins 150 are required to build any one test. Also, more than one plugin 150 is implemented for a specific type. In the above example, two navigation plugins 162 and two item plugins 156 are defined.
  • XXL compiler 140 reads information from exam source files 130 using IStream interface 340, iNode interface 1424, and IStreamVB interface 348. XXL compiler 140 instantiates the requested plugin 150 using, for example, the call CoCreatelnsance. CoCreatelnstance () creates a single, uninitialized object of the class associated with a specified CLSID, using a prog ID that has been converted into the CLSID.
  • XXL compiler 140 If the data referring to plugin 150 has been customized by the test developer, XXL compiler 140 may not recognize the new data. Therefore, XXL compiler 140 passes the data directly to plugin 150 and plugin 150 loads the data into a private memory (not shown).
  • the private memory is internal to plugin 150, and in another embodiment, the private memory is external to plugin
  • plugin 150 can then validate the data using the XXL schema. If the data is invalid, plugin 150 reports the error. In an alternative embodiment, plugin 150 can validate the data using an XML document type definition ("DTD").
  • DTD is a formal description in XML Declaration Syntax of a particular type of document. Similar to a schema, a DTD sets out what names are to be used to the different types of elements, where they may occur, and how they all fit together. However, the XXL schema is preferred for validation since schemas are easier to read than a DTD and are very flexible.
  • XXL compiler 140 prepares a POLESS storage object 300 in exam resource file 120 to which plugin 150 saves the data at a command from XXL compiler 140, in step II. As described previously, XXL compiler 140 determines where the data from plugin 150 is to be saved in exam resource file 120 and creates the appropriate storage location. The name, CLSLD, and data associated with plugin 150 is stored in plugins branch 900 in exam resource file 120 ( Figure 20). Plugin 150 implements IPersistResource interface 192 to store the data to exam resource file 120.
  • Data storage 906 stores, for example, the data as either a set of data or as a storage element if plugin 150 implements either IPersistResourceSet interface 192b or LPersistResourceStore interface 192c, respectively.
  • Data storage 908 stores, for example, the data as a stream of data if plugin 150 implements LPersistResourceStream interface 192a.
  • Plugin 150 can choose the format used to store the data into exam resource file 120. Steps I and II are repeated until exam source files 130 are completely compiled and exam resource file 120 is completely populated with the compiled test information.
  • Plugin compile sequence 1430 begins as XXL compiler 140 asks plugin 150 to validate the information from exam source files 130 that pertain to plugin 150 using IPlugin: :ValidateSource() call 1432, in step I.
  • Plugin 150 validates whether or not the data received from exam source files 140 is correctly formatted based on the XXL schema. If the data is not valid, plugin throws, for example, a structured COM error. Plugin 150 does not validate that all required source elements are present, but rather, that what is present is correctly formatted.
  • Step II contains two steps, indicated as step Ila and Lib.
  • XXL compiler 140 creates the appropriate storage element in exam resource file 120 using POLESS object 300.
  • the storage element type is determined based on the type of IPersistResource interface 192 that plugin 150 implements, for example: LPersistResourceStream interface 192a; IPersistResourceSet interface 192b; or LPersistResourceStore interface 192c.
  • XXL compiler 140 then calls IPersistResource* ::Save() call 1434 for the appropriate IPersistResource interface.
  • Plugin 150 saves the compiled information from exam source files 130 to exam resource file 120 through the POLESS object 300 passed by XXL compiler 140.
  • step lib XXL compiler 140 instructs plugin 150 to unload, or flush, its content using Unload() call 1436. As stated previously, steps I, Ila, and lib are repeated until all of exam source files 130 is compiled.
  • Step VI which is shown as steps Via and VIb, concerns amalgamation of exam resource file
  • Amalgamation enables data for a specific plugin to exist virtually as one storage location even if the data appears at different locations within the storage hierarchy. Amalgamation can be performed on exam resource file 120 if plugin 120 has implemented either IPersistResourceSet interface 192b or
  • LPersistResourceStore interface 192c which storing data to exam resource file 120.
  • XXL compiler 140 amalgamates one to three storage elements in exam resource file 120 and passes the amalgamated POLESS object to plugin 150 using LPersistResource*::ValidateResource() call 1438.
  • Plugin 150 determines whether or not the amalgamated POLESS object creates a complete and valid set.
  • Plugin 150 throws, for example, a structured COM error if the amalgamated POLESS object does not create a complete and valid set.
  • XXL compiler 140 instructs plugin 150 to unload, or flush, its content using Unload() call 1440. Steps Via and VIb are interspersed among steps I, Ila, and lib cycles and can also occur multiple times during the compilation of exam source files 130. Amalgamation will be explained below in greater detail.
  • test driver 1 10 reads the test specifications stored in exam resource file 120 through POLESS objects 300.
  • Test driver 110 reads information from exam resource file 120 through POLESS objects 300 in order to retrieve the encrypted, compressed, and structured elements within exam resource file 120.
  • the XXL test definition language calls a plugin 150 by a prog ID, as described previously, test driver 110 instantiates the plugin 150 that was called, in step III.
  • Test driver 1 10 provides the POLESS object 300 from exam resource file 120 and plugin 150 initializes itself from the POLESS object 300, for example, data storage 906 or data storage 908 stored under name attribute storage 902, using the appropriate IPersistResource interface 192.
  • the information loaded into plugin 150 is the same information as was stored into exam resource file 120 by plugin 150 during the test production cycle (step II). Since plugin 150 chose the storage format used to store the information into exam resource file 150, plugin 150 can always read the information from exam resource file 150, giving plugin 150 complete flexibility. Test driver 1 10 need not be able to read the information that is used by plugin 150. Therefore, any customizations to the test facilitated by plugin 150 does not require any changes to test driver 110. The test then progresses with plugin 150 enhancing the functionality of test driver 110 based on the new requirements from the client.
  • test driver 1 10 asks plugin 150 if plugin 150 is "dirty," meaning that plugin 150 is storing has some updated examination state information. For example, when the examinee selects distractor A on a multi-choice item, item plugin 156, in this case, becomes dirty. If plugin 150 is dirty, test driver 1 10 provides plugin 150 a POLESS object 300 in exam instance file 170 and plugin saves the examination state information to exam instance file 170 using
  • LPersistlnstance interface 196 in step LV.
  • item plugin 156 saves the examinee's answer to item plugin storage 1254 or to item plugin storage 1255 ( Figure 26).
  • Item storage 1254 stores, for example, the data as either a set of data or as a storage element if item plugin 156 implements either LPersistlnstanceSet interface 196b or LPersistlnstanceStore interface 196c, respectively.
  • Step V occurs if the test is interrupted, for example, because of a power failure, and the test needs to restart.
  • test driver 110 When test driver 110 is required to return to a particular operation state, test driver 110 reads the examination state information from exam instance file 170.
  • Plugin 150 is provided the storage object containing the state of plugin 150 as saved in step IV using LPersistlnstance interface 196.
  • item plugin 156 retrieves its state information from item plugin storage 1254 or for item plugin storage 1255. Plugin 150 is able to become operational from the retrieved state information, enabling a restart of the test from the point at which the test was interrupted.
  • Plugin delivery sequence 1520 begins, in step Ilia, when the current delivering presentation requests its template to activate with cTemplate: :Activate() call 1524.
  • Activate() call 1524 is activated when the examinee navigates on the test using a helm navigation control activated by helm plugin 154.
  • IContainerNotifyHelm interface 206 allows helm plugin 154 to request navigation from test driver 110. IContainerNotifyHelm interface 206 sends Activate() call 1524 to cTemplate class 236 in test driver 110 (see Figure 8).
  • cTemplate class 236 in test driver 110 uses IPlugin: :Load() call 1526 to set the core object references from test driver 110 into the plugin 150 being delivered.
  • the core object references include IContainerNotify interface 200, the cExam class (not shown), and the LAppointment interface 176, which passes information regarding the examinee and appointment to plugin 150.
  • Step V which is interspersed with step III, occurs only if the test is interrupted and plugin 150 loses state.
  • cTemplate class 236 in test driver 110 uses LPersistlnstance* ::Reload() call 1528 to call on the reload method of exam instance file 170.
  • Exam instance file 170 reloads plugin 150, through LPersistlnstance interface 192, for example, LPersistlnstanceSet 192b, with the state saved to the appropriate storage location in exam resource file 170 (see Figure 26).
  • Step IIIc is performed for both initial delivery of plugin 150 and during restart of the test, in conjunction with step V.
  • cTemplate class 236 in test driver 110 uses LPersistResource*: :Load() call 1530 to call on the load method of exam resource file 120.
  • Exam resource file 120 loads plugin 150, through IPersistResource interface 192, for example IPersistResourceSet interface 192b, with the test specification and content from the appropriate storage location in exam resource file 120.
  • Plugin 150 is loaded with test specification and content from exam resource file 120 when being initially delivered to the examinee.
  • Plugin 150 is also loaded with test specification and content from exam resource file 120 and with examination state information from exam instance file 170, as described above, when the test has been interrupted and plugin 150 must recover state.
  • cTemplate class 236 in test driver 1 10 uses,
  • I*::PresentationSta ⁇ ting() call 1532 (continued in Figure 30B) to inform visible plugin 150 that the presentation is starting, in step Hid.
  • I*::PresentationStarting() call 1532 is made to any visible plugin
  • LDisplay interface 169a LDisplay interface 169a
  • Iltem interface 169c or LHelm interface 169b.
  • an ILtem: :PresentationStarting() call is used for item plugin 156.
  • cTemplate class 236 then instruct visible plugins 150 to display using I01eObject::DoVerb(Show,...) command 1534, step ILIe.
  • IOleObject interface 1522 is the Active Document interface used to implement the Active Document presentation. (See Figure 3-1).
  • IOleObject interface 1522 is the combination of the Active Document interfaces described in conjunction with Figure 7.
  • test driver 110 awaits notification from each visible plugin 150 that the specific visible plugin 150 has successfully shown.
  • Visible plugins 150 call back to test driver 110 using IContainerNotify ::Activated() call 1536, step Illf (continued in Figure 30B). Now, the presentation is started and active such that the examinee can interact with the presentation.
  • the deactivation of the presentation begins with a request from the helm for navigation. For example, if the examinee has finished a question and wishes to move on to the next question on the next presentation, the examinee can choose the "NEXT" button on the helm.
  • the navigation request is sent from LHelm interface 169b, which receives the request from the examinee, to test driver 1 10 using IContainerNotifyHelm interface 206. As seen in Figure 30D, the request is made using IContainerNotifyHelm: :RequestMove() call 1538, step Illg. Test driver 110 then asks each item plugin 156 being used in the presentation template if the examinee is allowed to leave the current presentation and to proceed to the next presentation.
  • test driver 150 passes the navigation request to navigation plugin 162, which is an invisible plugin 150.
  • Test driver 110 passes the request using INavigate::RequestMove() call 1542, step ILIi.
  • Navigation plugin 162 determines the resultant location of the requested navigation. In Figure 30, for example, navigation plugin 162 determines the section of the test to which the examinee will proceed using ISection: :ChildNext() call 1544, step IIIj.
  • cTemplate class 236 in test driver 110 requests that visible plugins 150 hide from the Active Document using I01eObject::DoVerb(Hide,...) call 1548, step IL ⁇ 1.
  • cTemplate class 236 in test driver 110 informs visible plugins 150 that the current presentation is ending using I*::PresentationEnding() call 1550, step ILIm.
  • cTemplate informs helm plugin 154 that the current presentation is ending using the
  • IHel m PresentationEndingO cal 1.
  • Step IV which contains sub-steps LVa-c, is the process to save plugin state data to exam instance file 170.
  • Test driver 110 requests the "dirty" state of plugin 150 to determine whether plugin 150 is storing any state information that would be necessary if the test were to be interrupted.
  • Test driver 110 uses LPersistlnstance*: :IsDirty() call 1552 to make the request, step LVa.
  • LPersistlnstanceSet :IsDirty call 1552 if the state data is a property set. If plugin 150 is storing state data that is not already stored in exam instance file 170, _PersistInstance*::IsDirty() call
  • test driver 110 instructs plugin 150 to save the state data to exam instance file 170 in the POLESS object provided ( Figure 26) using LPersistLnstance*::Save() call
  • test driver 110 instructs plugins 150 to unload all objects using
  • the test publisher defines common plugin 150 test specification and common plugin 150 test content in early XXL elements in XXL files 134. These previous elements can be referenced by later XXL elements in XXL files 134.
  • XXL files 134 are initially compiled into exam resource file 120, the relational hierarchy of the XXL elements is maintained. However, through amalgamation, these separate elements are brought together, or amalgamated, into one, virtual storage location in exam resource file 120. The storage location is virtual because the storage location does not physically exist but acts identically to a physical storage for read-only purposes. If the later element defines nothing more then plugin 150 will receive just the common specification and content.
  • the later elements can define omitted non-common plugin 150 specification and contents. Further, the later elements can override previously defined common plugin 150 specification and contents with exception plugin 150 specification and contents. This process of new definition and overriding of specification and contents by later XXL elements is not limited to two stages. Multiple later stages are allowed.
  • plugin 150 uses the specification and contents to determine how and what to deliver.
  • the specification and contents used by plugin 150 is the amalgamation of all relevant XXL elements in exam resource file 120.
  • the amalgamation is invisible to plugin 150 and requires no special effort for plugin 150 during delivery.
  • the amalgamation occurs at delivery time and therefore common specification and content appear only once in the compiled exam resource file 120. Thus, the size of the exam resource file 120 is reduced.
  • the XXL language allows invisible plugins 150 (e.g., timer plugin 158, selection plugin 160, navigation plugin 162, scoring plugin 164, results plugin 166, and report plugin 168) two locations to define test specification and content. These two locations are the element storing identification and initialization data for plugin 150 stored under plugins branch 900 in exam resource file 120 and the storage location containing the information that defines the actual usage of plugin 150 on a unit of the test. Invisible plugins 150 are defined for usage in forms branch 600, sections branch 800, category branch 700, and groups branch 850 in exam resource file 120. Test driver 110 amalgamates these two levels during delivery of the test.
  • Data storage 906 under plugins branch 900 in exam resource file 120 stores, for example, plugin 150 specification and content.
  • Data storage 906 is the first data location for the amalgamation.
  • Data storage 908 which stores, for example, a stream of data, is not involved in amalgamation since amalgamation can only be performed on a set of data or a storage element.
  • the XXL source above defines three navigation plugins 162 for use in the test.
  • the first is named “controUedNav” and does not have any test specification or test content defined that is common to all "controUedNav” navigation plugins 162 for this particular test.
  • the second is named “controUedNav”
  • timer plugin timerStandard
  • groupRef name "gtestlet002 " /> ⁇ /section>
  • the XXL source above defines two navigation plugins 162 and two timer plugins 158 used to deliver the test.
  • the first navigation plugin 162 instance, "linearlntro”, which was defined in the previous example, is located in the section named "testletOOl”. Since no additional data is defined for "linearlntro" under “testletOOl", the section definition language does not override or extend the original common specification and content for "linearlntro".
  • the second navigation plugin 162 instance also using "linearlntro" is located in the section named "testlet002". Since "testlet002" does define data for "linearlntro", the new specification and content defined in the section definition language will be amalgamated with the common specification and content originally defined for "linearlntro". (See previous example.) The new specification and content does override the original value for "initialReview” and extends the original common specification by defining a value for "quitExamPrompt". Note, however, that although the initial and later values for "initialReview" are both "true", the later value does override the initial value. If the later value were different than the original value, the value for "initialReview" would change.
  • Figure 31 is a flow chart illustrating the overall process for amalgamation of invisible plugins, denoted generally by reference numeral 1549.
  • a first segment of the test specification and content, stored under plugins branch 900 is validated by the appropriate invisible plugin 150, step 1551.
  • a second segment of the test specification and content, stored in the storage location containing the information that defines the actual usage of the invisible plugin 150, is then validated by the same invisible plugin 150, step 1553.
  • the first segment and the second segment are then amalgamated, step 1555.
  • the amalgamated segment is validated by the invisible plugin 150 to determine whether the amalgamated segment forms a complete and valid set, step 1557.
  • the first segment and the second segment are reamalgamated, step 1558, and the reamalgamated segment is delivered to the examinee, step 1559.
  • the XXL language allows three locations for visible plugins 150 (e.g., display plugin 152, helm plugin 154, and item plugin 156) to define test specification and content. These first two locations are the element storing identification and initialization data for plugin 150 stored under plugins branch 900 in exam resource file 120 and data storage 772 under area storage 764 in templates branch 750 of exam resource file 120.
  • the third location is the events element 880, which is stored under groups branch 850, in the case of display plugins 152 or helm plugins 156. Alternatively, the third location is or data storage 662 under items branch 650, in the case of item plugins 156. Test driver 110 amalgamates these three locations during delivery by the test.
  • Data storage 772 under area storage 764 in templates branch 750 stores, for example, visible plugin 150 specification and content for use in a particular template.
  • Data storage 772 is the second data location for the amalgamation for visible plugins 150.
  • area elements in a template element note: “helmTitlebar” display plugin 152, "helmNextPrevious” helm plugin 154, and
  • itemMultiChoice item plugin 156 are previously defined in XXL that is not shown):
  • the XXL source above defines one template named "multiItem_Template”.
  • This template contains three areas, named “titlebar”, “item”, and “helm”, and each area references one visible plugin 150 utilized in that area.
  • the area named “titlebar” utilizes “helmTitlebar” display plugin 152.
  • the first and the third area define data in data storage 772 for display plugin 152, "helmTitlebar”, and helm plugin 156, "helmNextPrevious", respectively, to be amalgamated with the specification and content stored in data storage 906 under plugins branch 900 for "helmTitlebar" and
  • the second area accepts the common item plugin 156 level specification and content for "itemMultiChoice” without further refinement.
  • plugin 150 specification and content for item plugins 150 or display plugins 152 and helm plugins 156, respectively. These are the final data locations for the amalgamation of visible plugins
  • the XXL source above defines two items, "101 A” and "102A”. Both items use the same template “multiltem Template” and area “item”, both of which were defined in the previous example. Both items define a third level of data for the amalgamation. For example, the item named "101 A” defines two previously undefined attributes: “correctAnswer” and "URI”. Item “101 A” also overrides two previously defined attributes: maxResponses, minResponses and autoPrompt. (It should be noted that the data “maxResponses,” “minResponses,” and “autoprompt” are typically defined in the XXL language initializing a multi-choice item plugin 156 stored in data storage 906 under plugins branch 900.
  • Figure 32 is a flow chart illustrating the overall process for amalgamation of visible plugins, denoted generally by reference numeral 1560.
  • a first segment of the test specification and content, stored under plugins branch 900 is validated by the appropriate visible plugin 150, step 1561.
  • a second segment of the test specification and content, stored under area storage 764 in templates branch 750, is then validated by the same visible plugin 150, step 1562.
  • a third segment of the test specification and content stored either under groups branch 850 (for display plugins 152 or helm plugins 156) or under items branch 650 (for item plugins 156), are validated by the same visible plugin 150, step 1563.
  • the first segment, second segment, and third segment are then amalgamated, step 1564.
  • the amalgamated segment is validated by the visible plugin 150 to determine whether the amalgamated segment forms a complete and valid set, step 1565.
  • the first segment, second segment, and third segment are reamalgamated, step 1566, and the reamalgamated segment is delivered to the examinee, step 1567.
  • Figure 33 shows the flow chart for amalgamation process 400.
  • plugin 150 validates partial exam source in exam source files 130, step 1460.
  • plugin 150 validates the complete amalgamated test specification and content in exam resource file 130, step 1470.
  • test driver 110 and plugin 150 deliver the complete amalgamated test specification and content, step 1480.
  • Steps 1460 and 1470 generally occur during exam compilation and step 1480 generally occurs during delivery of the test to the examinee.
  • other sequences and/or functionality may be used and/or substituted in accordance with the functionality described herein.
  • plugins 150 implement generally either IPersistResourceSet interface 192b or LPersistResourceStore interface 192c.
  • POLESS performs the actual amalgamation on either a property set or a storage element in exam resource file 120.
  • IPersistResourceStream 192a cannot generally be amalgamated in one embodiment of the invention, although other amalgamation possibilities are possible.
  • Figure 34 illustrates the process for exam source validation, step 1460, in greater detail, in accordance with one embodiment.
  • Plugin 150 receives partial exam source from exam source files 140 through XXL compiler 140, step 1462.
  • XXL compiler asks plugin 150 to validate the partial exam source using IPlugin 150::ValidateSource() call 1432 ( Figure 29).
  • Plugin 150 determines whether or not the exam source is valid by comparing the exam source to the XXL schema, as previously described, step 1463. If the partial exam source is not valid, plugin 150 throws, for example, a structured COM error, step 1464, and the compilation is aborted.
  • Plugin 150 generally validates that what is present in the partial exam source is correctly formatted. Plugin 150 does not generally validate that all required source elements are present, only that what is present is legal. Plugin 150 also does not validate that attributes do not conflict. Conflict between elements cannot yet be determined since the amalgamated final view of plugin 150 data has not yet been established.
  • XXL compiler creates the appropriate storage element via
  • the storage element type is determined base on the IPersistResource interface 192 plugin 150 implements. If plugin 150 implements IPersistResourceSet interface 192b, the data will persist to exam resource file 120 as a property set. If plugin 150 implements LPersistResourceStore interface 192c, the data will persist to exam resource file 120 as a storage element. XXL compiler 140 then implements the LPersistResourceStore::Save() call or the LPersistResourceSet::Save() call, both calls being denoted by reference numeral 1434 in Figure 29. Plugin 150 saves the compiled content and specification to exam resource file 120 through the POLESS object passed, step 1468.
  • FIG 35 illustrates the process for complete amalgamated test specification and content validation, step 1470, in greater detail, in accordance with one embodiment.
  • step 1470 data stored in exam resource file 120 is amalgamated for validation.
  • the purpose of this step is the validation of the amalgamated vector of storage elements to provide a complete specification and content for plugin 150.
  • XXL compiler 140 calls for creation of an amalgamation storage vector, step 1472, using the calls cFileRoot::StorageAmalgamatedGet() or cFileRoot::PropertystorageAmalgamatedGet(). (See cRoot class 400 in Figure 10.)
  • XXL compiler 140 then adds storage elements or property sets to the amalgamation storage vector, step 1474.
  • XXL compiler 140 will amalgamate, for example, one to three storage elements or property sets depending on the type of plugin 150 at issue. As described above, for invisible plugins 150, XXL compiler 140 will amalgamate up to two storage elements or property sets. For visible plugins 150, XXL compiler 140 will amalgamate up to three storage elements or property sets. XXL compiler 140 uses the cStorageAmalgamated: :StorageAdd() call to add a storage element to a storage amalgamation and the cPropertyStorageAmaIgamated::PropertyStorageAdd() call to add an element to a property set amalgamation.
  • the data contained in the storage elements or property sets is amalgamated according to the rule of amalgamation.
  • the rule of amalgamation states that a later define element replaces an earlier defined element of the same name, as shown in the previous example of amalgamation for item plugin 156.
  • Alternative amalgamation rules may optionally be used.
  • XXL compiler 140 determines whether or not the last storage element or property set has been added to the amalgamation storage vector, step 1475. If the amalgamation is complete, XXL compiler 140 passes the amalgamated POLESS object stored in the amalgamation storage vector to the IPersistResourceStore::ValidateResource() call or the IPersistResourceSet: :ValidateResource() call, both calls being denoted by reference numeral 1438 in Figure 29. XXL compiler 140 queries the amalgamation for IStorage interface 344 or LPropertyStorage interface 413. The appropriate interface is passed to plugin 150 in the validation call. Plugin 150 uses this virtual storage or property set to perform the validation on the amalgamated POLESS object.
  • plugin 150 throws an error, for example, a structured COM error, if the combination of content does not create a complete and valid set.
  • plugin 150 determines what is required for a complete and valid set. Referring to the previous example of item "101 A", the associated item plugin
  • FIG 36 illustrates the process for delivery of amalgamated test specification and content for delivery, step 1480, in greater detail, in accordance with one embodiment.
  • the exam resource data in exam resource file 120 is amalgamated for delivery.
  • the purpose of this step is the delivery of the exam using the amalgamated plugin 150 specification and content.
  • Test driver 110 calls for creation of an amalgamation storage vector, step 1482, using the calls cFileRoot:: storageAmalgamatedGetO or cFileRoot:: PropertystorageAmalgamatedGet(). (See cRoot class 400 in Figure 10.) Test driver 110 then adds storage elements or property sets to the amalgamation storage vector, step 1484.
  • Test driver 1 10 will amalgamate one to three storage elements or property sets depending on the type of plugin 150 at issue. As described above, for invisible plugins 150, XXL compiler 140 will amalgamate up to two storage elements or property sets. For visible plugins 150, test driver 110 will amalgamate up to three storage elements or property sets. Test driver 110 uses the cStorageAmalgamated: :StorageAdd() call to add a storage element to a storage amalgamation and the cPropertyStorageAmalgamated: :PropertyStorageAdd() call to add an element to a property set amalgamation. (See cStorageAmalgamated class 404 and cPropertyStorageAmalgamated 416 in Figure 10.) The data contained in the storage elements or property sets is amalgamated according to the rule of amalgamation, as described previously.
  • Test driver 1 10 determines whether or not the last storage element or property set has been added to the amalgamation storage vector, step 1486. If the amalgamation is complete, test driver 110 passes the amalgamated POLESS object to the LPersistResourceStore: :Load() call or the IPersistResourceSet: :Load() call, step 1488 (see Figure N). Plugin 150 can safely assume a full set of data in the amalgamated POLESS object because the amalgamated data was validated in step 1470 ( Figure 35). Test driver 110 queries the amalgamation for IStorage interface 344 or LPropertyStorage interface 413.
  • plugin 150 uses this virtual storage element or property set to enable delivery of the test, never realizing that the POLESS object is an amalgamation.
  • Figure 37 illustrates three examples of amalgamation of property sets.
  • Property set 1500 contains "prompt” information for an item plugin 156.
  • "prompt” is assigned the value “select best” in data storage 906 under plugins branch 900 in exam resource file 120.
  • template A and template B are defined, which use item plugin 156.
  • Template A is used for item “101 A” and template B is used for items “102A” and "103A”.
  • Property set 1502 contains "min” information for item plugin 156 regarding item "101A”. In the level 2 specification for item “101A”, “min” is assigned the value “1” in data storage 772 under templates branch 750 in exam resource file 120. Continuing with item “101A”, property set 1504 contains "correctAnswer” and "URI” information for item plugin 156 regarding item “101 A”. In the level 3 specification for item “101 A”, “correctAnswer” is assigned the value “A” and "URI” is assigned the value "items.htm#101A" in data storage 662 under items branch 650 in exam resource file 120.
  • Virtual property set 1506 represents the amalgamated property sets from levels 1, 2, and 3 for item “101A”.
  • Virtual property set 1506 contains "min,” “prompt,” “correctAnswer,” and “URI” information for item plugin 156 regarding item “101 A”.
  • "prompt” has the value “select best” from the level 1 specification
  • “min” has the value “1” from the level 2 specification
  • "correctAnswer” has the value "A”
  • URI has the value "items.htm#101A” from the level 3 specification.
  • Item plugin 156 for item "101A” never has a value assigned for the attribute "max”.
  • Property set 1508 contains "min,” “max,” and “prompt” information for item plugin 156 regarding items “102A” and “103A”.
  • “min” is assigned the value “2”
  • “max” is assigned the value "2”
  • “prompt” is assigned the value "pick 2" in data storage 772 under templates branch 750 in exam resource file 120.
  • property set 1510 contains "correctAnswer” and "URI” information for item plugin 156 regarding item “102A”.
  • "correctAnswer” is assigned the value "B” and "URI” is assigned the value "items.htm#102A” in data storage 662 under items branch 650 in exam resource file 120.
  • Virtual property set 1514 represents the amalgamated property sets from levels 1, 2, and 3 for item "102A".
  • Virtual property set 1514 contains "min,” “max,” “prompt,” “correctAnswer,” and
  • Virtual property set 1516 represents the amalgamated property sets from levels 1, 2, and 3 for item "103A".
  • Virtual property set 1516 contains "min,” “max,” “prompt,” “correctAnswer,” and
  • FireOnRequestEdit (dispID : DISPID) : HRESU T
  • ControlQuer lnterface (iid : const IID&, ppv : void**) : HRESULT
  • CWindowImpI allows you to create a new window or subclass an existing window
  • the DSA multichoice item Does graphic and text distracters. Handles voice-overs and
  • IOIeDocumentView ** EnumViews (ppEnnm : lEnumOlePocument Views**, ppView : IOIeDocumentView **) : GetDocMiscStatus (pdwStatus : DWORD *) :
  • UIActivate (fUIActivate : BOOL) :
  • OnWindowMessage (msg : UINT, wParam : WPARAM, IPara : LPARAM, plResult : LRESULT*) :
  • OnUIActivate 0 HRESULT
  • OnlnPlaceDeactivate 0 HRESULT
  • OnPosRectChange (IprcPosRect : LPCRECT) : HRESULT
  • RequestBorderSpace (pborderwidths : LPCBORDERW1DTHS) : HRESULT
  • SetBorderSpace (pborderwidths : LPCBORDERWIDTHS) : HRESULT
  • HRESULT EnumVerbs ppEnumOleVerb : EEnumOLEVERB**
  • HRESULT Update : void
  • HRESULT IsUpToDate : void
  • HRESULT GetUserClassID pClsid : CLSID*
  • HRESULT Advise (pAdvSink : IAdviseSink*, pdwConnection : DWORD*) : HRESULT Unadvise (dwConnection : DWORD) : HRESULT EnumAdvise (ppenumAdvise : IE ⁇ umSTATDATA**) : HRESULT GetMiscStatus (dwAspect : DWORD, pdwStatus : DWORD*) : HRESULT SetColorSche e ( : LOGPALETTE*) : HRESULT
  • HRESULT EnumVerbs ppEnumOleVerb : IEnumOLEVERB**
  • HRESULT Update : void
  • HRESULT IsUpToDate : void
  • HRESULT GetUserClassID pClsid : CLSID*
  • HRESULT Advise pAdvSink : lAdviseSink*, pdwConnection : DWORD*
  • HRESULT Unadvise dwConnection : DWORD
  • HRESULT EnumAdvisc ppenumAdvise : IEnumSTATDATA**
  • HRESULT GetMiscStatus d Aspect : DWORD, pdwStatus : DWORD*
  • HRESULT SetColorScheme ( : LOGPALETTE*) : HRESULT
  • ContextSensitiveHelp (fEnterMode : BOOL) : HRESULT
  • the UTD browser control is an Active X control that
  • WASH INGTON I 49764V 1 is a customized web browser control that is locked down for security reasons and supports scripting objects from UTDCore
  • AllowFile flag get_AllowURL 0 gets the AllowURL flag put_AllowURL 0 :
  • the UTD sections scores control This control displays bar-graph of a score or scores. This control will run if requested from HTML passed to the UTDC.Browser control.
  • the time points and the messages are configurable.
  • nWidth Integer
  • the size of the area on the split axis. Can be a percentage, pixel count or *.
  • IEvents contains deliverable classes as IDispatch objects. They must be queried individually to determine their exact type.
  • IEvents can be: cExam, cForm, cSection, cltem, cPresentation, cReport, cResults.
  • colAIISections UTDCore-cSections
  • Any one with a reference to this object can initiate this process. It could be an item plugin that wants to save user interaction and state information.
  • FatalError ((Number : Long, sSource : String, sDescription : String) :
  • An example usage could be the user clicks a button on your plugin item. The resulting actions generate an error.
  • Trace (INumbe : Long, sSource : String, sDescription : String) : Reports an event in a plugin to the test driver.
  • GetMsgBox 0 UTDCore.cMsgBox Returns a message box object.
  • the collection pf all sections chosen (that is, returned by a selection plugin) in the exam.
  • colAHGroup The collection of all items of the exam regardless of their level.
  • SectionPrevious 0
  • SectionGoto (vSection : Variant):
  • This class displays a message box with a candidate prompt.
  • This class is returned from IExam::GetMsgBox().
  • the message in the box is supplied by the called.
  • the title can optionally be supplied.
  • the buttons are supplied by the caller, but the button labels are defined at the exam level in the resource file.
  • the parent of the message box is the main exam window.
  • the message box can be set to time out after a duration.
  • the timeout value on the message box in seconds. Read Write.
  • the parent window handle Read/Write
  • Default is the main UTDCore window.
  • the data and current state of the presentation could be a item or a display.
  • the collection maybe empty.
  • bSelected Boolean
  • the score report plugin The score report plugin.
  • a script written in VBscript or JScript To be executed or evaluated by the ActiveX Scripting Engine. This object will expose UTD objects to the script code. These objects will be based on the scope of the "Parent" object provided-to this object.
  • the script will also see oCurSection.
  • the script will also see oCurSection and oCurPresentation.
  • the parent UTD object It could be a form, a section or a presentation.
  • the script will have access to these as the oCurSection, oCurPresentation, and oCurltem objects. Any/All may be passed as NULL parameters indicating that the script does not have access to that particular object.
  • the oStorage parameter is a storage containing an attribute set and a stream containing the global script.

Abstract

A system and method for computer-based testing (100) for producing a test and delivering the test to an examinee includes a storage device (130) that has a first storage location, which stores a first segment of a test definition language, and a second storage location, which stores a second segment of the test definition language, a validation expansion module that validates the first segment and the second segment of the test definition language, a test packager that amalgamates the first storage location and the second storage location and transmits the amalgamated segment to the validation expansion module such that the validation expansion module can determine whether the amalgamated segment forms a complete and valid set, and a test driver (110) that has an executable code that controls functionality that enables the test driver (110) to deliver the test to an examinee.

Description

METHOD AND SYSTEM FOR COMPUTER BASED TESTING USING AN AMALGAMATED RESOURCE FILE
CROSS REFERENCE TO RELATED APPLICATIONS
This application is related to and claims the priority of U.S. Provisional Application Serial No. 60/331,228, filed November 13, 2001 and incorporated herein by reference, and is further related to: U.S. Patent Application entitled "EXTENSIBLE EXAM LANGUAGE (XXL) PROTOCOL FOR COMPUTER BASED TESTING" and having inventors Clarke Daniel Bowers, Tronster Maxwell Hartley, Kyle Michael Kvech, and William Howard Garrison (Docket No. 26119-146); U.S. Patent Application entitled "METHOD AND SYSTEM FOR COMPUTER BASED TESTING USING CUSTOMIZABLE TEMPLATES" and having inventor Clarke Daniel Bowers (Docket No. 26119- 143); U.S. Patent Application entitled "METHOD AND SYSTEM FOR COMPUTER BASED TESTING USING A NON-DETERMINISTIC EXAM EXTENSIBLE LANGUAGE (XXL) PROTOCOL" and having inventor Clarke Daniel Bowers (Docket No. 26119-144); and U.S. Patent Application entitled "METHOD AND SYSTEM FOR COMPUTER BASED TESTING USING PLUGINS TO EXPAND FUNCTIONALITY OF A TEST DRIVER" and having inventor Clarke Daniel Bowers (Docket No. 26119-142) all of which are being filed concurrently herewith and all of which are incorporated by reference in their entirety herein.
BACKGROUND OF THE INVENTION
FIELD OF THE INVENTION The present invention generally relates to the field of computer-based testing, and in particular, the present invention relates to amalgamating an exam resource file to combine first defined and later defined test content and specification of a computer-based test to reduce the amount of authored XXL and to reduce the size of the size of the exam resource file during delivery of the computer-based test to an examinee. ' BACKGROUND OF THE RELATED ART
For many years, standardized testing has been a common method of assessing examinees as regards educational placement, skill evaluation, etc. Due to the prevalence and mass distribution of standardized tests, computer based testing has emerged as a superior method for providing standardized tests, guaranteeing accurate scoring, and ensuring prompt return of test results to examinees. Tests are developed based on the requirements and particulars of test developers. Typically, test developers employ psychometricians or statisticians and psychologists to determine the specific requirements specific to human assessment. These experts often have their own, unique ideas regarding how a test should be presented and regarding the necessary contents of that test, including the visual format of the test as well as the data content of the test. Therefore, a particular computer based test has to be customized to fulfill the client's requirements. Figure 1 illustrates a prior art process for computerized test customization, denoted generally by reference numeral 10. First, a client details the desired test requirements and specifications, step 12.
The computerized test publisher then creates the tools that allow the test publisher to author the items, presentations, etc., required to fulfill the requirements, step 14. The test publisher then writes an item viewer, which allows the test publisher to preview what is being authored, step 16.
An item presenter is then written to present the new item, for example, to the test driver, step
18. Presenting the new item to the test driver requires a modification of the test driver's executable code. The test driver must be modified so that it is aware of the new item and can communicate with the new item presenter, step 20. The test packager must then also be modified, step 22. The test packager, which may also be a compiler, takes what the test publisher has created and writes the result as new object codes for the new syntax. Subsequently, the scoring engine must also be modified to be able to score the new item type, step 24. Finally, the results processor must be modified to be able to accept the new results from the new item, step 26. This process requires no less than seven software creations or modifications to existing software. During authoring of the test, the test author might desire to define some portion of the test specification for re-use. Alternatively the test author might redefine or override certain portions of the test specification that were defined early in the test specification. Previous test definition languages attempted to solve this problem by repeating selected pieces of the test specification at a higher level of the delivery hierarchy. However, it has been determined that this solution does not cover the content of the test specification and assumes that the pattern of repetition is based on the unit of delivery. For example, test specification based on the order of delivery and the hierarchy may be: a two-section exam of math followed by English. All math questions have a minimum response value of 2 and all English questions minimum response value of 3. An example that will not support repetition of test definition is a two-section exam of math and English. Where half the math questions have a minimum response of 2 and the other half of the math questions have a minimum response of 3. In this situation, the hierarchy does not match the pattern of repetition.
Furthermore, it has been determined that this solution requires specific units of the test to have detailed knowledge of shared specification and to communicate the shared specification to other units of the test. U.S. Patent No. 5,827,070 (Kershaw et al.) and U.S. Patent No. 5,565,316 (Kershaw et al.) are incorporated herein by reference. The '070 and '316 patents, which have similar specifications, disclose a computer-based testing system comprising a test development system and a test delivery system. The test development system comprises a test document creation system for specifying the test contents, an item preparation system for computerizing each of the items in the test, a test preparation system for preparing a computerized test, and a test packaging system for combining all of the items and test components into a computerized test package. The computerized test package is then delivered to authorized examinees on a workstation by the test delivery system. Figure 2 illustrates the relationship among session scripts 30, test scripts 32, and units. A script consists of a series of files and further specifies the option settings and configuration data, which the
Test Delivery Application (TDA) needs for operation. During test preparation, scripts are prepared and combined with the items prepared during item preparation. Scripts control the sequence of events during a testing session. Two types of scripts are preferably used, for example: the session script 30 and one or more test scripts 32. The session script 30 controls the order in which units within the testing session are presented. Units provide specific services to the examinee, such as delivering a test or presenting a score report. Just as the session script controls the session, the test script controls what is presented to the examinee during the testing unit. Each testing unit may include one or more delivery units, which are separately timed and scored subdivisions of a test. The system can dynamically select, or spiral, scripts and other test components so that examinees are given what appear to be different tests. FIG. 24 shows the relationship among session scripts 30, test scripts 32, and units.
The session script is the second-level component of the testing package. It performs two primary functions: First, it specifies the Session Control Information, which defines the default options that are in effect for the entire examinee testing session. Second, it controls the order in which units within the testing session are presented and the options used to present them. The units that can be presented within a session script are: General information screen units, Tutorial units, Break units, Data collection units, Scoring and Reporting units, and Testing units.
The session control information contains the default options in effect for the entire session. Control information can be provided at multiple levels within the testing session. Thus, the control information provided at the session level can be overridden by information that occurs later in the session. The information provided at the session level would generally include the following: Name-- the session script name to be used by administrators in selecting a specific session script from Administrative Application menus; Input device—the input device to be used during the session (e.g., mouse or keyboard); Color— the colors to be used during the session; Messages— program-specific messages to override default messages during the session; Demo Script— indicates whether the script presents a demonstration or operational test; Research Indicator— indicates whether the script presents a research pilot test; Special Timing— indicates whether the script is standard or specially timed version. The testing unit presents a test, based on the contents of a test script that may have been selected at runtime. The following units can be included within a testing unit: general information screen unit; tutorial unit; break unit; delivery unit, which delivers items to the examinee. This permits testing programs to interleave general information screens, tutorials, and breaks with sections of a test. The testing unit contains the following information: script selection mode indicates whether dynamic runtime selection is to be used to select the test script; reference to a test script which controls the sequence of events and options used during the testing unit. If dynamic runtime selection is to be used, the reference is to a set of test scripts. Like the session script, the test script performs two primary functions. First, it specifies the test and delivery unit control information. Test control information defines the options that are in effect for the testing unit. Delivery unit control information defines the options that are in effect for a particular delivery unit within a testing unit. It controls the order in which units are presented within the testing unit and the options used to present them. The rules for presentation of units are the same as those for the session script, except that an additional unit, the delivery unit, can be included within a test script.
U.S. Patent No. 5,513,994 (Kershaw et al.), which is incorporated herein by reference, discloses a centralized administrative system and method of administering standardized test to a plurality of examinees. The administrative system is implemented on a central administration workstation and at least one test workstation located in different rooms at a test center. The administrative system software, which provides substantially administrative functions, is executed from the central administration workstation. The administrative system software, which provides function carried out in connection with a test session, is executed from the testing workstations.
None of the Kershaw et al. patents appear to make any mention of a test definition language that does not required the overriding language to be repeated at higher levels of the delivery hierarchy. What is required is a computer based testing system that supports a non-deterministic test definition language that, for example, allows subsequent specifications to be added to the test definition language without requiring repetitive definitions and that enables a reduction in size of exam source files, and an exam resource file, which contains the test definition language during delivery of the test. Other features and advantages in addition to the above, or in the alternative to the above, are described in the Summary of the Invention and the Detailed Descriptions provided below.
SUMMARY OF THE INVENTION
It is one feature and advantage of the present invention to amalgamate test specification and content to reduce the size of a resource file that contains the test specification and content, where a test driver and an expansion module uses the test specification and content to deliver a test to an examinee. It is another optional feature and advantage of the present invention to allow a test publisher to define different aspects of test specification and content for a particular feature of the test in multiple locations and to amalgamate those multiple locations at the time of delivery of the test such that the test publisher does not have to repeat the same aspects if the test specification and content for the particular feature at the multiple locations. It is another optional feature and advantage of the present invention to allow later defined elements of a particular feature to override previously defined elements that exist at a higher level of the test specification and contents.
It is another optional feature and advantage of the present invention to allow later defined elements to define previously omitted non-common test specification and contents. It is another optional feature and advantage of the present invention that specific units of the test need not have detailed knowledge of shared specification and need not communicate the shared specification with other units of the test. These and other features and advantages of the present invention are achieved in a system for computer-based testing for producing a test and delivering the test to an examinee. The test has a presentation format that determines the visual presentation of the test and data content that determines the functional properties of the test. The system includes a storage device that has a first storage location and a second storage location. The first storage location stores a first segment of a test definition language and the second storage location stores a second segment of the test definition language. The first segment and the second segment define information comprising at least one of the data content, the presentation format, progression, scoring, printing, timing, and/or results reporting of the test. In an alternative embodiment, the first segment and the second segment of the test definition language further comprise the same category of information, where the category is at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and/or the results reporting of the test.
The system also includes a validation expansion module that validates the first segment and the second segment of the test definition language by determining whether the first segment and the second segment are correctly formatted. The validation expansion module also stores the first segment to one of the first storage location and the second storage location and the second segment to another one of the first storage location and the second storage location in the storage device. In an alternative embodiment, the validation expansion module is a plugin.
The system further includes a test packager that transmits the first segment and the second segment of the test definition language to the validation expansion module during delivery of the test. The test packager determines to which of the first storage location and the second storage location in the storage device the first segment and the second segment are stored by the validation expansion module. The test packager also amalgamates the first storage location and the second storage location and stores an amalgamated segment of the test definition language in a first virtual storage location and transmits the amalgamated segment to the validation expansion module such that the validation expansion module can determine whether the amalgamated segment forms a complete and valid set of the first segment and second segment of the test definition language. In an alternative embodiment, the test packager is a compiler.
The system further includes a test driver that has an executable code that controls functionality that enables the test driver to deliver the test to an examinee using a display device, manage the test, control the progression of the test, control the scoring of the test, control the printing of the test, control the timing of the test, and control the results reporting of the test based on the test definition language. During delivery of the test, the test driver amalgamates the first storage location and the second storage location into a second virtual storage location such that the validation expansion module can retrieve the amalgamated segment from the second virtual storage location to enable the functionality of the test driver. In another embodiment of the present invention, a system for computer-based testing includes a storage device that has a first storage location, a second storage location, and a third storage location.
The first storage location stores a first segment of a test definition language, the second storage location stores a second segment of the test definition language, and the third storage location stores a third segment of the test definition language. The first segment, the second segment, and the third segment comprise at least one of the data content, the presentation format, progression, scoring, printing, timing, and/or results reporting of the test. In an alternative embodiment, the first segment, the second segment, and the third segment of the test definition language further comprise the same category of information, where the category is at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and the results reporting of the test.
The system also includes a validation expansion module that validates the first segment, the second segment, and the third segment of the test definition language, and stores the first segment to one of the first storage location, the second storage location, and the third storage location, the second segment to another one of the first storage location, the second storage location, and the third storage location, and the third segment to another one of the first storage location, the second storage location, and the third storage location. In an alternative embodiment, the validation expansion module is a plugin.
The system further includes a test packager that transmits the first segment, the second segment, and the third segment of the test definition language to the validation expansion module during production of the test such that the validation expansion module is capable of validating the first segment, the second segment, and the third segment. The test packager determines to which of the first storage location, the second storage location, and the third storage location in the storage device the first segment, the second segment, and the third segment are stored by the validation expansion module. The test packager also amalgamates the first storage location, the second storage location, and the third storage location and stores an amalgamated segment of the test definition language in a first virtual storage location, and transmits the amalgamated segment to the validation expansion module such that the validation expansion module is capable of determining a complete and valid set of the first segment, the second segment, and the third segment of the test definition language. In an alternative embodiment, the test packager is a compiler. The system also includes a test driver that has an executable code that controls functionality performed by the test driver that enables the test driver to deliver the test to an examinee using a display device, manage the test, control progression of the test, control scoring of the test, control printing of the test, control timing of the test, and control reporting of test results based on the test definition language. During delivery of the test, the test driver amalgamates the first storage location, the second storage location, and the third storage location into a second virtual storage location such that the validation expansion module is capable of retrieving the amalgamated segment from the second virtual storage location to enable the functionality of the test driver. In another embodiment of the present invention, a method of computer-based testing for a test is provided, where the test has a presentation format that determines the visual presentation of the test and data content that determines the functional properties of the test. Delivery of the test is controlled by a test driver that has an executable code that enables the test driver to deliver the test to an examinee using a display device, manage the test, control progression of the test, control scoring of the test, control printing of the test, control timing of the test, and control results reporting of the test.
The method includes the sequential, non-sequential, and/or sequence independent steps of validating a first segment of the test definition language during a test production cycle and validating a second segment of the test definition language during the test production cycle. The first segment and the second segment define information comprising at least one of the data content, the presentation format, progression, scoring, printing, timing, and/or results reporting of the test. In an alternative embodiment, the first segment and the second segment further define the same category of information, which is at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and the results reporting of the test. The method also include amalgamating the first segment and the second segment of the test definition language during the test production cycle such that an amalgamated segment is formed, validating the amalgamated segment during the test production cycle, such that a validated amalgamated segment is created and such that the amalgamated segment is valid if the amalgamated segment forms a complete and valid set. The method further includes amalgamating the first segment and the second segment of the test definition language during a test delivery cycle. The validated amalgamated segment is reformed and retrieved by a validation expansion module to enable the functionality of the test driver.
In another embodiment of the present invention, a method for computer-based testing includes validating a first segment of the test definition language during a test production cycle, validating a second segment of the test definition language during the test production cycle, and validating a third segment of the test definition language. The first segment, the second segment, and the third segment define information comprising at least one of the data content, the presentation format, progression, scoring, printing, timing, and/or results reporting of the test. In an alternative embodiment, the first segment, the second segment, and the third segment of the test definition language further define the same category of information, the same category of information being at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and the results reporting of the test.
The method also includes amalgamating the first segment, the second segment, and the third segment of the test definition language during the test production cycle, such that an amalgamated segment is formed and validating the amalgamated segment during the test production cycle, such that a validated amalgamated segment is created and such that the amalgamated segment is valid if the amalgamated segment forms a complete and valid set. The method further includes amalgamating the first segment, the second segment, and the third segment of the test definition language during a test delivery cycle, such that the validated amalgamated segment is reformed and retrieved by a validation expansion module to enable the functionality of the test driver.
In another embodiment of the present invention, a method for computer-based testing is provided, which includes defining the presentation format and the data content in at least two locations comprising a plugin element and actual usage of the plugin element on at least one unit of the test, the at least one unit comprising form, section and group associated with the test and amalgamating the presentation format and the data content defined in the two locations by at least one test driver to deliver the test to an examinee. In another embodiment of the present invention, a method for computer-based testing includes validating by a plugin at least partial exam source information that is received and amalgamating exam resource data associated with the test. The method also includes validating the exam resource data that has been amalgamated to provide a substantially complete amalgamated exam specification and content and delivering the substantially complete amalgamated exam specification and content validated by the validating step.
In another embodiment of the present invention, a method for computer-based testing is provided for a test that has a first presentation format, a second presentation format, a first data content, and a second data content. The method includes defining the first presentation format and the second presentation format in at least two locations comprising a plugin element and actual usage of the plugin element on at least one unit of the test, the at least one unit comprising form, section and group associated with the test. The method further includes defining the first data content and the second data content in at least two locations comprising the plugin element and the actual usage of the plugin element on at least one unit of the test, the at least one unit comprising form, section and group associated with the test. The method also includes amalgamating at least one of the first and second presentation format and the first and second data content defined in the two locations by at least one test driver to deliver the test to an examinee.
There has thus been outlined, rather broadly, the more important features of the invention and several, but not all, embodiments in order that the detailed description thereof that follows may be better understood, and in order that the present contribution to the art may be better appreciated. There are, of course, additional features of the invention that will be described hereinafter and which will form the subject matter of the claims appended hereto.
In this respect, before explaining at least one embodiment of the invention in detail, it is to be understood that the invention is not limited in its application to the details of construction and to the arrangements of the components set forth in the following description or illustrated in the drawings. The invention is capable of other embodiments and of being practiced and carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein are for the purpose of description and should not be regarded as limiting. As such, those skilled in the art will appreciate that the conception, upon which this disclosure is based, may readily be utilized as a basis for the designing of other structures, methods and systems for carrying out the several purposes of the present invention. It is important, therefore, that the claims be regarded as including such equivalent constructions insofar as they do not depart from the spirit and scope of the present invention.
Further, the purpose of the foregoing abstract is to enable the U.S. Patent and Trademark Office and the public generally, and especially the scientists, engineers and practitioners in the art who are not familiar with patent or legal terms or phraseol gy, to determine quickly from a cursory inspection the nature and essence of the technical disclosure of the application. The abstract is neither intended to define the invention of the application, which is measured by the claims, nor is it intended to be limiting as to the scope of the invention in any way.
These, together with other objects of the invention, along with the various features of novelty, which characterize the invention, are pointed out with particularity in the claims annexed to and forming a part of this disclosure. For a better understanding of the invention, its operating advantages and the specific objects attained by its uses, reference should be had to the accompanying drawings and descriptive matter in which there is illustrated preferred embodiments of the invention.
BRIEF DESCRIPTION OF THE DRAWINGS
Figure 1 is a flow diagram of a prior art method for computerized test customization;
Figure 2 is a block diagram of a prior art testing script; Figure 3 is a schematic diagram of a computer-based testing system according to the present invention;
Figure 4 is a block diagram illustrating different types of plugins that are used with the computer-based testing system according to the current invention;
Figure 5 illustrates various components that comprise an exam source file; Figures 6A and 6B are schematics illustrating the components, classes, and interfaces that comprise a test definition language compiler according to the present invention;
Figure 7 is a schematic illustrating the components that comprise a test driver and a test administration system according to the present invention;
Figures 8A and 8B are schematic illustrating the classes and interfaces that comprise the test driver;
Figure 9 illustrates the interfaces that comprise a structured storage according to the present invention;
Figures 10A and 10B are schematics illustrating the classes and interfaces that comprise the structure storage and associated operations; Figure 1 1 is a block diagram of main storage branches of an exam resource file according to the present invention;
Figure 12 is a block diagram illustrating an exams branch of the exam resource file; Figure 13 is a block diagram illustrating a forms branch of the exam resource file;
Figure 14 is a block diagram illustrating an items branch of the exam resource file;
Figure 15 is a block diagram illustrating a categories branch of the exam resource file;
Figure 16 is a block diagram illustrating a templates branch of the exam resource file; Figure 17 is a block diagram illustrating a sections branch of the exam resource file;
Figure 18 is a block diagram illustrating a groups branch of the exam resource file;
Figures 19A, 19B, 19C, and 19D are block diagrams illustrating an events sub-branch of the groups branch of the exam resource file;
Figure 20 is a block diagram illustrating a plugins branch of the exam resource file; Figure 21 is a block diagram illustrating a data branch of the exam resource file;
Figure 22 is a block diagram illustrating a formGroups branch of the exam resource file;
Figure 23 is a block diagram illustrating an attributes branch of the exam resource file;
Figure 24 is a block diagram illustrating a scripts branch of the exam resource file;
Figure 25 is a block diagram illustrating a message box branch of the exam resource file; Figures 26A, 26B, 26C, and 26D are block diagrams of an exam instance file according to the present invention;
Figure 27 is a flow diagram of a method for computerized test customization according to the present invention;
Figure 28 is a diagram of a life cycle of a plugin according to the present invention; Figure 29 is a flow diagram of a process for compiling plugins according to the present invention;
Figures 30A, 30B, 30C, and 30D are flow diagrams of a process for delivering plugins to an examinee during a computer-based test;
Figure 31 is a flow chart illustrating a process for amalgamation of invisible plugins according to the present invention;
Figure 32 is a flow chart illustrating a process for amalgamation of visible plugins according to the present invention;
Figure 33 is a flow chart of a process for amalgamation according to the present invention;
Figure 34 is a flow chart for a process for validating exam source according to the present invention;
Figure 35 is a flow chart for a process for validating amalgamated test specification and content using a plugin according to the present invention;
Figure 36 is a flow chart for a process of delivering amalgamated test specification and content using a test driver and a plugin according to the present invention; and Figure 37 is a flow diagram for an example of amalgamation of test specification and content relating to items according to the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
Reference now will be made in detail to the presently preferred embodiments of the invention. Such embodiments are provided by way of explanation of the invention, which is not intended to be limited thereto. In fact, those of ordinary skill in the art may appreciate upon reading the present specification and viewing the present drawings that various modifications and variations can be made. For example, features illustrated or described as part of one embodiment can be used on other embodiments to yield a still further embodiment. Additionally, certain features may be interchanged with similar devices or features not mentioned yet which perform the same or similar functions. It is therefore intended that such modifications and variations are included within the totality of the present invention.
The present invention discloses a system and method of computer-based testing using a test driver that is, for example, object-oriented and is architected to dynamically add functionality through, for example, the use of an expansion module, and preferably through the use of plugins. The test driver preferably references component object model servers using standard interfaces, and uses, for example, class names (that can be an Active Document) defined in a custom test definition language entitled extensible eXam Language ("XXL") based on extensible Markup Language ("XML") format to interact with existing applications while offering the flexibility of allowing development of new plugins. These new plugins can be customized to a client's needs without changing the core test driver. The specific format and protocol of XXL is also described in the co-pending application filed on the same date, entitled "EXTENSIBLE EXAM LANGUAGE (XXL) PROTOCOL FOR COMPUTER BASED TESTING," incorporated herein by reference.
The plugins advantageously enable the test driver to support, for example, new item types, navigation algorithms, information displays, scoring algorithms, timing algorithms, test unit selection algorithms, results persistence reporting, printed score reporting, and/or helm types without change to the test driver's executable. Plugins also allow expansion of the test driver's functionality without requiring the test driver to be recompiled or re-linked, and without requiring the test publisher to learn to program. Since plugins are written independently of the test driver, plugins can be written long after the test driver is built.
The client and the software developer can design and test the plugins and distribute the plugins to each test site. By using this method, large-scale regression testing of other examinations will not usually be necessary unless changes are made to the plugins that may be used by many examinations. A test publisher defines common plugin test specification and common plugin test content in early XXL elements. These previous elements can be referenced by later XXL elements. If the later element defines nothing more then it will receive just the common specification and content. Optionally, the later elements can define omitted non-common plugin specification and contents.
Further, the later elements can override previously defined common plugin specification and contents with exception plugin specification and contents. This process of new definition and overriding of specification and contents by later XXL elements is not limited to two stages. Multiple later stages are allowed.
When a plugin delivers it uses its specification and contents to determine how and what to deliver. This will be the amalgamation of all relevant XXL elements. The amalgamation is invisible to the plugin and requires no special effort for the plugin during delivery. The amalgamation occurs at delivery time and therefore common specification and content appears only once in the compile exam resource file. Thus the size of the resource file is reduced.
I. Overview of Computer-Based Test Delivery System
Figure 3 shows an overview of the software architecture for the computer-based test delivery system of the present invention, denoted generally by reference numeral 100. Test driver 110 is responsible for controlling all aspects of the computer-based test. Test driver 1 10 identifies examinees scheduled to take the computer-based test and identifies and creates the appropriate test. Test driver 110 then presents all of the test components to examinees using a display device (not shown), such as a computer monitor, and enables examinees to enter responses to test questions through the use of an input device (not shown), such as a keyboard, a mouse, etc. Test driver 110 also monitors the security of the test. For example, test driver 110 can prevent access to the Internet and can validate examinees, although, these functions are preferably performed by the test center administration system. Test driver 110 also monitors the timing of the test, providing relevant warnings to examinee regarding the elapsed time of the test and the time remaining for a particular section of the test or for the entire test. Test driver 110 is also responsible for scoring the test, once the test is completed or while the test is in progress, and for reporting the results of the test by physical printout using printer 182 or in a file format using candidate exam results file 180. If the test is interrupted while in progress, for example, due to a power failure, test driver 110 restarts the test, preferably at the point at which the test was interrupted, as will be described subsequently in more detail. Finally, if the test is left incomplete, test driver 110 cleans up the incomplete test. An incomplete test will have an exam instance file in the examinee's directory but will not have created a results file. A results file is created even though generally the candidate will fail. The number of items delivered to the examinee is recorded in the results file. Test driver 1 10 picks up where the event was interrupted and invisibly deliveries the rest of the units of the test. A test specification is authored by a test publisher according to the specifications of the client and stored in exam source files 130. Exam source files 130 include data files 132, XXL files 134, multimedia files 136, and hypertext markup language ("HTML") files 138. XXL files 134 include the test specification, which contains the client's requirements for the test, a bank of test items or questions, templates that determine the physical appearance of the test, plugins, and any additional data necessary to implement the test. Additional data is also stored in data files 132. For example an adaptive selection plugin may need a, b &c theta values. These values are stored in a binary file created by a statistical package. HTML files 130 include, for example, any visual components of the test, such as the appearance of test items or questions, the appearance of presentations on the display device, the appearance of any client specified customizations, and/or the appearance of score reports. HTML files
130 preferably also include script, for example, VBscript and Jscript, or Java script. HTML files 130 are preferably authored using Microsoft's FrontPage 2000. FrontPage 2000 is preferably also used to manage the source files in a hierarchy that is chosen by the test publisher. Multimedia files 136 include, for example, any images (.jpg, .gif, etc.) and/or sound files (.mp3, .wav, .au, etc.) that are used during the test.
XXL compiler 140 retrieves XXL files 134 from exam source files 130 using interface 190 and compiles the XXL test content stored in XXL files 134. XXL compiler 140 stores the compiled test files in exam resource file 120. In another embodiment, exam source files 130 do not contain XXL files 134 and contains, for example, only multi-media files. In this embodiment, XXL compiler 140 is merely a test packager that writes the data directly to exam resource file 120 without modification or validation. The data appears in a stream under the "data" branch of exam resource file 120. The name of the stream is specified by the test author.
In a preferred embodiment, XXL files 134 also include XXL language that defines plugins 150, in which case, plugins 150 assist XXL compiler 140 in compiling XXL files 134. Test driver 110 preferably supports, for example, nine different types of plugins 150, including, for example: display plugin 152; helm plugin 154; item plugin 156; timer plugin 158; selection plugin 160; navigation plugin 162; scoring plugin 164; results plugin 166; and report plugin 168. Plugins 150, which are also included in XXL files 134, are the first XML files compiled into exam resource file 120.
Plugins 150 allow a test designer to customize the behavior of test driver 110 and are divided into two types, for example: visible plugins and invisible plugins, as shown in Figure 4. The visible plugins, which include display plugin 152, helm plugin 154, and item plugin 156, enable the test driver to control what is presented visually to an examinee on the display device. The invisible plugins, which include timer plugin 158, selection plugin 160, navigation plugin 162, scoring plugin 164, results plugin 166, and report plugin 168, enable the test driver to control more functional aspects of the test. Plugins 150 are used to validate data stored in exam source files 130 that is to be used by one of plugins 150 during delivery of the test to the examinee, as is described below in greater detail. Plugins 150 are, preferably, component object model ("COM") objects, as described below. Plugins 150, may also utilize Java implementation. Plugins 150 are preferably written using Microsoft Visual C++ or Visual Basic 6.0 or any fully COM enabled language. Plugins 150 may be in or out-of-process, and, therefore, can exist as executable (".EXE") files or as dynamic link library (".DLL") files.
An application or component that uses objects provided by another component is called a client. Components are characterized by their location relative to clients. An out-of process component is an .exe file that runs in its own process, with its own thread of execution. Communication between a client and an out-of-process component is therefore called cross-process or out-of-process communication.
An in-process component, such as a .dll or .oxc file, runs in the same process as the client, it provides the fastest way of accessing objects, because property and method calls don't have to be marshaled across process boundaries. However, an in-process component must use the client's thread of execution.
Exam resource file 120 receives the compiled test content from XXL compiler 140 and plugins
150, if applicable, and stores the compiled test content in an object-linking and embedding ("OLE") structured storage format, called POLESS, which is described in greater detail below. Other storage formats may optionally be used. OLE allows different objects to write information into the same file, for example, embedding an Excel spreadsheet inside a Word document. OLE supports two types of structures, embedding and linking. In OLE embedding, the Word document of the example is a container application and the Excel spreadsheet is an embedded object. The container application contains a copy of the embedded object and changes made to the embedded object affect only the container application. In OLE linking, the Word document of the example is the container application and the Excel spreadsheet is a linked object. The container application contains a pointer to the linked object and any changes made to the linked object change the original linked object. Any other applications that link to the linked object are also updated. POLESS supports structured storage such that only one change made to an object stored in exam resource file 120 is globally effective. Test driver 110 comprises Active Document container application 112 for the visible plugins, display plugin 152, helm plugin 154, and item plugin 156, which function as embedded objects, preferably COM objects.
Figure 3-1 shows an example of Active Document container application 112 being used with several item plugins 156. Each item plugin 156, in this case representing a multiple choice ("multi- choice") item, a hot area item, and a fill in the blank item, are linked to Active Document container application ] 12 through the Iltem and IPlugin COM interfaces 169.
Both XXL compiler 140 and plugins 150 are involved in storing the compiled test content into exam resource file 120, if any of plugins 150 are being used. Exam resource file 120 comprises, for example, a hierarchical storage structure, as will be described in further detail below. Other storage structures may optionally be used. XXL compiler 140 determines to which storage location a specific segment of the compiled test content is to be stored. However, if any of plugins 150 are used to validate the portion of any of the data from exam source files 130, then the plugins 150 store the data directly to the exam resource file, based upon directions from XXL compiler 140. XXL compiler uses IPersistResource interface 192, co-located with I-Plugin interface 167 in Figure 3, to control the persistence of the data to exam resource file 10. XXL compiler 140 and plugins 150 write the data to exam resource file 120 using POLESS interfaces 191. Figure 5 illustrates the contents of exam source file 130, which are compiled into exam resource file 120 by XXL compiler 140 and plugins 150. FrontPage 2000 Web 200 is used, for example, to author the test. Exam source files 130 contain media files 210, visual files 220, and logic files 230. Media files 210 are multimedia files used to enhance the presentation of the test, including, for example, XML data files 212, sound files 214, image files 216, and binary files 218. XML data files 212 include the XXL test definition language and the XXL extensions from the plugins 150 that use XML. The test specification, presentation, scoring and other information is specified in the XML files. Sound files 214 include any sounds that are to be used during the test, such as .mp3 files, .au files, etc. Image files 216 include any images to be used during the test, such as jpg files, .gif files, etc. Binary files 218 include any data needed by a plugin 150 that is not in XXL format. Visual files 220 are HTML files that specify the visual presentation of the test as presented to the examine on the display device, including items files 222, presentation files 224, score report files 226, and custom look files 228. Items files 222 include HTML files that are used to specify the visual component of test questions, e.g., stems and distractors. Items files 222 are capable also of referencing external exhibits. An exhibit could be a chart, diagram or photograph. Formats of exhibits include, for example: jpg, .png, etc. Presentation files 224 define what is seen by the examinee on the display device at a particular instant during the test. Score report files 226 include is typically an HTML file with embedded script that includes, for example candidate demographics, appointment information, and candidate performance. The performance might include pass/fail, achievement in different content areas, etc. Custom look files 228 include are typically HTML files with embedded script to layout, for example, the title bar and information contained therein. Logic files 230 are XML files that specify the functional aspects of the test, including test specification files 232, plugin files 234, item bank files 236, and template files 238. Test specification files 232 specify the content and progression of the test as provided by the client. Plugin files 234 define plugins 150 and contain any data necessary to implement plugins 150. Item bank files 236 include the data content and properties of the items, or test questions, that are to be presented to the examinee during the test. Properties of an item include the correct answer for the item, the weight given to the item, etc. Template files 238 define visual layouts that are used with the display screen during the test.
Referring again to Figure 3, once a test has begun, test driver 110 accesses exam resource file 120 for the instructions and files needed to implement the test, using POLESS interfaces 193. Test driver 110 also accesses plugins 150 for additional data that expands the functionality of test driver 110 in the areas of items, navigation algorithms, information displays, scoring algorithms, timing algorithms, test unit selection algorithms, results persistence reporting, printed score reporting, and/or helm types. Test driver 110 communicates with plugins 150 using various COM interfaces 169. COM interfaces facilitate OLE linking. As stated previously, test driver 110 is an Active Document container application and plugins 150 are embedded objects. The COM interfaces function as communications paths between the container application and the objects. There are, for example, ten COM interfaces utilized in computer-based test delivery system 100. IPlugin interface 167, which is also a COM interface, is supported by all of plugins 150. COM interfaces 169, therefore, includes the IPlugin interface. The IPlugin interface contains generic operations such as loading and unloading required of all plugins 150. In addition to the global IPlugin interface, each plugin 150 also uses, for example, a second, individual COM interface 169 to communicate with test driver 110. Alternative structures of the IPlugin interface may also be used. Table 1 shows the relationship between each plugin 150 and the COM interface 169 used with that particular plugin 150.
TABLE 1: COM INTERFACEFOR PLUGINS
Figure imgf000017_0001
Figure imgf000018_0001
Exam instance file 170 is used to restart a test if the test has been interrupted, for example, because of a power failure. During delivery of the test, exam instance file 170 receives examination state information from test driver 110 and plugins 150 regarding the state of all running objects being used to deliver the test. The examination state information includes the presentation that was being delivered on the display device before the interruption, the responses the examinee had entered in that presentation, etc. When the test is restarted, the exam instance file 170 loads the state information back to test driver 110 and plugins 150, allowing the test to return to operation at the point where the test had been interrupted. Preferably, the running state of all objects is saved to exam instance file 170 rather than of only some of the objects. Saving the state of only some of the objects to exam instance file 170 causes the potential problem of only a portion of the test information being restored after a test interruption. Exam instance file 170 may also store additional information relating to the test, including, for example: the timing utilized and time remaining on units of the exam, the current unit of delivery, candidate score, etc. Test driver 110 and plugins 150 communicate with exam instance file
170 using POLESS interfaces 195. Test driver 110 controls communications between test driver 110 and plugins 150 using IPersistlnstance interface 196, which is co-located with COM interfaces 169 in
Figure 3.
Several administrative environments perform the administrative functions of computer-based test delivery system 100, for example: Test Center Manager ("TCM") Bridge 172; Educational Testing
Service ("ETS") Bridge 174; and Unified Administration System ("UAS") 174. Administrative functions include, for example: checking-in an examinee, starting the test, aborting the test, pausing the test, resuming the test, and transmitting results.
There are preferably two ways to run Test driver 110. The first is through a series of command line options and the second is using COM interfaces describing appointment information. The command line option exists for backwards compatibility in a standard ETS environment and a TCM environment.
Table 2 shows a list of command line options test driver 110 supports. There are four programs which launch the test through the COM interface, for example: 1) LaunchTest.exe (for test production and client review); 2) UAS; 3) UTD2ETS.dll (an internal compatibility module for use with the ETS administration environment); and 4) UTD2TCM (for the Test Center Manger environment). Other number of environments and/or programs may optionally be used.
Figure imgf000019_0001
The administration environments use several interfaces to communicate with test driver 110.
-Appointment interface 176 is part of UAS 174 and allows access by test driver 110 to examinee information for the examinee taking the test, such as demographics. The examinee information is included in candidate exam results file 180, which is created by the test driver. ILaunch2 interface 177 functions as the primary control interface for UAS 174 and allows UAS 174 to control various components such as test driver 110, screen resolution change, accommodations for disabled candidates, examinee check-in, etc., in a test center, which is the physical location where the examinee is taking the test. ITransfer interface 199 transfers candidate exam results file 180 and other files back to UAS 174.
IPrint interface 198 sends information regarding any reports to printer 182. II. XXL Compiler Interfaces and Classes
Figures 6A and 6B illustrate the main diagram for XXL compiler 140. XXL compiler 140 comprises the following classes, for example: cCompile 2000; cData 2004; cArea 2006; cTemplate 2008; cCategory 2010; cltem 2012; cPresentation 2014; cGroup 2016; cSection 2018; cForm 2020; cFromGroup 2022; cExam 2024; cMsgBox 2026; cChecksum 2028; cEvent 2030; cResult 2032; cReport 2024; cPlugin 2036; and cXXL 2038.
The main interface to XXL compiler 140 is ICo pile interface 2002. ICompile interface 2002 is implemented by cCompiler class 2000. All control and initiation of compilation of exam source files 130 into exam resource file 120 occurs by way of this single public interface. The core, non-plugin related elements of the XXL test definition language, as stored in XXL files 134, are compiled by classes in XXL compiler 140. For example, cSection class 2018, compiles the section element, and cGroup class 2016 compiles the group element.
ICompile interface 2002 supports the following operations, for example: createResource(); addSourceO; addData(); closeResource(); about(); linkResource(); openResource() and getCryptoObject(). CreateResource() creates a resource file, for example, an XXL based resource file such as exam resource file 120. AddSource() compiles an XXL file into the resource file.
AddData()adds a file directly to a data branch of the resource file. CloseResource() closes the resource file. LinkResourceO links a resource in the resource file and is performed after all compiling of the source files are completed. GetCryptoObject() returns an ICrypto object containing the current encryption setting of POLESS, as described below. The classes of XXL compiler 1040, e.g., cForm 2020 and cltem 2012, handle individual XXL core language elements. All of these classes compile the specific XXL source element into exam resource file 120. All of these class language elements are also symbols used in later references. Therefore, the classes all derive from cSymbol class 2040. cSymbol class 2040 allows the classes of XXL compiler 140 to reside in a symbol table. For example, the XXL element plugin 150 appears as follows in XXL files 134: <plugin name="helmNextPrevious" progid="UTDP.cNextPrevious" />
This XXL call causes an instance of cPlugin class 2036 to be created, compiles the source, and writes the compiled result to exam resource file 120. The name and ID of Plugin 150 is also added to the symbol table for later reference.
XXL compiler 140 also contains the following token classes, for example: cToken 2042; cTokenCreatorNoRef 2044; cTokenCreator 2046; CtokenCreatorRef 2048; cTokenCreatorBase 2050; and cTokenFactory 2054. These token classes are involved in the identification of tokens. Tokens turn into symbols after identification. Symbols are any class derived from cSymbol, e.g., cTemplate, cSection, etc.
XXL compiler 140 also contains the following symbol table classes, for example: cPluginSymbolTable 2058; cTemplateSymbolTable 2060; cSymbolTable 2062; cFFGSymbolTable 2064; cSGPSymbolTable 2066; and cSymbolTableBase 2068. These classes are varieties of symbol tables. There are different symbol tables for different groups of symbols. A group of symbols define a name space for the symbol. Common symbol table functions are located in the base symbol table classes and templates.
All content and specification destined for a plugin 150 appears in the data element in XXL. For example, below is an item definition in XXL: < item name= "wan ABreakl " skipAllo ed= " f alse " > <data>
<multiChoice correctAnswer="A" maxResponses=" 1 " minResponses="1 " autoPrompt="false"
URI="itembank/info_item.htm#wantABrea "/>
</data>
</item>
The item element is handled by a cltem class 2012 object. The data element in the XXL definition is handled by a cData class 2004 object. Item plugin 156 Plugin 150 will receive the source to compile from the cData class 2004 object, in this example, a multiChoice element. cWrapXML class 2052, a wrapper class for XML DOM nodes, supports error handling. cCustomAttributes class 2056 compiles the custom attributes XXL element. cWrapPropertySet class 2070 is a wrapper class for a POLESS property storage. III. Test Driver Interfaces and Classes
A. Interfaces
Figure 7 shows test driver 110, UAS 174, and the interfaces used by and between the test driver 110 and UAS 174 to deliver the test. UAS 174 defines LLaunch2 interface 177, which is used by UAS 174 to initiate test events. ILaunch2 interface 177 is an extension of ILaunch interface 178, which, in other embodiments of the present invention, is also used by UAS 174 to initiate test events. UAS 174 also defines and implements additional interfaces, for example: LAppointment interface 176; IPrint interface 198; and ITransfer interface 199. LAppointment interface 176 transfers examinee candidate information and appointment details from UAS 174 to test driver 110, as is illustrated by the dashed arrow connecting LAppointment interface 176 to test driver 110. IPrint interface 198 allows UAS 174 to send print requests to printer 198 regarding reports, for example, score reports. ITransfer interface 199 allows UAS 174 to request the transfer of information from candidate exam results file 180 back to UAS 174.
Test driver 110 defines various interfaces to allow test driver 110 to communicate with different parts of computer-based test delivery system 100. Test driver 110 includes, for example, ten COM interfaces 169 to communicate and transfer data with plugins 150. (See Table 1 above) The COM interfaces 169 are denoted in Figure 7 as follows, for example: LDisplay interface 169a; IHelm interface 169b; Iltem interface 169c; IUnitTimer interface 169d; ISelection interface 169e; .Navigate interface 169f; IScore interface 169g; IResults interface 169h; IReport interface 169i; and IPlugin interface 169j.
Test driver 110 and plugins 150 communicate and transfer data with exam resource file 120 using, for example, three LPersistResource interfaces 192: LPersistResourceStream interface 192a; IPersistResourceSet interface 192b; and LPersistResourceStore interface 192. LPersistResource interfaces 192 are used by plugins 150 during compilation of exam source files 130 and are used by both test driver 110 and plugins 150 during delivery of the test. During compilation of exam source files 130, XXL compiler 140 directs plugins 150 in which storage location of exam resource file 120 to store any information that plugins 150 have validated. Plugins 150 can then retrieve the stored information from exam resource file 150 during delivery of the test. Other number of interfaces and different combination of functionality may alternatively be used. Information is saved from plugins 150, or from XXL compiler 140 in general, to exam resource file 120, for example, as either a stream of data, as a set of data, or as a storage structure, depending on which of the three LPersistResource interfaces 192 is implemented to save the information from plugins 150, to exam resource file 120. IPersistResourceStream interface 192a saves the information, for example, as a stream of data. A stream of data is simply a stream of bytes stored as a linear sequence. IPersistResourceSet interface 192b saves the information, for example, as a set of data. A set of data is preferably a name-value property pair. For example, the name of a particular property for an item is distractors and the value is the number of distractors required for that item. IPersistResourceSet interface 192 allows the name-value property pair to be saved together in exam resource file 120.
LPersistResourceStore interface 192c saves the information, for example, in a directory format with storage areas. The directory format allows other streams of data to be saved within the storage area and for sub-storages to be saved under the storage area. LPersistlnstance interface 196, likewise, comprises, for example, three, different interfaces, for example: IPersistlnstanceStream interface 196a; LPersistlnstanceSet interface 196b; and
LPersistlnstanceStore interface 196c. Examination state information is saved to exam instance file 170 as, for example, a stream of data, as a set of data, or as a storage element, depending on which of the three LPersistResource interfaces 192 is implemented. Two of the interfaces, IContainerNotify interface 200 and IContainerNotifyHelm interface 206, function as callback interfaces from plugins 150 to test driver 110. IContainerNotify interface 200 allows a visible plugin to inform test driver 110, for example, that the plugin is displayed and ready for examinee interaction. IContainerNotifyHelm interface 206 allows helm plugin 154 to request navigation from test driver 110 after receiving an input from the examinee to move to another section of the test. EVIore interface 202 is used to convey whether the examinee has seen all content in a presentation. For example, a "more" button appears in place of the next button when the content exceeds the window length. When the examinee scrolls to the bottom, the "more" button disappears and is replaced with the "next" button. Collection interface 204 is used by test driver 110 to hold any group entities, for example, categories and sections of the test. The remaining interfaces are, for example, Microsoft defined Active Document interfaces, used to implement OLE linking functions of test driver 110 and the visible plugins, display plugin 152, helm plugin 154, and item plugin 156. IOlelnPlaceFrame interface 210 controls the container's top-level frame window, which involves allowing the container to insert its menu group into the composite menu, install the composite menu into the appropriate window frame, and remove the container's menu elements from the composite menu. IOlelnPlaceFrame interface 210 sets and displays status text relevant to the end-place object. IOlelnPlaceFrame interface 210 also enables or disables the frames modeless dialogue boxes, and translates accelerator key strokes intended for the container's frame. IOlelnPlaceUI window interface 21 1 is implemented by container applications and used by object applications to negotiate boarder space on the document or frame window. The container provides a RECT structure in which the object can place toolbars and other similar controls, determine if tools can in fact be installed around the objects' window frame, allocates space for the boarder, and establishes a communication channel between the object and each frame and document window. IAdviseSync interface 212 enables containers and other objects to receive notifications of data changes, view changes, and compound-document changes occurring in objects of interest. Container applications, for example, require such notifications to keep cached presentations of their linked and embedded objects up-to-date. Calls to IAdviseSync interface 212 methods are a synchronous, so the call is sent and then the next instruction is executed without waiting for the calls return. lOleWindow interface 213 provides methods that allow an application to obtain the handle to the various windows that participate in-place activation, and also to enter and exit context-sensitive help mode. IOlelnPlaceSite interface 214 manages interaction between the container and the objects in-place client site. The client site is the display site for embedded objects, and provides position and conceptual information about the object.
IOleClientSite interface 215 is the primary means by which an embedded object obtains information about the location and extent of its display site, its moniker, its user interface, and other resources provided by its container. Test driver 110 called IOleClientSite interface 215 to request services from the container. A container must provide one instance of IOleClientSite interface 215 for every compound-document it contains. IOleDocumentSite interface 216 enables a document that has been implemented as a document object to bypass the normal activation sequence for in-place-active objects and to directly instruct its client site to activate it as a document object. A client site with this ability is called a "document site". B. Core Classes
Figures 8A and 8B illustrate the main classes of test driver 110 and the interfaces between test driver 110 and plugins 150. Also shown are the classes that interface to UAS 174. ITransfer interface 199, IPrint interface 198, LLaunch2 interface 177, and lAppointment interface 176 represent the connections from test driver 110 to UAS 174, as described previously. Some of the lines depicted in Figure 8 are solid and some are dashed. The solid lines, for example, between IcResults interface 240 and cEvent class 252, represent inheritance. The dashed lines, for example, between IExam interface 222 and IPlugin interface 169j, represent instantiation.
Inheritance, or generalization, relates to a generalized relationship between classes that shows that the subclass shares the structure or behavior defined in one or more superclasses. A generalized relationship is a solid line with an arrowhead pointing to the superclass. Instantiation, or dependency, represents a relationship between two classes, or between a class and an interface, to show that the client class depends on the supplier class/interface to provide certain services. The arrowhead points to the supplier class/interface. Some services from a supplier class to a client class include, for example: the client class access a value (constant or variable) defined in the supplier class/interface; methods of the cline class invoke methods of the supplier class/interface; and methods of the client class have signatures whose return class or arguments are instances of the supplier class/interface. For instantiation, the cardinality of the relationship is illustrated in Figure 8 if the relationship represents containment. Cardinality specifies how many instances of one class may be associated with a single instance of another class. Cardinality can be shown for relationships to indicate the number of links allowed between one instance of a class and the instances of another class.
Test driver 110 also has several interfaces and implementing classes. Test driver 1 10 interfaces include, for example: LExam interface 222; LMsgBox interface 224; ICategory interface 232; LForm interface 238; IcResults interface 240; IcReport interface 242; IScript interface 246; ISection interface
250; LPresentation interface 248; and/or Icltem interface 256. The classes that implement the main interfaces include, for example: cScreenMinimum class 226; cFormGroup class 228; cPlugin class 230; cArea class 234; cTemplate class 236; cActivePlugin class 250; and cEvent class 252. The interfaces that are prefaced by "lc" have names that already exist for plugins 150 to enact, for example, item plugin 156 implements LLtem interface 169c. Icltem interface 256, however, is the interface implemented by test driver 110 class cltem (not shown). Of course, any number of interfaces may be used, depending on the necessary functionality.
The core class cExam (not shown) implements LLaunch2 interface 177 so that UAS 174 can control test driver 110. The appointment object, which implements LAppointment interface 176, is the main object UAS 174 supplies to test driver 110. The appointment object is available to plugins 150 by way of IPlugin interface 169j. Furthermore, all plugins 150 get access to the test (Iexam) using the IPlugin interface 169, also.
The cExa class selects and delivers the form, using cFormGroup class 228 and LForm interface 238. The form delivers results using IcResults interface 240, reports using IcReport interface 242, and sections contained with in the test using ISection interface 250. Classes that are in the test delivery chain preferably derive from cEvent class 252.
The cResults class (not shown) delivers a results plugin 166 that implements IResult interface 169i. The cReport class (not shown) delivers a report plugin 168 that implements IReport interface 169h. The cSection, cGroup, and cForm classes (not shown) use several invisible plugins 150 to control the delivery of the test. These plugins 150 are timer plugins 158, which implement IUnitTimer interface 169d, selection plugins 160, which implement ISelection interface 169e, scoring plugins 164, which implement IScore interface 169g, and navigation plugins 162, which implement LNavigate interface 169f. The cPresentation class (not shown) supplies data to its template for the display of the presentation. The three visible plugins 150 are created and controlled through cTemplate class 236 and child objects cArea class 234. Item plugins 156 have an extension class in the cltem class (not shown) that wraps the item plugin 156 and provides generic extended services that all item plugins 156 implements. The cltem class in test driver 110 is a wrapper class. The cltem class provides two base services, for example: generic item functionality and access to item plugin 156, which is the wrapping function. Item generic functionality includes, for example: having an item name, having an item title, determining if the item is scored or un-scored, determining whether the item has been presented to the examinee, etc. These services are generic to all items and are provided by test driver 110. Item plugins 156 perform the actual scoring of the item, which is unique to each item type. Item plugins 156 present the content of the item and allow the examinee to interact with the item. These services are unique to each item type.
In addition to the interfaces described previously, test driver 1 10 implements IRegistry interface 220, which allows VB code to access the Windows registry. Test driver 1 10 also implements LLegacyltem interface 258 and LLegacyScore interface 260, which are defined by test driver 1 10 and are implements by certain item plugins 156 and scoring plugins 164. ILegacyltem interface 258 and
LLegacyScore interface 260 allow old item types that existed in previous test drivers to report results like the previous test drivers. For some tests, test driver 110 must report results for old item types, which had very specific ways of reporting results. ILegacyltem interface 258 and ILegacyScore interface 260 allow the new item plugins 156 that represent old item types to report this legacy format of information to result plugins 166 trying to imitate previous test drivers.
A complete description of test driver 110 classes and interfaces is included in Appendix A. V. POLESS All persistent storages, exam resource file 120 and exam instance file 170, preferably utilize
POLESS. POLESS allows data to be embedded, linked, or references as external files from the persistent storage to test driver 110 and Active Document container application 112 (Figure 3). POLESS supports a hierarchical tree structure with node or branch level additions, replacements, and deletions. POLESS also supports optional data encryption at the node level. The type of encryption employed depends on the destination of the information in the persistent storage. For example, different encryption keys may optionally be used for data being routed to test centers, data being routed to administrative data centers, and data being routed for client use (e.g., client review). Microsoft Crypto-API is preferably used to perform encryption of data in the persistent storage. Finally, POLESS also supports optional compression at the node level, preferably using Lempal-Zev compression. POLESS is an extension of OLE structured storage compound document implementation. A compound document is a single document that contains a combination of data structures such as text, graphics, spreadsheets, sound and video clips. The document may embed the additional data types or reference external files by pointers of some kind. There are several benefits to structured storage. Structured storage provides file and data persistence by treating a single file as a structured collection of objects known as storage elements and streams. Another benefit is incremental access. If test driver 110 or plugins 150 need access to an object within a compound file, only that particular object need be loaded and saved, rather than the entire file. Additionally, structure storage supports transaction processing. Test driver 1 10 or plugins 150 can read or write to compound files in transacted mode, where changes made can subsequently be committed or reverted. A. POLESS Components
Figure 9 shows the major components that support POLESS and the interfaces that connect the components. POLESS 300 may be either exam resource file 120 or exam instance file 170. POLESS 300 utilizes PKware library component 330 for storage compression and decompression. POLESS 300 uses Crypto API component 332, a Microsoft application, for storage encryption and decryption. Crypto API component 332 relies on a crypto service provided ("CSP") 334 to perform the actual encryption algorithms. Access to the services of these components is facilitated by standard (API) interfaces exposed by these components.
OLE2SS component 310 contains all the interface definition that makeup structure storage.
These interfaces can be realized by any structured storage implementation, such as compound document implementation OLE2 320 and POLESS 300. The interfaces include, for example: IStream interface 340; ISequential Stream interface 342; IStorage interface 344; and IRootstorage interface 346.
POLESS 300 additionally implements IStreamVB interface 348 and IStorageVB interface 350.
IStreamVB interface 348 supports several functions, for example: ReadVB(); WriteVB();
Clear(); Reset(); get_sName(); get_oStream(); and CopyTo(). ReadVB() reads a specified number of bytes to a data array. WriteVB() writes the byte data to the stream. Clear() clears the stream of all data. Reset() sets position to the beginning of the stream. get_sName() is a read-only function that returns the name of the stream. get_oStream() is a read-only function that returns the IStream interface 348. CopyTo() copies a source stream to a destination stream.
IStorageVB interface 350 supports several functions, for example: Clear(); CommittVB(); RevertVBO; sElementName(); bStorage(); oElement(); CreateStream(); OpenStream();
CreateS torage(); OpenStorage(); get_sName(); get_oStorage(); get_nCount(); GetCompression(); GetEncryptionO; GetCRC(); CreateStreamLinked(); CreatePropertyStg(); OpenPropertyStgO; SetClassO; RegisterAlias(); DestroyO; and get_ElementType(). Clear() clears the storage of all elements. CommittVB() causes transacted mode changes to be reflected in the parent. RevertVB() discards changes made since the last commit. sElementName() returns the name of the element. bStorage() returns TRUE if the element is a sub-storage. oElement() returs IStreamVB interface 348 or IStorage interface VB 350 for the element. CreateStream() creates and opens a stream and returns IStreamVB interface 348.
OpenStreamO opens a stream and returns IStreamVB interface 348. CreateStorage() creates and opens a nested storage and returns IStreamVB interface 348. OpenStorage() opens an existing storage and returns IStreamVB interface 348. get_sName() is a read-only function that returns the name of the storage. get_oStorage() is a read-only function that returns IStorage interface 350. get_nCount() is a read-only function that returns a count of the elements. GetCompression() returns the status of file compression. GetEncryptionO returns the status of file encryption. GetCRC()returns the status of file CRC checking. CreateStreamLinked() creates and opens a linked stream and returns IStreamVB interface 348. CreatePropertyStgO creates and opens a property storage and returns IpropertyStorageVB interface 414. OpenPropertyStgO opens a property storage and returns IpropertyStorageVB interface 414. SetClassO sets the CLSLD for the storage. RegisterAlias() registers a pluggable protocol. DestroyO destroys the specified elements. getJElementType() is a read-only function that returns the type of the element. B. POLESS Classes
Figures 10A and 10B illustrate the main class of POLESS 300, the interfaces used to implement the classes, and the flow of the creation of streams 424 and storages 426. cFileRoot class 400 is the first object instantiated and is used to create a new or open an existing a POLESS file. cStorageRoot class 406 is returned, which is a slightly overloaded version of cStorage class 410. From cStorageRoot class 406 creates or opens cStream class 408 and cStorage class 410, from which any streams or storages and sub-storages of those can be created or opened, respectively. For instance, cStorage class 410 creates cPropertyStorage class 412, which creates storage for property sets. The classes implement interfaces that perform operations and/or define attributes that further define the function or properties of the class. A complete description of POLESS 300 classes and interfaces is included in Appendix B.
1 ) cFileRoot Class cFileRoot class 400 is the root POLESS class and controls the creation and opening of all POLESS files. cFileRoot class 400 is generally instantiated first before any other POLESS objects can be created, although other sequences are possible. cFileRoot class 400 implements LFileRoot interface 401 , which is collocated in Figure 10 with cFileRoot class 400. LFileRoot interface 401 is used to open one file at a time and is not released until all other storage object 426, stream object 424, and property storage interfaces are released and the file is ready to be closed. cFileRoot class 400 and IRoot interface support the following operations, for example: StorageFileCreate(); StorageFileOpen(); CryptoGetQ; bStorageFile(); StorageAmalgamatedGet(); DeltaFileCreate();DeltaFileApply();
GetObjectFromPathO; CreateStreamFromBSTR(); MemoryStreamFromStream(); GetPicture(); and SavePicture().
StorageFileCreateO creates a new storage file, returns the root storage to interface, marks the new structured storage file as a POLESS file by storing the class ID ("CLSID") of this class in a stream in the root storage. StorageFileOpen() opens an existing storage file and returns the root storage interface. CryptoGet() gets a default configured crypto class and should be set and used on the open or create of the storage file. bStorageFile() returns true if the file provided is an OLLE structured storage file and not a POLESS storage file. StorageAmalgamatedGet() gets an empty small cStorageAmalgamated class 404. DeltaFileCreate() creates a POLESS difference file by comparing the original POLESS file to the updated POLESS file. DeltaFileApplyO applies a POLESS delta file and applies the original POLESS file to the delta file to create an updated POLESS file. GetObjectFromPathO uses monikers to retrieve the object named by the path and returns a pointer to the object retrieved. CreateStreamFromFile() creates a structured storage stream and populates it with the contents of the file. CreateStreamFromBSTR() creates a structures storage stream and fills it with the specified string. MemoryStreamFromStream() is used to copy a stream to a newly created memory stream object. GetPicture() loads a picture from stream object 424. SavePicture() saves the picture into the stream 426.
2) cCrypto Class cCrypto class 402 controls the configuration of the encryption/decryption of POLESS 300. cCrypto class 402 has the following attributes, for example: sProviderName; eProviderType; sContainerName; and sPassword. SProviderName represents the name of CSP 334 being used to perform the encryption/decryption services. eProviderType is the type of CSP 334. The field of cryptography is large and growing. There are many different standard data formats and protocols. These are generally organized into groups or families, each of which has its own set of data formats and way of doing things. Even if two families used the same algorithm, for example, the RC2 block cipher, they would often use different padding schemes, different key links, and different default modes. Crypto API is designed so that a CSP provider type represents a particular family. sContainerName is the key name and must be provided by cCrypto class 402. sPassword is an optional password on the public/private key pair and can only be entered by a human operator. The password can be used for review disks and their resource files. cCrypto class 402 implements ICrypto interface 401 and they support the following properties and method, for example: ProviderName; Password; FileType; Algorithm; EnumProviders(); and EnumAIgorithms(). Get_ProviderName() returns the name of the Crypto provider. Put_ProviderName() sets the name of the Crypto provider. Get_Password() and Put_Password() are only used for sponsor resource files. Get_FileType() gets the file type and put_FileType() sets the file type. Get_Algorithm() gets the encryption algorithm and put_Algorithm() sets the encryption algorithm. EnumProviders() returns an enumerator for the list of installed providers. EnumAlgorithmsO enumerate a list of algorithms for the current provider.
3) cStorageAmalgamated Class cStorageAmalgamated class 404 is an implementation of IStorage interface 344. cStorageAmalgamated class 404 holds references to an ordered collection of IStorage objects. When a stream is opened, cStorageAmagalmated class 404 searches the collection of storage objects in order to find the first storage object that has the requested stream and returns this stream. cStorageAmalgamated class 404 handles compound storage resolution and delegates all other work to cStorage class 410. cStorageAmalgamated class 404 is, for example, read-only. cStorageAmalgamated class 404 will not allow stream or storages to be created but is primarily for reading exam resource file 120. cStorageAmalgamated class 404 implements IStorage Amalgamated interface 405. cStorageAmalgamated class 404 and IStorageAmalgamated interface 405 support the following operations, for example: StorageAdd(); Clearstorage(); OpenStorageAmalgamated(); and OpenPropertyStgAmalgamated(). StorageAdd() adds a new storage to the collection of storages.
ClearstorageQ clears all the storage objects from the collection. OpenStorageAmalgamated() opens a sub-storage of the current amalgamated storages in an amalgamated fashion.
OpenPropertyStgAmalgamatedO opens a property storage of the current amalgamated storages in an amalgamated fashion. Amalgamation is described in greater detail, in the co-pending application filed on the same date, entitled "EXTENSLBLE EXAM LANGUAGE (XXL) PROTOCOL FOR COMPUTER BASED TESTING," incorporated herein by reference.
4) cStorageRoot Class cStorageRoot class 406 is the POLESS implementation of IStorage interface 344 and IRootstorage interface 346. cStorageRoot class 406 handles any storage object 426 that is POLESS specific and then delegates work to the cStorage class 410. IRootstorage interface 346 supports the SwitchToFile() operation, which copies the current file associated with the storage object to a new file, which is then used for the storage object and any uncommitted changes. cStorageRoot class 406 also implements IPersistFile interface 418, which provides methods that permit an object to be loaded from or saved to a disk file, rather than a storage object or stream. Because the information needed to open a file varies greatly from one application to another, the implementation of IPersistFile: :Load on the object preferably also open its disk file. IPersistFile interface 418 inherits its definition from IPersist, so all implementations must also include the GetClassID() method of IPersist interface 418.
5) cStream Class cStream class 408 is the POLESS implementation of IStream interface 340. cStream class 408 handles any storage object 426 that is POLESS specific and then delegates work to compound document implementation OLE2 320. The specific work includes compression/decompression and encryption/decryption of stream object 424.
IStream interface 340 supports the following operations, for example: Seek(); SetSize();CopyTo(); Committ(); Revert(); LockRegion(); UnlockRegion(); Stat(); and Clone(). Seek() changes the seek pointer to a new location relative to the beginning of stream object 424, the end of stream object 424, or the current seek pointer. SetSize() changes the size of stream object 424.
CopyTo() Copies a specified number of bytes from the current seek pointer in stream object 424 to the current seek pointer in another stream object 424. Commit() ensures that any changes made to a stream object 424 open in transacted mode are reflected in the parent storage object. Revert() discards all changes that have been made to a transacted stream since the last call to IStream: .Commit. LockRegionO restricts access to a specified range of bytes in stream object 424. Supporting this functionality is optional since some file systems do not provide this operation. UnlockRegion() removes the access restriction on a range of bytes previously restricted with IStream: :LockRegion. Stat() retrieves the STATSTG structure for the stream object 424. Clone() creates a new stream object that references the same bytes as the original stream but provides a separate seek pointer to those bytes. IStreamVB interface 348 is an automation friendly version of IStream interface 340.
IStreamVB interface 348 supports the following operations, for example: Read(); WriteQ; ClearQ; Reset(); get_sName(); get_oStream; and CopyTo(). Read() reads data from stream object 424. Write() writes data, including the entire byte array, to stream object 424. Clear() clears stream object 424 of all data. Reset() resets the position in stream object 424 to the beginning of stream object 424.
Get_sName() returns the name of the stream. Get_oStream() returns the IDispatch interface. CopyTo() copies the contents of a source stream to a destination stream.
6) cStorage Class cStorage class 410 is the POLESS implementation of IStorage interface 344 and IcStorage interface 41 1. cStorage class 410 handles any storage object 426 that is POLESS specific and then delegates work to compound document implementation OLE2 320. IStorage interface 344 supports the following operations, for example: CreateStream();
OpenStreamO; CreateStorage(); OpenStorage(); CopyTo(); MoveElementTo(); Commit(); Revert(); EnumElementsO; DestroyElement(); RenameElement(); SetElementTimes(); SetClassO; SetStateBits(); and Stat(). CreateStream() creates and opens a stream object 424 with the specified name contained in a storage object. OpenStreamO opens an existing stream object 424 within a storage object using specified access permissions. CreateStorage() creates and opens a new stream object 424 within a storage object. OpenStorage() opens an existing storage object 426 with the specified name according to the specified access mode. CopyTo() copies the entire contents of an open storage object 426 into another storage object. The layout of the destination storage object may differ from the layout of the source storage object. MoveElementTo() copies or moves a sub-storage or stream object 424 from one storage object 426 to another storage object.
CommitO reflects changes for a transacted storage object 426 to the parent level. Revert() discards all changes that have been made to the storage object 426 since the last IStorage::Commit operation. EnumElementsO returns an enumerator object that can be used to enumerate storage objects 426 and stream objects 424 contained within a storage object. DestroyElement() removes the specified storage object 426 or stream object 424 from a storage object. RenameElement() renames the specified storage object 426 or stream object 424 in a storage object. SetElementTimes() sets the modification, access, and creation times of the indicated storage element, if supported by the underlying file system. SetClassO assigns the specified CLSID to a storage object. SetStateBits() stores state information in a storage object, for example up to 32 bits. Stat() returns the STATSTG structure for an open storage object.
IStorageVB interface 350 is an automation friendly version of IStorage interface 344. IStorageVB interface 350 supports the following operations, for example: Clear(); Commit(); Revert(); sElementNameO; bstorage(); bElement(); CreateStream(); OpenStreamO; Createstorage();Openstorage(); get_sName(); getoStorage(); get_nCount(); GetCompression(); GetEncryptionO; GetCRC(); CreateStreamLinked(); CreatePropertyStg(); OpenPropertyStgO; SetClassO; Register Alias(); DestroyO; and get_ElementType(). Clear () clears the storage of all elements, e.g. sub-storages and streams. Commit() ensures that any changes made to a storage object opened in transacted mode are reflected in the parent storage. For non-root storage objects in direct mode, this method has no effect. For a root storage, it reflects the changes in the actual device, for example, a file on disk. For a root storage object open in direct mode, the commit() method is always called prior to releasing the object. Commit() flushes all memory buffers to the disk for a root storage in direct mode and will return an error code upon failure. Although releasing the object also flushes memory buffers to disk, it has no capacity to return any error codes upon failure. Therefore, calling releasing without first calling commit() causes indeterminate results. Revert() discards all changes that have been made to the storage object since the last Commit() operation. sElement() returns the name of the element. bstorage() returns true if the element is a sub- storage. bElement() returns either iStreamVB interface 412 or iStreamVB interface 414 or IStorageVB interface 412 for the selected element. CreateStream() creates and opens a stream object with the specified name contained in the storage object. Nothing is returned if the stream cannot be created. OpenStreamO opens an existing stream object within this storage object in the specified access mode. Nothing is returned if the stream cannot be opened. Createstorage() creates and opens a new storage object nested within the storage object. Nothing is returned if the storage cannot be created.
Openstorage() opens an existing storage object with a specified name in the specified access mode. Nothing is returned if the storage cannot be opened. Get_sName() returns the name of the storage. Get_oStorage() returns the IDispatch interface, which exposes objects, methods and properties to programming tools and other applications that support Automation. COM components implement the IDispatch interface to enable access by Automation clients, such as Visual Basic.
Get_nCount() returns the count of elements in the storage. GetCompression() determines if streams may be compressed in the file and if enabled streams may optionally be compressed when created. GetCRC() indicates whether a cyclic-redundancy-check ("CRC"), or a digital signature, check is to be performed on the file. CreateStreamLinked() creates a link to a stream in another POLESS file. CreatePropertyStgO creates a property storage. OpenPropertyStgO opens a property storage.
SetClassO assigns the specified CLSID to a storage object. Register Alias() registers an alias to a storage in the POLESS file for access by the pluggable protocol. DestroyO destroys the specified element. Get_ElementType() is a read-only command that returns the type of the element.
7) cPropertyStorage Class cPropertyStorage class 412 implements LPropertyStorage interface 413, which supports the following operations, for example: ReadMultiple(); WriteMultiple(); DeleteMultiple();
ReadPropertyNamesO; WritePropertyNames(); DeletePropertyNames(); SetClassO; Commit();
Revert(); Enum(); Stat(); and SetTimes(). ReadMultiple() reads property values in a property set.
WriteMultipleO writes property values in a property set. DeleteMultiple() deletes property values in a property set. ReadPropertyNamesO gets corresponding strung names fro given property identifiers.
WritePropertyNamesO creates or changes string names corresponding to given property identifiers.
DeletePropertyNamesO deletes string names for given property identifiers. SetClassO assigns a CLSID to a property set. Commit() flushes or commits changes to a property storage object, as is done with the command IStorage: .Commit, described previously. Revert() discards all changes made since the last commit call when a property storage is opened in transacted mode. Enum() creates and gets a pointer to an enumerator for properties within a property set. Stat() receives statistics about a property set. SetTimes() sets modification, creation, and access times for a property set.
LPropertyStorageVB interface 414 is an automation friendly version of LPropertyStorage interface 413 that manages the persistent properties of a single property set. IPropertyStrorageVB interface 414 supports the following operations, for example: ReadVB(); Write VB(); Delete();
CommitVBO; Revert VB(); SetClassO; get_nCount(); CopyTo(); GetName(); WriteMultiple(); and ReadMultiple(). ReadVB() reads the value of a specified property from the property set. Write VB() writes a value for a specified property to the property set. If the property does not exist the property/value pair will be created. If the property already exists, the value will be updated if opened in eAccess_Write mode. Delete() removes a property from the property set. Commit VB()flushes or commits changes to a property storage object, as is done with the command IStorage: :Commit, described previously. RevertVB() discards all changes made since the last commit call when a property storage is opened in transacted mode. SetClassO assigns the specified CLSLD to a property storage object. Get_nCount() returns the count of properties in the property set. CopyTo() copies the contents of the source property set to a destination property set. GetName() returns the name of the specified property. WriteMultiple()writes property values in a property set. ReadMultiple()reads property values in a property set.
8) cPropertyStorageAmalgamated Class cPropertyStorageAmalgamated class 416 implements IPropertyStorageAmalgamated interface 417, which supports the following operations, for example: PropertyStorageAdd() and ClearStorage(). PropertyStorageAdd() adds a property set to the collection of property sets. ClearStorage() clears the collection of property sets.
C. POLESS Exam Resource File
Figures G and H1-H14 illustrate the POLESS layout of exam resource file 120 according to the present invention. Exam resource file 120 stores the various pieces of compiled information from exam source files 130, as shown in Figure 5. Exam resource file 120 contains all of the content required to deliver the test. However, where the test is media-intense, exam resource file 120 will contain the core elements for the test with "links" to the external content. XXL compiler 140 and plugins 150 store the compiled information to exam instance file 120 using one of LPersistResourceStream interface 192a, IPersistResourceSet interface 192b, or LPersistResourceStore interface 192 to store the compiled information as a stream of data, a set of data, or a storage element, respectively. In a preferred embodiment, the layout of exam resource file 120 is in a hierarchical POLESS format that directly implements the format of the XXL test definition language. The test developer uses the XXL test definition language to create the logic files 230 and data files 212 (Figure 5) of exam source file 130.
By having a storage structure that follows the format of the XXL test definition language, the incremental access aspect of POLESS is easily implemented. XXL compiler 140 determines the storage location in exam resource file 120 that stores a particular piece of compiled information, even information stored into exam resource file 120 by one of plugins 150.
Figure 11 illustrates the main storage branches of exam resource file 120, which corresponds to the top-level elements of the XXL test definition language, denoted by reference numeral 500. The main storage branches of exam resource file 120 are, for example: exams branch 550; forms branch
600; items branch 650; category branch 700; templates branch 750; sections branch 800; groups branch 850; plugins branch 900; data branch 950; formGroups branch 1000; attributes branch 1050; scripts branch 1100; and message box ("Msgbox") branch 1150. Other storage branches may altematively be used.
Exam branch 550, as seen in Figure 12, stores, for example, the primary attributes, properties, and data that govern the test. Exam branch 550 can store information for various tests, as is denoted by the three, vertical ellipses. A specific test is identified by the data stored in name attribute storage 552. Again, the various tests may each be identified, for example, by a different name, as denoted by the solid border around name attribute storage 552 or other identification scheme. Attributes storage 554 stores, for example, version information 555, and title information 556 of the test as a stream of data. Title information 556 is optional, as is denoted by the broken border. Any optional, customized information regarding the test is stored in custom properties 558 as a property storage. Information relating to the forms of the test are optionally stored in forms property storage 560. A form is a fixed or substantially fixed order of testing events. Many different forms can be stored in forms storage 560, giving flexibility to test driver 110 in controlling progression of the test. FormGroups storage 562 optionally stores information relating to a collection of exam forms as a stream of data. Preferably, a single form from the formGroup is chosen to deliver to an examinee. The slection of the form from the group is performed by a selection plugin 160. Exam branch 550 preferably contains at least one forms storage 560 either independently or within formGroups storage 562. Other information relating to the test may be stored under exam branch 550. Other storage formats may optionally be used.
Forms branch 600, as seen in Figure 13, stores, for example, the primary attributes, properties, and data that govern the progress of the test. Forms branch 600 can store information for various forms, as is denoted by the three, vertical ellipses. As described previously, a form is a fixed or substantially fixed order of testing events. A single form is identified by the data stored in name attribute storage 602. Other identification formats may optionally be used. Again, the various forms may each be identified by a different name, as denoted by the solid border around name attribute storage 602. Attribute storage 604 stores, for example, begin section information 605, end section information 606, event information 607, and optionally stores version information 608, title information 609, skip allowed information 610, restartable information 611, with information 612, height information 613, and bit depth information 614. All information stored in attribute storage 604 is stored as a stream of data or other data storage format. Begin section information 605 and end section information 606 indicates, for example, respectively which section of the test begins and ends the test.
Event information 607 indicates, for example, the order of events of the test for that form. Each event has a name and is prefixed with an event type and a colon. Other formats are optional. The event type includes "section", "report", and "results". Version information 608 and title information
609 indicate the version and title of the form, respectively. Skip allowed information 610 indicates, for example, whether or not by default skipping of sections is allowed. Restartable information 61 1 indicates, for example, whether the form can be restarted. Any optional, customized information regarding the form is stored in custom storage 616 as a property set or other data storage format. Timer storage 628 stores, for example, information relating to how the form is to be timed as a storage element. Attributes storage 630 stores, for example, the names of Timer Plugin 158 to be used with the form. Plugin data storage 632 and plugin data storage 633 store any data necessary for timer plugin 158 as a storage element and a stream of data, respectively. Plugin data storage 632 and plug in data storage 633 are optional. Scoring storage 634 stores, for example, information relating to the scoring of the form. Attributes storage 636 stores, for example, the name of scoring plugin 164 to be used with the form. Plugin data 638 and plugin data 639 optionally store any data needed for scoring Plugin 164 as a storage element and a stream of data respectively.
Items Branch 650, as seen in Figure 14, stores, for example, the primary attributes, properties, and data that govern the items, or test questions, to be delivered to the examinee during the test. Items branch 650 can store information for various items, as is denoted by the three, vertical ellipses. A single item is identified by the data stored in name attributes storage 652. Again, the various items may each be identified by a different name, as denoted by the solid border around name attributes storage 652. Attributes storage 654 stores, for example, weight information 654, scored information 655, and optionally stores, for example, skip allowed information 656, title information 657, start information 658, finish information 659, and condition information 660. Weight information 654 indicates, for example, a value used for judging and scoring the item. In one embodiment, by default an item is given a weight of one in accordance with one embodiment, but other values may be utilized. Scored information 655 indicates, for example, whether or not the item is scored as opposed to whether the item is being used as an example. The default of scored information 655 is true. Skip allowed information 656 indicates, for example, whether the examinee can skip the item without answering.
Start information 658 indicates, for example, script execution at the beginning of the item and finish information 659 indicates, for example, script execution at the end of the item. Condition information 660 indicates, for example, whether or not there is a condition on the item being delivered to the examinee. The information stored in attributes storage 654 is stored as a stream of data or other data storage format. Data storage 662 and data stream 664 store any information regarding the properties of the item. For example, data storage 662 or data stream 664 can store the correct answer of a multiple choice item. Data storage 662 and data stream 664 stored the information as a storage element and a stream of data respectively. Any optional, customized information regarding the item is stored in customs storage 666 as a stream of data or other data storage format. Category storage 668 stores, for example, information relating to each category to which the item belongs. The information stored in category storage 668 preferably and optionally is redundant, as category branch 700 stores, for example, all the items within the specific categories. The reason for the optional redundancy is so that test driver 110 can quickly look up the category of any item.
Category branch 700, as seen in Figure 15, stores, for example, the primary attributes, properties, and data that govern the test categories. A test category provides a grouping mechanism, which is independent of delivery of the test, allowing for exotic reporting and scoring if necessary. Category branch 700 is optional as denoted by the broken border. Category branch 700 can store information for various categories, as is denoted by the three, vertical ellipses. A single category is identified by the data stored in name attributes storage 702. Again, the various categories may each be identified by a different name, as denoted by the solid border around name attributes storage 702. Attributes storage 704 stores, for example, complete information 705, duplicates information 706, contents information 707, and optionally stores, for example, description information 708. Complete information 705 indicates, for example, whether or not every item in the category must appear within the category or within its subcategories. Duplicates information 706 indicates, for example, whether the item can appear more than once within the category or within the subcategories. Contents information 707 determines what can exist within a category. Description information 708 is used within the category to contain a description of the category's contents. Category storage 710 stores, for example, information relating to any subcategories under the category identified in name attribute storage 702. Items storage 712 indicates, for example, any items that exist within the category. Sections storage 714 contains information indicating what any sections that exist within the category. Scoring storage 716 contains information relating to the scoring of the items within the category. Attributes storage 718 stores, for example, the name of the scoring plugin to be used with the item. Data storage 720 and data stream 722 contain the information needed to initialize scoring plugin 164. Data storage 720 and data stream 722 store the information as a storage element and a stream of data respectively. Templates branch 750, as seen in Figure 16, stores, for example, the primary attributes, properties, and data that govern the templates used in the test. Template branch 750 can store information for various main templates, as is denoted by the three, vertical ellipses. A single main template is identified by the data stored in name attributes storage 752. Again, the various templates may each be identified by a different name, as denoted by the solid border around name attributes storage 752. Attributes storage 754 stores, for example, split information 756, order information 757, and optionally stores, for example, size information 759. Split information 656 defines how a specific area within the template is to be split or separated, for example, either by rows or columns. Size information 759 indicates, for example, possible values for describing the size of the template, for example, pixels, percentages, or html syntax. Template storage 760 stores, for example, information relating to any sub-templates to be used under the templates specified by the information in name attributes storage 752. Sub-templates are identified by the information in name attributes storage 762. Many sub-templates 760 can exist as denoted by the three vertical ellipses.
Areas storage 764 indicates, for example, information relating to the areas used within the template denoted by the information in name attributes storage 752. Many areas may exist within a template as denoted by the three vertical ellipses. Each area is identified by the information stored in name attribute storage 766. Attribute storage 768 stores, for example, visible plugin name information 760, size information 770, and allow more information 771. Plugin name information 760 indicates, for example, the name of the visible plugin to be used with the area. Size information 770 indicates, for example, the size of the area, as for example a pixel value, a percentage value, or HTML syntax. Plugin data 772 and plugin data 774 store information relating to the visible plugin to be used in the area. The data stored in either plugin data storage 772 or plugin data stream 774 is executed by the visible plugin when the template is loaded. Plugin data storage 772 and plugin data stream 774 stores, for example, the information as either a storage element or a stream of data, respectively. Other information may optionally be stored.
Section branch 800, as seen in Figure 17, stores, for example, the primary attributes, properties, and data that govern test sections. Test sections dictate the navigation and timing of groups of items as well as displays within the test. Sections branch 800 can store information for various sections, as is denoted by the three, vertical ellipses. A single section is identified by the data stored in name attribute storage 802. Again, the various sections may each be identified by a different name, as noted by the solid border around name attributes storage 802. Attributes storage 804 stores, for example, group information 805 and optionally stores, for example, title information 806, skip allowed information 807, start information 808, finish information 809, and condition information 810. Group information 805 indicates, for example, to which group of the test the section belongs. Skip allowed information 807 indicates, for example, whether or not the items within the section may be skipped. Start information 808 indicates, for example, script execution at the beginning of the section and finish information 809 indicates, for example, script execution at the end of the section. Condition information 810 indicates, for example, any conditions that exist regarding the section. Any optional, customized information regarding this section is stored in custom property storage 812 as a stream of data or other data storage format. Custom attributes will be stored as a property set. The "key" for each attribute will be a string.
Timer storage 814 stores, for example, information regarding the timing of the section. Attribute storage 816 stores, for example, information identifying timer plugin 158, which is to be used with a section. Plugin data storage 818 and plugin data storage 820 stores, for example, data needed for timer plugin 158. Plugin data storage 818 and plugin data storage 820 stores, for example, information as a storage element and a string of data or other acceptable format respectively. Navigation storage 822 stores, for example, information relating to the delivery of presentations and groups within the section. Attributes storage 824 stores, for example, information indicating which navigation plugin 162 is to be used with this section. Plugin data storage 826 and plugin data stream 828 store information needed for the navigation plugin 162. Plugin data storage 826 and plugin data stream 828 store the information as a storage element and a stream of data respectively. Groups branch 850, as seen in
Figure 18, stores, for example, the primary attributes, properties, and data that govern the groups within the test. A group determines the order of events within the test. Groups branch 850 can store information for various groups, as is denoted by the three, vertical ellipses. A single group is identified by the data store in name attributes storage 852. The various groups may each be identified by a different name, as noted by the solid border around name attributes storage 852. Attributes storage 854 stores, for example, type information 855, event information 856, title information 857, and reviewed name information 858. Type information 855 indicates, for example, whether the group is either a "group holder" (group of presentations), or a "section holder" (group of sub-sections). These are mutually exclusive. Event information 856 indicates, for example, the order of events within the test. Review name information 858 indicates, for example, whether or not a presentation within the group is to be used as a review screen. Any optional, customized information regarding the group is stored in custom storage 860 as a stream of data or other data storage format. Events storage 862 stores, for example, event information as is described in further detail in Figure 19. Scoring storage 864 stores, for example, information relating to the scoring of items within the group. Attributes storage 866 stores, for example, information indicating which scoring plugin 164 is to be used with the group. Selection storage 872 stores, for example, information relating to the selection of items within the group. Attributes storage 874 indicates, for example, which selection plugin 160 is to be used with the group. Figures 19A, 19B, 19C, and 19D illustrate the events sub-branch of groups branch 850 in greater detail in accordance with one embodiment of the invention. In Figure 19A, events sub-branch 862 can store information for various events. For example, events sub-branch 862 is storing information in events name sub-branch 880, event name sub-branch 890, and event name sub-branch 897. Attributes storage 881, in Figure 19B, under events name storage 880 stores, for example, type information 882, template information 883, and optionally stores title information 884, counted information 885, start information 886, finish information 887, and condition information 888. Type information 882 indicates, for example, whether the event is an item or a display. Template information 883 indicates, for example, which template is being used with the event. Counted information 885 indicates, for example, whether a presentation should be included in the totals of presentations presented to the examinee in a section. Generally, presentations with items, or questions, are counted and introductory presentations are not counted.
Start information 886, finish information 887, and condition information 888 indicates, for example, start, finish, and conditional scripts respectively. Any optional, customized information regarding the event is stored in custom storage 889. The "key" for each custom attribute will be a string. Referring again to Figure 19A, event name storage 890 indicates, for example, a different event, which contains different attributes. Additionally, area information 891, in Figure 19B, indicates, for example, which area is rendering the presentations content and item information 892 indicates, for example, the name of the associated item if the event is of the item type. Additionally, data storage
893, data stream 894, data storage 895, and data storage 896 contain information used in a nested presentation. The data off of a nested presentation are the contents of the item or the presentation. This data may be a stream, a storage, a link to a stream, a link to a storage, or other format. In Figure 19C, event name 897 indicates, for example, another event, which includes a sub-event 898, in Figure 19D. Plugins branch 900, as seen in Figure 20, stores, for example, the primary attributes, properties, and data that govern any plugins 150 used for the test. Plugins branch 900 can store information for various plugins, as is denoted by the three, vertical ellipses. A single plugin is identified by the data stored in name attribute storage 902. A CLSID is stamped with the name of the plugin 150. Attributes storage 904 stores, for example, information identifying the plugin 150 by a program LD. Data storage 906 stores, for example, the data, for example, as either a stream, set of data, or as a storage element if plugin 150, respectively.
Data branch 950, as indicated in Figure 21, stores, for example, any global data needed for the test. Data stored optionally under data branch 950 may be stored as either a storage element or a stream of data as indicated by data storage 952 and data storage 954. Data stored under data branch 950 may be directly used by a plugin 150 or the data may be resources (.gif, jpeg, .wab, .mpeg, etc.) used internally by a plugin 150.
FormGroups branch 1000, as seen in Figure 22, stores, for example, the primary attributes properties and data that govern the formGroups of the test. FormGroups branch 1000 can store information for various formGroups, as is denoted by the three, vertical ellipses. A single formGroup is identified by the data stored in name attributes storage 1002. The various formGroups may each be identified by a different name, as denoted by the solid border around name attributes storage 1002. Attributes storage 1004 stores, for example, information indicating which forms are to be used within the formGroup. Selections storage 1006 stores, for example, information relating to the selection of items within the formGroup. Attributes storage 1008 indicates, for example, which selection plugin 160 is to be used with the formGroup. Plugin data storage 1010 and plugin data storage 1012 store any information needed for the selection plugin 160. Attributes storage branch 1050 stores, for example, attribute information that is global to exam resource file 120. This includes the last execution state of XXL compiler 140 [sMode], the major [iXXLMaj or Version] and the minor version [iXXLMinorVersion] of the XXL language. Scripts branch 1100 stores, for example, information relating to scripts used within the test.
Attributes storage 1 102 stores, for example, type information that specifies which type of language the script is in. For example, VB script of J script. Scripts storage 1104 stores, for example, global scripts used within the test that may be referenced by the test driver. MsgBox branch 1150 stores, for example, information relating to the size and content of any message boxes that may be delivered to the examinee during the test. Message boxes may be triggered by plugins 150 during the exam.
D. POLESS Exam Instance File Figures 26A, 26B, 26C, and 26D illustrate the POLESS layout of exam instance file 170 according to the present invention. Exam instance file 170 stores, for example, information regarding the current examinee' s test. Exam instance file 170 is created when a test starts for an examinee. Exam instance file 170 is destroyed when the test successfully completes. If the examinee must restart her test due to some interruption, for example, a power failure, the state of the test is restored from Exam instance file 170. In a preferred embodiment, the layout of exam instance file 170 is in a hierarchical POLESS format. As seen in Figure 26A, the top-level storage branches of exam instance file 170 from root 1200 are, for example: running branch 1202; contents branch 1310; and history branch 1320. Root 1200 relates to POLESS cStorageRoot class 406 (Figure 26), which instantiates exam instance file 170. Running branch 1202 stores, for example, the state information of all running objects in test driver 110 and plugins 150. Plugins 150 use one of LPersistlnstanceStream interface 196a,
IPersistlnstanceSet interface 196b, or LPersisflnstanceStore interface 196c to store information to exam instance file 170 as a stream of data, a set of data, or a store of data, respectively. Any of plugins 150, except display plugin 152, results plugin 166, report plugin 168, and helm plugin 154, which do not contain examination state information, store examination state information to exam instance file 170. Test driver 110 determines the storage location in exam instance file 170 that stores, for example, a particular piece of examination state information.
Exam sub-branch 1204 contains examination state information relating to the exam. Contents storage 1206 stores, for example, exam status information 1207 and version information 1208. Exam status information 1207 indicates, for example, the status of the exam, for example, initializing or terminating. Template storage branch 1210 stores, for example, examination state information relating to templates running in the exam. Name attribute storage 1212 stores, for example, count information 1214 and observed ever information 1215. Observed ever information 1215 indicates, for example, whether or not the template's content has ever been fully seen by the examinee.
Form storage branch 1216 contains information relating to the forms used within the exam. Contents storage branch 1218 stores, for example, seconds information 1219, date start information 1220, date finish information 1221 , current section information 1222, and version information 1223. Current section information 1222 indicates, for example, the current section being delivered to the examinee in the form. Version information 1223 indicates, for example, the identification of the form. Sections chosen storage branch 1224, as illustrated in Figure 26B, stores, for example, information relating to sections in the form being delivered to the examinee. Contents storage 1226 stores, for example, the names of the sections that have been or will be delivered to the examinee. Name attribute storage 1228 indicates, for example, the name of a particular section. Contents storage 1230 stores, for example, current child information 1231, seconds information 1232, date start information 1233, and date finish information 1234. Navigation storage 1236 and navigation storage
1237 store the state information of navigation plugin 162. Navigation storage 1236 stores, for example, the examination state information from navigation plugin 162 if navigation plugin 162 implements the LPersistlnterfaceSet 196b or LPersistlnterfaceStore 196c. Navigation storage 1237 stores, for example, the information from navigation plugin 162 if navigation plugin 162 implements
IPersistlnterfaceStream 196a. Timers storage 1238 and timers storage 1239 store information from timer plugin 158. Timer storage 1238 is used if timer plugin 158 implements IPersistlnterfaceSet 196b or LPersistlnterfaceStore 196c. Timers storage 1239 is used if timer plugin 158 uses IPersistlnterfaceStream 196a.
Items chosen sub-branch storage 1240 stores, for example, information relating to items that have been or will be delivered to the examinee. Contents storage branch 1242 stores, for example, the names and order of all the items that have been or will be delivered to the examinee. Name attributes storage 1244 indicates, for example, the identification of a particular item. Contents storage branch 1246 stores, for example, presented information 1244, complete information 1248, skipped information 1249, seconds information 1250, dehydrated information 1251, and observed ever information 1252. Presented information 1247 indicates, for example, whether the item has ever been delivered to the examinee. Completed information 1248 indicates, for example, whether or not the item has been completed. Skipped information 1249 indicates, for example, whether the item has been skipped. Item plugin storage 1254 and item plugin storage 1255 stores, for example, examination state information from item plugin 156. Item plugin storage 1254 is used if item plugin 156 uses IPersistlnterfaceSet 196b or LPersistlnterfaceStore 196c. Item plugin storage 1255 is used if item plugin 156 uses iPersist interface 196A.
In Figure 26C, item light storage 1256 exists only if the item was dehydrated (to save memory or when a section ends). The dehydrated item stores the data but actions on the data are no longer available until the item is re-hydrated. Item light storage 1256 stores, for example, score candidate information 1257. Score minimum information 1258, score nominal information 1259, score maximum information 1260, complete information 1261, skipped information 1262, correct answer display 1263, response results 1264, and correct answer results 1266. Timers storage 1268 and timers storage 1269 store information from timer plugin 158. Timer storage 1268, as seen in Figure 26B, is used if timer plugin 158 implements IPersistlnterfaceSet 196b or LPersistlnterfaceStore 196c. Timers storage 1269 is used if timer plugin 158 uses IPersistlnterfaceStream 196a. Score storage 1270 and Score storage 1271 store information from timer plugin 158. Timer storage 1270 is used if timer plugin 158 implements LPersistlnterfaceSet 196b or LPersistlnterfaceStore 196c. Score storage 1271 is used if timer plugin 158 uses IPersistlnterfaceStream 196a.
In Figure 26C, groups chosen sub-branch storage 1272 indicates, for example, which groups have been or will be delivered to the examinee. Contents storage 1274 stores, for example, the names of the groups. Name attributes storage 1276 indicates, for example, the name of a particular group.
Contents storage 1278 stores, for example, names of groups and the order of groups. Scoring storage
1280 and scoring storage 1281 store examination state information from score plugin 164. Scoring storage 1280 is used if score plugin 164 implements IPersistlnterfaceSet 196b or LPersistlnterfaceStore 196c. Scoring storage information 1281 is used if score plugin 164 implements IPersistlnterfaceStream
196a. Selection storage 1282 and selection storage 1283 store information from selection plugin 160.
Selection storage 1282 is used if selection plugin 160 implements IPersistlnterfaceSet 196b or
LPersistlnterfaceStore 196c. Selection storage 1283 is used if selection plugin 160 implements
IPersistlnterfaceStream 196a. Delivered storage 1284, in Figure 26D, stores, for example, an ordered list of groups chosen for delivery. Delivered storage 1285 stores an ordered list of the sub-classes of the form, for example: sections, reports and results.
Presentations chosen storage sub-branch 1286 indicates, for example, any presentations that have been or will be delivered to the examinee. Contents storage 1288 stores, for example, the names of the presentations. Names storage sub-branch 1290 stores, for example, the name of the presentation. Names storage 1290 also stores, for example, comment information 1291, marked information 1292, count information 1293, name information 1294, observed ever information 1295, name information 1296, and observed ever information 1297. Name information 1294 and observed information 1295 relate to the name of the first presentation area stored under presentations chosen sub-branch 1286 and whether or not the presentation has ever been observed, and name information 1296 indicates, for example, the last presentation area that was delivered to the examinee and whether or not the presentation was ever observed. Contents storage 1298 stores, for example, information leading to events. Contents storage 1298 stores ready information 1299 ever checked information 1300, ever started information 1301, and ever finished information 1302. Ready information 1299 indicates, for example, whether the event is ready to be delivered to the examinee. Ever checked information 1300 indicates, for example, whether an event's conditional delivery script ever been checked. Preferably, the conditional delivery script is only checked once. Ever started information 1301 indicates, for example, whether the event was ever started by the examinee. Ever finished information 1302 indicates, for example, whether the event was completed by the examinee.
Referring again to Figure 26A, contents branch 1310 stores, for example, a property set containing information to identify the examination instance and the examination start count 1312. The identifying information used is the examinee appointment identification 1311, the name 1313 of exam resource file 120, and the name 1314 of the specified form or group.
History branch 1320 is a single stream of chronological text messages that logs the history of the test. These text messages are used by staff at system headquarters to diagnose problems that occurred in the field. Each text message is prefixed with the date, time, and a level of severity, for example: information, warning, or error. Test driver 110 will filter the text messages to a level of diagnostics desired for test driver 110, such as determining errors in test driver 110 or detail history tracking, including general information.
V. Expansion of Test Driver Using Plugins
Figure 27 illustrates the process for customizing test based on specific requirement from the client using plugins 150, denoted generally by reference numeral 1400. First, the client presents the new requirements, for example, a new item type, to the test developer, step 1402. The test developer then writes and XML schema to define the XXL test specification, step 1404. The schema is subsequently used to validate the XXL test specification. An example of the XXL schema is as follows:
< ! - - [ l inear_navigate - schema . xml ] _ - - >
i _ .
<!-- <linearNavigate>
<!-- ATTRIBUTE REQ? DESCRIPTION
- ->
<!-- initial-Review no [true] (false)
Whether a candidate may -->
<! -- review items from the very beginning of a -->
< ii _ _ section.
- - > <!-- markAllowed no [true]
(false) Whether a candidate may mark items during the exam for_review i _ . purposes. -->
< ! -- incompleteEndAllowedno [true] (false) Whether a candidate may --> i _ _ end a section that contains incomplete -> i _ . items
->
< ! - - endSectionPrompt no The message to disply when ending a section.
<!-- endlncompleteSectionPrompt
no The message to display when ending a - - >
1 - - section with incomplete items .
< ! - - qui ExamPrompt no The message to disply when qui ing an exam.
< ! - - comment no [false] (true) If the candidate can
I _ - make comments during this section. <!-- readonly no [false]
(true) If the items are set to -->
<!-- be readonly. --> <!-- nextOrMore no [true] (false)
Whether to show "Next" -->
<!-- button with "More" button --> i _ .
<!-- SUB-ELEMENTS
- - > none
NOTES
<!-- - Non-adaptive navigation plug- in.
- -> <!-- - Allows for simple "movement" between items and sections -->
<!-- - For "markAllowed" to have an effect a_helm which supports marking -->
<!-- of items must be used in the exam too. <!-- - The button labels will appear exactly_as entered including -->
<!-- capitalization.
- -> <!-- - It is a common case to set comment="true" and readOnly="true" and -->
<!-- re-deliver a section for the sole purpose of commenting. -->
- - >
<ElementType name="linearNavigate" order="many" content= "empty" model="closed" >
<AttributeType name= " initialReview" dt : type="enumeration" dt :values="true false" default=" true" required="no" /> <AttributeType name= "markAllowed" dt : type="enumeration" dt :values="true false" default="true" required="no" />
<AttributeType name="incompleteEndAllowed" dt : type="enumeration" dt :values="true false" default="true" required="no" />
<AttributeType name="endSectionPrompt " dt :type=" string" required="no" default="This will end your section. Do you wish to end?"/>
<AttributeType name="endIncompleteSectionPrompt " dt : type="string" required="no" default="You have not fully answered all items. If you end incomplete items will be marked as incorrect. Do you wish to end?"/>
<AttributeType name="quitExamPromp " dt : type=" st ing" required="no" default="You are about to exit the exam. Do you wish to exit?"/>
<AttributeType name= "comment" dt : type="enumeration" dt :values="true false" default="false" required="no"/>
<AttributeType name=" readonly" dt :type="enumeration" dt :values="true false" default=" false" required="no"/> <AttributeType name="nextOrMore" dt :type="enumeration" dt :values="true false" default="true" required= "no" />
<attribute type="initialReview"/> <attribute type= "markAllowed" /> <attribute type="incompleteEndAllowed"/>
<attribute type= "endSectionPrompt " /> <attribute type="quitExamPrompt "/> <attribute type="endlncompleteSectionPrompt " /> <attribute type= "comment" /> <attribute type="readonly"/>
<attribute type=nnextOrMore"/> </ElementType>
The above sample schema is defining the attributes and elements associated with the top-level XXL element "linearNavigate." A more detail description of the XXL schema is given in the co-pending application filed on the same date, entitled "EXTENSIBLE EXAM LANGUAGE (XXL) PROTOCOL
FOR COMPUTER BASED TESTING," incorporated herein by reference.
The test developer next writes the appropriate plugin 150, in this example, item plugin 156.
The test developer also implements the IPlugin interface 167 and IPlugin interface and Iltem interfaces 169. Additionally, the test developer implements IPersistResource interface 192 (Figure 3) to enable persistence of compiled test information from item plugin 156 to exam resource file 120. The test developer can optionally implement fPersistlnstance interface 196 (Figure 3), step 1408, to enable persistence of examination state information from item plugin 156 to exam instance file 170. After the appropriate interfaces have been implemented, item plugin 156 is valid and operating. Finally, after the test is delivered to the examinee, the result processor accumulates results from the examinee, 1410.
The results processor must be able to understand the new item type to correctly process the results. Customization process 1400 only required the test developer to write one piece of software, item plugin
156, to accommodate the client's customizations rather than multiple pieces of software.
A. Plugin Life Cycle
Figure 28 illustrates the life cycle of plugin 150 from test production to test delivery, denoted generally by reference numeral 1420. Dashed vertical line 1422 divides the plugin life cycle 1420 into a test production cycle, to the left of dashed vertical line 1422, and a test delivery cycle, to the right of dashed vertical line 1422. The test production cycle occurs only occasionally when new plugins 150 are developed to satisfy the requirements of a client. The test delivery cycle occurs whenever the test is delivered to the examinee, for example, daily.
Exam source files 130, of which data files 132 and XXL files 134 are shown, contain every aspect of the test as written by the test publisher. In step I, XXL compiler 140 reads from XXL files 134 and interprets instructions that call for the use of a plugin 150. Plugin 150 is identified in the XXL test definition language by both a name and a program identification ("prog ID"). When XXL compiler receives the prog ID from XXL files 134, XXL compiler knows that a plugin 150 is required to complete the compilation of exam source files 130. An example of plugin 150 calls in XXL is as follows:
<?xml version="l .0" ?>
<xxl version="l .16" xmlns= "x-schema : c : \UTDSDK\xxl -master- schema .xml ">
This contains all of the plugins used for this exam.
I - . < ! - - TIMERS - - >
<plugin name="clockTime" progid="UTD . StandardTimer"/>
<!-- SCORING -->
<plugin name="testScore" progid="UTDP. ScoreTable"/> < ! - - RESULTS - - >
<plugin name="testResults" progid="slsOutputPlugin. cOutputResults"/>
<!-- NAVIGATIONS -->.
<plugin name="refNav" progid="REF. cNavigation"/> <plugin name="linearNav" progid="UTDP. cLinearNavigate"/>
<!-- SELECTIONS -->
<plugin name=" sequential" progid="UTDP . SequentialExhaustive"/>
<!-- DISPLAYS --> <plugin name=" label" progid="REF.udLabel "/>
<!-- ITEMS -->
<plugin name="hotArea" progid="hotArea.hotAreaItem"/>
<plugin name="multi" progid="UTDP .MultiChoiceItem"/>
< ! - - HELMS - - > <plugin name="backForward" progid="REF.udBackForward"/> </xxl>
The ten plugins defined in the previous example represent eight different types of plugins 150. Not all of the possible types of plugins 150 are required to build any one test. Also, more than one plugin 150 is implemented for a specific type. In the above example, two navigation plugins 162 and two item plugins 156 are defined. XXL compiler 140 reads information from exam source files 130 using IStream interface 340, iNode interface 1424, and IStreamVB interface 348. XXL compiler 140 instantiates the requested plugin 150 using, for example, the call CoCreatelnsance. CoCreatelnstance () creates a single, uninitialized object of the class associated with a specified CLSID, using a prog ID that has been converted into the CLSID.
If the data referring to plugin 150 has been customized by the test developer, XXL compiler 140 may not recognize the new data. Therefore, XXL compiler 140 passes the data directly to plugin 150 and plugin 150 loads the data into a private memory (not shown). In one embodiment, the private memory is internal to plugin 150, and in another embodiment, the private memory is external to plugin
150. Plugin 150 can then validate the data using the XXL schema. If the data is invalid, plugin 150 reports the error. In an alternative embodiment, plugin 150 can validate the data using an XML document type definition ("DTD"). A DTD is a formal description in XML Declaration Syntax of a particular type of document. Similar to a schema, a DTD sets out what names are to be used to the different types of elements, where they may occur, and how they all fit together. However, the XXL schema is preferred for validation since schemas are easier to read than a DTD and are very flexible.
If plugin 150 declares that the data is valid, XXL compiler 140 prepares a POLESS storage object 300 in exam resource file 120 to which plugin 150 saves the data at a command from XXL compiler 140, in step II. As described previously, XXL compiler 140 determines where the data from plugin 150 is to be saved in exam resource file 120 and creates the appropriate storage location. The name, CLSLD, and data associated with plugin 150 is stored in plugins branch 900 in exam resource file 120 (Figure 20). Plugin 150 implements IPersistResource interface 192 to store the data to exam resource file 120. Data storage 906 stores, for example, the data as either a set of data or as a storage element if plugin 150 implements either IPersistResourceSet interface 192b or LPersistResourceStore interface 192c, respectively. Data storage 908 stores, for example, the data as a stream of data if plugin 150 implements LPersistResourceStream interface 192a. Plugin 150 can choose the format used to store the data into exam resource file 120. Steps I and II are repeated until exam source files 130 are completely compiled and exam resource file 120 is completely populated with the compiled test information.
The compile sequence of a plugin 150, as shown in steps I and LI in Figure 28, are illustrated in greater detail in Figure 29. Plugin compile sequence 1430 begins as XXL compiler 140 asks plugin 150 to validate the information from exam source files 130 that pertain to plugin 150 using IPlugin: :ValidateSource() call 1432, in step I. Plugin 150 validates whether or not the data received from exam source files 140 is correctly formatted based on the XXL schema. If the data is not valid, plugin throws, for example, a structured COM error. Plugin 150 does not validate that all required source elements are present, but rather, that what is present is correctly formatted.
Step II contains two steps, indicated as step Ila and Lib. In step Lla, XXL compiler 140 creates the appropriate storage element in exam resource file 120 using POLESS object 300. The storage element type is determined based on the type of IPersistResource interface 192 that plugin 150 implements, for example: LPersistResourceStream interface 192a; IPersistResourceSet interface 192b; or LPersistResourceStore interface 192c. XXL compiler 140 then calls IPersistResource* ::Save() call 1434 for the appropriate IPersistResource interface. Plugin 150 saves the compiled information from exam source files 130 to exam resource file 120 through the POLESS object 300 passed by XXL compiler 140. In step lib, XXL compiler 140 instructs plugin 150 to unload, or flush, its content using Unload() call 1436. As stated previously, steps I, Ila, and lib are repeated until all of exam source files 130 is compiled.
Step VI, which is shown as steps Via and VIb, concerns amalgamation of exam resource file
120. Amalgamation enables data for a specific plugin to exist virtually as one storage location even if the data appears at different locations within the storage hierarchy. Amalgamation can be performed on exam resource file 120 if plugin 120 has implemented either IPersistResourceSet interface 192b or
LPersistResourceStore interface 192c which storing data to exam resource file 120. In step Via, XXL compiler 140 amalgamates one to three storage elements in exam resource file 120 and passes the amalgamated POLESS object to plugin 150 using LPersistResource*::ValidateResource() call 1438. Plugin 150 determines whether or not the amalgamated POLESS object creates a complete and valid set. Plugin 150 throws, for example, a structured COM error if the amalgamated POLESS object does not create a complete and valid set. In step VIb, XXL compiler 140 instructs plugin 150 to unload, or flush, its content using Unload() call 1440. Steps Via and VIb are interspersed among steps I, Ila, and lib cycles and can also occur multiple times during the compilation of exam source files 130. Amalgamation will be explained below in greater detail.
Referring again to Figure 28, during the test delivery cycle, test driver 1 10 reads the test specifications stored in exam resource file 120 through POLESS objects 300. Test driver 110 reads information from exam resource file 120 through POLESS objects 300 in order to retrieve the encrypted, compressed, and structured elements within exam resource file 120. When the XXL test definition language calls a plugin 150 by a prog ID, as described previously, test driver 110 instantiates the plugin 150 that was called, in step III. Test driver 1 10 provides the POLESS object 300 from exam resource file 120 and plugin 150 initializes itself from the POLESS object 300, for example, data storage 906 or data storage 908 stored under name attribute storage 902, using the appropriate IPersistResource interface 192. The information loaded into plugin 150 is the same information as was stored into exam resource file 120 by plugin 150 during the test production cycle (step II). Since plugin 150 chose the storage format used to store the information into exam resource file 150, plugin 150 can always read the information from exam resource file 150, giving plugin 150 complete flexibility. Test driver 1 10 need not be able to read the information that is used by plugin 150. Therefore, any customizations to the test facilitated by plugin 150 does not require any changes to test driver 110. The test then progresses with plugin 150 enhancing the functionality of test driver 110 based on the new requirements from the client.
Periodically, based on a request either from test driver 1 10 or from plugin 150, the state of all running objects will save to exam instance file 170, which is a unique file for each examinee, indicating the progress and the status of the test for that examinee. Test driver 1 10 asks plugin 150 if plugin 150 is "dirty," meaning that plugin 150 is storing has some updated examination state information. For example, when the examinee selects distractor A on a multi-choice item, item plugin 156, in this case, becomes dirty. If plugin 150 is dirty, test driver 1 10 provides plugin 150 a POLESS object 300 in exam instance file 170 and plugin saves the examination state information to exam instance file 170 using
LPersistlnstance interface 196, in step LV. For example, item plugin 156 saves the examinee's answer to item plugin storage 1254 or to item plugin storage 1255 (Figure 26). Item storage 1254 stores, for example, the data as either a set of data or as a storage element if item plugin 156 implements either LPersistlnstanceSet interface 196b or LPersistlnstanceStore interface 196c, respectively. Item storage
1255 stores, for example, the data as a stream of data if item plugin 156 implements
LPersistlnstanceStream interface 196a.
Step V occurs if the test is interrupted, for example, because of a power failure, and the test needs to restart. When test driver 110 is required to return to a particular operation state, test driver 110 reads the examination state information from exam instance file 170. Plugin 150 is provided the storage object containing the state of plugin 150 as saved in step IV using LPersistlnstance interface 196. Using the previous example, item plugin 156 retrieves its state information from item plugin storage 1254 or for item plugin storage 1255. Plugin 150 is able to become operational from the retrieved state information, enabling a restart of the test from the point at which the test was interrupted. The delivery sequence of a plugin 150, as shown in steps II, IV, and V in Figure 28, are illustrated in greater detail in Figures 30A, 30B, 30C, and 30D. As seen in Figure 30A, delivery sequence 1520 particularly relates to visible plugins 150, e.g., display plugin 152, helm plugin 154, and item plugin 156. Step III contains sub-steps labeled Ilia - IJIb. Plugin delivery sequence 1520 begins, in step Ilia, when the current delivering presentation requests its template to activate with cTemplate: :Activate() call 1524. Activate() call 1524 is activated when the examinee navigates on the test using a helm navigation control activated by helm plugin 154. IContainerNotifyHelm interface 206 allows helm plugin 154 to request navigation from test driver 110. IContainerNotifyHelm interface 206 sends Activate() call 1524 to cTemplate class 236 in test driver 110 (see Figure 8).
In step Illb, cTemplate class 236 in test driver 110 uses IPlugin: :Load() call 1526 to set the core object references from test driver 110 into the plugin 150 being delivered. The core object references include IContainerNotify interface 200, the cExam class (not shown), and the LAppointment interface 176, which passes information regarding the examinee and appointment to plugin 150.
Step V, which is interspersed with step III, occurs only if the test is interrupted and plugin 150 loses state. cTemplate class 236 in test driver 110 uses LPersistlnstance* ::Reload() call 1528 to call on the reload method of exam instance file 170. Exam instance file 170 reloads plugin 150, through LPersistlnstance interface 192, for example, LPersistlnstanceSet 192b, with the state saved to the appropriate storage location in exam resource file 170 (see Figure 26).
Step IIIc is performed for both initial delivery of plugin 150 and during restart of the test, in conjunction with step V. cTemplate class 236 in test driver 110 uses LPersistResource*: :Load() call 1530 to call on the load method of exam resource file 120. Exam resource file 120 loads plugin 150, through IPersistResource interface 192, for example IPersistResourceSet interface 192b, with the test specification and content from the appropriate storage location in exam resource file 120. Plugin 150 is loaded with test specification and content from exam resource file 120 when being initially delivered to the examinee. Plugin 150 is also loaded with test specification and content from exam resource file 120 and with examination state information from exam instance file 170, as described above, when the test has been interrupted and plugin 150 must recover state. After plugin 150 is properly loaded, cTemplate class 236 in test driver 1 10 uses,
I*::PresentationStaιting() call 1532 (continued in Figure 30B) to inform visible plugin 150 that the presentation is starting, in step Hid. I*::PresentationStarting() call 1532 is made to any visible plugin
150 being used in the presentation on the appropriate interface, for example: LDisplay interface 169a,
Iltem interface 169c, or LHelm interface 169b. For example, an ILtem: :PresentationStarting() call is used for item plugin 156. cTemplate class 236 then instruct visible plugins 150 to display using I01eObject::DoVerb(Show,...) command 1534, step ILIe. IOleObject interface 1522 is the Active Document interface used to implement the Active Document presentation. (See Figure 3-1). IOleObject interface 1522 is the combination of the Active Document interfaces described in conjunction with Figure 7. After instructing visible plugins 150 to display, test driver 110 awaits notification from each visible plugin 150 that the specific visible plugin 150 has successfully shown. Visible plugins 150 call back to test driver 110 using IContainerNotify ::Activated() call 1536, step Illf (continued in Figure 30B). Now, the presentation is started and active such that the examinee can interact with the presentation.
The deactivation of the presentation begins with a request from the helm for navigation. For example, if the examinee has finished a question and wishes to move on to the next question on the next presentation, the examinee can choose the "NEXT" button on the helm. The navigation request is sent from LHelm interface 169b, which receives the request from the examinee, to test driver 1 10 using IContainerNotifyHelm interface 206. As seen in Figure 30D, the request is made using IContainerNotifyHelm: :RequestMove() call 1538, step Illg. Test driver 110 then asks each item plugin 156 being used in the presentation template if the examinee is allowed to leave the current presentation and to proceed to the next presentation. The query is made using Iltem: :bProceed() call 1540, step ILIh. If all item plugins 156 respond in the affirmative, test driver 150 passes the navigation request to navigation plugin 162, which is an invisible plugin 150. Test driver 110 passes the request using INavigate::RequestMove() call 1542, step ILIi. Navigation plugin 162 determines the resultant location of the requested navigation. In Figure 30, for example, navigation plugin 162 determines the section of the test to which the examinee will proceed using ISection: :ChildNext() call 1544, step IIIj.
The active presentation then instructs the template to deactivate using cTemplate: :Deactivate() call 1546, step IILk (continued in Figure 30C). Referring back to Figure 30D, cTemplate class 236 in test driver 110 requests that visible plugins 150 hide from the Active Document using I01eObject::DoVerb(Hide,...) call 1548, step ILΪ1. cTemplate class 236 in test driver 110 informs visible plugins 150 that the current presentation is ending using I*::PresentationEnding() call 1550, step ILIm. For example, cTemplate informs helm plugin 154 that the current presentation is ending using the
IHel m: : PresentationEndingO cal 1.
Step IV, which contains sub-steps LVa-c, is the process to save plugin state data to exam instance file 170. Test driver 110 requests the "dirty" state of plugin 150 to determine whether plugin 150 is storing any state information that would be necessary if the test were to be interrupted. Test driver 110 uses LPersistlnstance*: :IsDirty() call 1552 to make the request, step LVa. For example, test driver 110 uses LPersistlnstanceSet: :IsDirty call 1552 if the state data is a property set. If plugin 150 is storing state data that is not already stored in exam instance file 170, _PersistInstance*::IsDirty() call
1552 returns true. If plugin 150 is "dirty", test driver 110 instructs plugin 150 to save the state data to exam instance file 170 in the POLESS object provided (Figure 26) using LPersistLnstance*::Save() call
1554, step LVb. Finally, test driver 110 instructs plugins 150 to unload all objects using
LPlugin::Unload() call 1556, step IVc.
B. Amalgamation of Plugins in Exam Resource File
The test publisher defines common plugin 150 test specification and common plugin 150 test content in early XXL elements in XXL files 134. These previous elements can be referenced by later XXL elements in XXL files 134. When XXL files 134 are initially compiled into exam resource file 120, the relational hierarchy of the XXL elements is maintained. However, through amalgamation, these separate elements are brought together, or amalgamated, into one, virtual storage location in exam resource file 120. The storage location is virtual because the storage location does not physically exist but acts identically to a physical storage for read-only purposes. If the later element defines nothing more then plugin 150 will receive just the common specification and content.
Optionally, the later elements can define omitted non-common plugin 150 specification and contents. Further, the later elements can override previously defined common plugin 150 specification and contents with exception plugin 150 specification and contents. This process of new definition and overriding of specification and contents by later XXL elements is not limited to two stages. Multiple later stages are allowed.
When test driver 110 uses plugin 150 to deliver the test, plugin 150 uses the specification and contents to determine how and what to deliver. The specification and contents used by plugin 150 is the amalgamation of all relevant XXL elements in exam resource file 120. The amalgamation is invisible to plugin 150 and requires no special effort for plugin 150 during delivery. The amalgamation occurs at delivery time and therefore common specification and content appear only once in the compiled exam resource file 120. Thus, the size of the exam resource file 120 is reduced.
1) Invisible Plugins
The XXL language allows invisible plugins 150 (e.g., timer plugin 158, selection plugin 160, navigation plugin 162, scoring plugin 164, results plugin 166, and report plugin 168) two locations to define test specification and content. These two locations are the element storing identification and initialization data for plugin 150 stored under plugins branch 900 in exam resource file 120 and the storage location containing the information that defines the actual usage of plugin 150 on a unit of the test. Invisible plugins 150 are defined for usage in forms branch 600, sections branch 800, category branch 700, and groups branch 850 in exam resource file 120. Test driver 110 amalgamates these two levels during delivery of the test.
Data storage 906 under plugins branch 900 in exam resource file 120 stores, for example, plugin 150 specification and content. Data storage 906 is the first data location for the amalgamation.
Data storage 908, which stores, for example, a stream of data, is not involved in amalgamation since amalgamation can only be performed on a set of data or a storage element. Below is a sample of plugin 150 elements:
<!-- Navigation -->
<plugin name="controlledNav" progid="AICPAPlugins . controlledNav" />
<plugin name="linearStandard" progid="UTDP. cLinearNavigate"> <data>
<linearNavigate initialReview="true" endIncompleteSectionPrompt="You have not answered all the questions" quitExamPrompt="Click 'Yes' to quit. Otherwise, click
'No' to continue." endSectionPrompt="Are you ready? "/>
</data>
</plugin> <plugin name="linearIntro" progid="UTDP . cLinearNavigate">
<data>
<1inearNavigate initialReview="true" endIncompleteSectionPrompt="Do you wish to begin the exam? " endSectionPrompt="Do you wish to begin the exam?" />
</data> < /plugin>
The XXL source above defines three navigation plugins 162 for use in the test. The first is named "controUedNav" and does not have any test specification or test content defined that is common to all "controUedNav" navigation plugins 162 for this particular test. The second is named
"linearStandard" and does have test specification and test content defined that is common to all "linearStandard" navigation plugins 162 in the test. The third is named "linearlntro" and also has test specification and test content defined that is common to all "linearlntro" navigation plugins 162 in the test. Notice that both "linearStandard" and "linearlntro" utilize the same navigation plugin 162 code component with the prog ID of UTDP.cLinearNavigate. The elements of both "linearStandard" and "linearlntro" contain many of the same attributes, but only the "linearStandard" has the quitExamPrompt defined for it. Also, some of the common attributes have the same values and some have different values, for example, "endSectionPrompt".
Data storage 826 under navigation storage 822 and data storage 818 under timer storage 814 in sections branch 800 in exam resource file 120 also store plugin 150 specification and content. These are the second data locations for the amalgamation. Below is a sample of navigation and timer elements under the section element: < sect ion name= " testlet001 " >
<navigation plugin="linearIntro" /> <timer plugin="timerStandard">
<data>
<stanardTimer timed="false" contributing="true" />
</data>
</timer> <groupRef name="gtestlet001" />
</section>
<section name="testlet002 " >
<navigation plugin= " linearlntro" >
<data> <linearNavigate initialReview="true" quitExamPrompt^"Click 'Yes' to quit the test then, raise your hand." </data>
</navigation>
<timer plugin="timerStandard" /> <groupRef name="gtestlet002 " /> </section>
The XXL source above defines two navigation plugins 162 and two timer plugins 158 used to deliver the test. The first navigation plugin 162 instance, "linearlntro", which was defined in the previous example, is located in the section named "testletOOl". Since no additional data is defined for "linearlntro" under "testletOOl", the section definition language does not override or extend the original common specification and content for "linearlntro".
The second navigation plugin 162 instance, also using "linearlntro", is located in the section named "testlet002". Since "testlet002" does define data for "linearlntro", the new specification and content defined in the section definition language will be amalgamated with the common specification and content originally defined for "linearlntro". (See previous example.) The new specification and content does override the original value for "initialReview" and extends the original common specification by defining a value for "quitExamPrompt". Note, however, that although the initial and later values for "initialReview" are both "true", the later value does override the initial value. If the later value were different than the original value, the value for "initialReview" would change. Figure 31 is a flow chart illustrating the overall process for amalgamation of invisible plugins, denoted generally by reference numeral 1549. A first segment of the test specification and content, stored under plugins branch 900 is validated by the appropriate invisible plugin 150, step 1551. A second segment of the test specification and content, stored in the storage location containing the information that defines the actual usage of the invisible plugin 150, is then validated by the same invisible plugin 150, step 1553. The first segment and the second segment are then amalgamated, step 1555. The amalgamated segment is validated by the invisible plugin 150 to determine whether the amalgamated segment forms a complete and valid set, step 1557. During delivery of the test, the first segment and the second segment are reamalgamated, step 1558, and the reamalgamated segment is delivered to the examinee, step 1559. 2) Visible Plugins
The XXL language allows three locations for visible plugins 150 (e.g., display plugin 152, helm plugin 154, and item plugin 156) to define test specification and content. These first two locations are the element storing identification and initialization data for plugin 150 stored under plugins branch 900 in exam resource file 120 and data storage 772 under area storage 764 in templates branch 750 of exam resource file 120. The third location is the events element 880, which is stored under groups branch 850, in the case of display plugins 152 or helm plugins 156. Alternatively, the third location is or data storage 662 under items branch 650, in the case of item plugins 156. Test driver 110 amalgamates these three locations during delivery by the test.
Data storage 772 under area storage 764 in templates branch 750 stores, for example, visible plugin 150 specification and content for use in a particular template. Data storage 772 is the second data location for the amalgamation for visible plugins 150. Below is a sample of area elements in a template element (note: "helmTitlebar" display plugin 152, "helmNextPrevious" helm plugin 154, and
"itemMultiChoice" item plugin 156 are previously defined in XXL that is not shown):
<template name="multiItem_Template" split="rows"> <area name="titlebar" size="40" plugin="helmTitlebar">
<data>
<titlebarHelm done="false" calc="false" help="false" />
</data>
</area> <area name="item" plugin=" itemMultiChoice" size="*" />
<area name="helm" size="40" plugin= "helmNextPrevious " >
<data>
<nextPrevious bgcolor= "#D4D0C8 " >
<button action="next" img=" .. /images/continue .bmp" /> </nextPrevious>
</data>
</area>
</template>
The XXL source above defines one template named "multiItem_Template". This template contains three areas, named "titlebar", "item", and "helm", and each area references one visible plugin 150 utilized in that area. For example, the area named "titlebar" utilizes "helmTitlebar" display plugin 152. The first and the third area define data in data storage 772 for display plugin 152, "helmTitlebar", and helm plugin 156, "helmNextPrevious", respectively, to be amalgamated with the specification and content stored in data storage 906 under plugins branch 900 for "helmTitlebar" and
"helmNextPrevious". The second area accepts the common item plugin 156 level specification and content for "itemMultiChoice" without further refinement. Data storage 662 under items branch 650 or events element 880 in XXL under groups branch
850 contain plugin 150 specification and content for item plugins 150 or display plugins 152 and helm plugins 156, respectively. These are the final data locations for the amalgamation of visible plugins
150. Below is a sample of item elements: < item name= " 101A" template= "multi Item_Template " area=" item" >
<data>
<multiChoice correctAnswer="A" maxResponses= " 1 " minResponses="l" autoPromp ="false"
URI=" . ,\itembank/itembank.htm#101A"/> </data> </item> <item name="102A" template="multiItem_Template" area="item"> <data>
<multiChoice correctAnswer= "B"
URI=" ..\itembank/itembank.htm#102A"/> </data> </item>
The XXL source above defines two items, "101 A" and "102A". Both items use the same template "multiltem Template" and area "item", both of which were defined in the previous example. Both items define a third level of data for the amalgamation. For example, the item named "101 A" defines two previously undefined attributes: "correctAnswer" and "URI". Item "101 A" also overrides two previously defined attributes: maxResponses, minResponses and autoPrompt. (It should be noted that the data "maxResponses," "minResponses," and "autoprompt" are typically defined in the XXL language initializing a multi-choice item plugin 156 stored in data storage 906 under plugins branch 900. This XXL language is not shown.) The second item, "102A", only defines the two previously undefined attributes. Item "102A" relies on amalgamation for the other attribute definitions, "maxResponses," "minResponses," and "autoPrompt". Figure 32 is a flow chart illustrating the overall process for amalgamation of visible plugins, denoted generally by reference numeral 1560. A first segment of the test specification and content, stored under plugins branch 900 is validated by the appropriate visible plugin 150, step 1561. A second segment of the test specification and content, stored under area storage 764 in templates branch 750, is then validated by the same visible plugin 150, step 1562. A third segment of the test specification and content, stored either under groups branch 850 (for display plugins 152 or helm plugins 156) or under items branch 650 (for item plugins 156), are validated by the same visible plugin 150, step 1563. The first segment, second segment, and third segment are then amalgamated, step 1564. The amalgamated segment is validated by the visible plugin 150 to determine whether the amalgamated segment forms a complete and valid set, step 1565. During delivery of the test, the first segment, second segment, and third segment are reamalgamated, step 1566, and the reamalgamated segment is delivered to the examinee, step 1567.
3) Amalgamation Process
Figure 33 shows the flow chart for amalgamation process 400. There are three basic steps in amalgamation process 400. First, plugin 150 validates partial exam source in exam source files 130, step 1460. Secondly, plugin 150 validates the complete amalgamated test specification and content in exam resource file 130, step 1470. Finally, test driver 110 and plugin 150 deliver the complete amalgamated test specification and content, step 1480. Steps 1460 and 1470 generally occur during exam compilation and step 1480 generally occurs during delivery of the test to the examinee. Of course, other sequences and/or functionality may be used and/or substituted in accordance with the functionality described herein.
To support amalgamation, plugins 150 implement generally either IPersistResourceSet interface 192b or LPersistResourceStore interface 192c. POLESS performs the actual amalgamation on either a property set or a storage element in exam resource file 120. IPersistResourceStream 192a cannot generally be amalgamated in one embodiment of the invention, although other amalgamation possibilities are possible.
Figure 34 illustrates the process for exam source validation, step 1460, in greater detail, in accordance with one embodiment. Plugin 150 receives partial exam source from exam source files 140 through XXL compiler 140, step 1462. XXL compiler asks plugin 150 to validate the partial exam source using IPlugin 150::ValidateSource() call 1432 (Figure 29). Plugin 150 determines whether or not the exam source is valid by comparing the exam source to the XXL schema, as previously described, step 1463. If the partial exam source is not valid, plugin 150 throws, for example, a structured COM error, step 1464, and the compilation is aborted.
Plugin 150 generally validates that what is present in the partial exam source is correctly formatted. Plugin 150 does not generally validate that all required source elements are present, only that what is present is legal. Plugin 150 also does not validate that attributes do not conflict. Conflict between elements cannot yet be determined since the amalgamated final view of plugin 150 data has not yet been established.
If the partial exam source is valid, XXL compiler creates the appropriate storage element via
POLESS, step 1466. The storage element type is determined base on the IPersistResource interface 192 plugin 150 implements. If plugin 150 implements IPersistResourceSet interface 192b, the data will persist to exam resource file 120 as a property set. If plugin 150 implements LPersistResourceStore interface 192c, the data will persist to exam resource file 120 as a storage element. XXL compiler 140 then implements the LPersistResourceStore::Save() call or the LPersistResourceSet::Save() call, both calls being denoted by reference numeral 1434 in Figure 29. Plugin 150 saves the compiled content and specification to exam resource file 120 through the POLESS object passed, step 1468.
Figure 35 illustrates the process for complete amalgamated test specification and content validation, step 1470, in greater detail, in accordance with one embodiment. In step 1470, data stored in exam resource file 120 is amalgamated for validation. The purpose of this step is the validation of the amalgamated vector of storage elements to provide a complete specification and content for plugin 150. XXL compiler 140 calls for creation of an amalgamation storage vector, step 1472, using the calls cFileRoot::StorageAmalgamatedGet() or cFileRoot::PropertystorageAmalgamatedGet(). (See cRoot class 400 in Figure 10.) XXL compiler 140 then adds storage elements or property sets to the amalgamation storage vector, step 1474. These storage elements and sets were previously stored in exam resource file 120 by the plugin 150, step 1468 (Figure 34). XXL compiler 140 will amalgamate, for example, one to three storage elements or property sets depending on the type of plugin 150 at issue. As described above, for invisible plugins 150, XXL compiler 140 will amalgamate up to two storage elements or property sets. For visible plugins 150, XXL compiler 140 will amalgamate up to three storage elements or property sets. XXL compiler 140 uses the cStorageAmalgamated: :StorageAdd() call to add a storage element to a storage amalgamation and the cPropertyStorageAmaIgamated::PropertyStorageAdd() call to add an element to a property set amalgamation. (See cStorageAmalgamated class 404 and cPropertyStorageAmalgamated class 416 in Figure 10.) The data contained in the storage elements or property sets is amalgamated according to the rule of amalgamation. The rule of amalgamation states that a later define element replaces an earlier defined element of the same name, as shown in the previous example of amalgamation for item plugin 156. Alternative amalgamation rules may optionally be used.
XXL compiler 140 determines whether or not the last storage element or property set has been added to the amalgamation storage vector, step 1475. If the amalgamation is complete, XXL compiler 140 passes the amalgamated POLESS object stored in the amalgamation storage vector to the IPersistResourceStore::ValidateResource() call or the IPersistResourceSet: :ValidateResource() call, both calls being denoted by reference numeral 1438 in Figure 29. XXL compiler 140 queries the amalgamation for IStorage interface 344 or LPropertyStorage interface 413. The appropriate interface is passed to plugin 150 in the validation call. Plugin 150 uses this virtual storage or property set to perform the validation on the amalgamated POLESS object.
Plugin 150 validates that the amalgamated POLESS object forms a complete and valid set, step
1478. Otherwise, plugin 150 throws an error, for example, a structured COM error, if the combination of content does not create a complete and valid set. Plugin 150 determines what is required for a complete and valid set. Referring to the previous example of item "101 A", the associated item plugin
156 requires "correctAnswer," "autoPrompt," and "URI" for completeness. Therefore, all of these attributes must be defined within the amalgamation storage vector to form a complete and valid set.
The attributes need not exist at the same level in the exam source so long as they exist in the final amalgamation. "MaxResponses" and "minResponses" are not required for completeness. However, for a valid set, the value assigned to "maxResponses" must be greater than or equal to the value assigned to "minResponses". The validity of these attributes can only be checked on the amalgamation since "maxResponses" and "minResponses" can be defined at different levels in the exam source.
Figure 36 illustrates the process for delivery of amalgamated test specification and content for delivery, step 1480, in greater detail, in accordance with one embodiment. In step 1480, the exam resource data in exam resource file 120 is amalgamated for delivery. The purpose of this step is the delivery of the exam using the amalgamated plugin 150 specification and content. Test driver 110 calls for creation of an amalgamation storage vector, step 1482, using the calls cFileRoot:: storageAmalgamatedGetO or cFileRoot:: PropertystorageAmalgamatedGet(). (See cRoot class 400 in Figure 10.) Test driver 110 then adds storage elements or property sets to the amalgamation storage vector, step 1484.
Test driver 1 10 will amalgamate one to three storage elements or property sets depending on the type of plugin 150 at issue. As described above, for invisible plugins 150, XXL compiler 140 will amalgamate up to two storage elements or property sets. For visible plugins 150, test driver 110 will amalgamate up to three storage elements or property sets. Test driver 110 uses the cStorageAmalgamated: :StorageAdd() call to add a storage element to a storage amalgamation and the cPropertyStorageAmalgamated: :PropertyStorageAdd() call to add an element to a property set amalgamation. (See cStorageAmalgamated class 404 and cPropertyStorageAmalgamated 416 in Figure 10.) The data contained in the storage elements or property sets is amalgamated according to the rule of amalgamation, as described previously.
Test driver 1 10 determines whether or not the last storage element or property set has been added to the amalgamation storage vector, step 1486. If the amalgamation is complete, test driver 110 passes the amalgamated POLESS object to the LPersistResourceStore: :Load() call or the IPersistResourceSet: :Load() call, step 1488 (see Figure N). Plugin 150 can safely assume a full set of data in the amalgamated POLESS object because the amalgamated data was validated in step 1470 (Figure 35). Test driver 110 queries the amalgamation for IStorage interface 344 or LPropertyStorage interface 413. The appropriate interface is passed to plugin 150 in the LPersistResourceStore: :Load() call or the LPersistResourceSet::Load() call. Plugin 150 uses this virtual storage element or property set to enable delivery of the test, never realizing that the POLESS object is an amalgamation.
Figure 37 illustrates three examples of amalgamation of property sets. Property set 1500 contains "prompt" information for an item plugin 156. In the level 1 specification, "prompt" is assigned the value "select best" in data storage 906 under plugins branch 900 in exam resource file 120.
In the level 2 specification, template A and template B are defined, which use item plugin 156.
Template A is used for item "101 A" and template B is used for items "102A" and "103A".
Property set 1502 contains "min" information for item plugin 156 regarding item "101A". In the level 2 specification for item "101A", "min" is assigned the value "1" in data storage 772 under templates branch 750 in exam resource file 120. Continuing with item "101A", property set 1504 contains "correctAnswer" and "URI" information for item plugin 156 regarding item "101 A". In the level 3 specification for item "101 A", "correctAnswer" is assigned the value "A" and "URI" is assigned the value "items.htm#101A" in data storage 662 under items branch 650 in exam resource file 120.
Virtual property set 1506 represents the amalgamated property sets from levels 1, 2, and 3 for item "101A". Virtual property set 1506 contains "min," "prompt," "correctAnswer," and "URI" information for item plugin 156 regarding item "101 A". In the amalgamated POLESS object, "prompt" has the value "select best" from the level 1 specification, "min" has the value "1" from the level 2 specification, and "correctAnswer" has the value "A" and "URI" has the value "items.htm#101A" from the level 3 specification. Item plugin 156 for item "101A" never has a value assigned for the attribute "max".
Property set 1508 contains "min," "max," and "prompt" information for item plugin 156 regarding items "102A" and "103A". In the level 2 specification for items "102A" and "103A", "min" is assigned the value "2," "max" is assigned the value "2," and "prompt" is assigned the value "pick 2" in data storage 772 under templates branch 750 in exam resource file 120. Continuing with item "102A", property set 1510 contains "correctAnswer" and "URI" information for item plugin 156 regarding item "102A". In the level 3 specification for item "102A", "correctAnswer" is assigned the value "B" and "URI" is assigned the value "items.htm#102A" in data storage 662 under items branch 650 in exam resource file 120.
Virtual property set 1514 represents the amalgamated property sets from levels 1, 2, and 3 for item "102A". Virtual property set 1514 contains "min," "max," "prompt," "correctAnswer," and
"URI" information for item plugin 156 regarding item "102A". In the amalgamated POLESS object, "min" has the value "2," "max" has the value "2," and "prompt" has the value "pick 2" from the level 2 specification. It should be noted that the final value for "prompt" was determined from the level 2 specification which overrides the initial value assigned to "prompt" in the level 1 specification, which was "select best". "CorrectAnswer" has the value "B" and "URI" has the value "items.htm#102A" from the level 3 specification. Property set 1512 contains "max," "prompt," "correctAnswer," and "URI" information for item plugin 156 regarding item "103A". In the level 3 specification, "max" is assigned the value "5,"
"prompt" is assigned the value "select best," "correctAnswer" is assigned the value "C," and "URI" is assigned the value "items. htm#103A" in data storage 662 under items branch 650 in exam resource file 120.
Virtual property set 1516 represents the amalgamated property sets from levels 1, 2, and 3 for item "103A". Virtual property set 1516 contains "min," "max," "prompt," "correctAnswer," and
"URI" information for item plugin 156 regarding item "103A". In the amalgamated POLESS object,
"min" has the value "2" from the level 2 specification. "Max" has the value "5" from the level 3 specification. The value assigned to "max" in the level 3 specification overrides the value assigned to "max" in the level 2 specification, which was "2". "Prompt" has the value "select best" from the level 3 specification. The value assigned to "prompt" in the level 2 specification overrides the value assigned to "prompt" in the level 1 specification, which was "select best". However, the value assigned to "prompt" in the level 3 specification overrides the value assigned to "prompt" in the level 2 specification, which was "pick 2", bringing the value for "prompt" back to the value originally assigned in level 1, "select best". "CorrectAnswer" has the value "C" and "URI" has the value "items.htm#103A" from the level 3 specification.
The many features and advantages of the invention are apparent from the detailed specification, and thus, it is intended by the appended claims to cover all such features and advantages of the invention, which fall within the true spirit and scope of the invention. Further, since numerous modifications and variations will readily occur to those skilled in the art, it is not desired to limit the invention to the exact construction illustrated and described, and accordingly, all suitable modifications and equivalence may be resorted to, falling within the scope of the invention.
APPENDIX A-
UTD CLASSES AND INTERFACES
TABLE OF CONTENTS
TABLE OF CONTENTS 2
LOGICAL VIEW REPORT 4
LOGICAL VIEW 4
CComControl 4
CWindowImpl 4
DSADocs.DocDisplay 4
DSADocs.docl tern 4
DSADocs.docReview 4 lAdviseSink 5
IDispatch 5
IOleClientSite 5 lOleDocumentlmpl 5 lOleDocumentSite 5 lOleDocumentViewlmpl 5
IOlelnPlaceFrame 6 lOlelnPlaceObjectWindowlesslmpl 6
IOlelnPlaceSite 6 lOlelnPlaceUlWindow 7
IOleObject 7 lOleObjectlmpl. 8 lOleWindow 8
IPersistStoragelmpl 8
UTDC.Browser 8
UTDC.GraphScores 10
UTDC.TimeRemaining 10
UTDComm.cCommentNavigation 10
UTDCore.Collecύon 10
UTDCore.ICompilerServices 10
UTDCore.cArea 11
UTDCore. Attribute 11
UTDCore.cAttributes 11
UTDCore.cCategories 11
UTDCore.cCategory 12
UTDCore.cEvents 12
UTDCore.cExam 12
UTDCore.cForm 14
UTDCore. cGroup 15
UTDCore.cGroups 16
UTDCore.cItem 16
UTDCore.cltems 16
UTDCore.cMsgBox 17
UTDCore. cPresentation 17
UTDCore. cPresentations 18
UTDCore.cRegistry 18
UTDCore.cReport 19
UTDCore.cResults 19
UTDCore.cScript 19
UTDCore.cSection 20
UTDCore.cSections 27
UTDCore. eDirection 2/
Page 2
WAS HINGTON I 49764V I UTDCore.ePluginModes 27
UTDCore.eScoreStatus 27
UTDCore.eStates 21
UTDCore.iContainerNotify 22
UTDCore.iContainerNotifyHelm 22
UTDCore.iDisplay 22
UTDCore.iHelm 22
UTDCore.iltem 23
UTDCore.iNavigate 23
UTDCore.iPersistlnstanceSet 24
UTDCore.iPersistlnstanceStore 24
UTDCore.iPersistlnstanceStream 24
UTDCore.iPersistResourceSet 25
UTDCore.iPersistResourceStore 25
UTDCore.iPersistResourceStream 25
UTDCore.iPlugin 25
UTDCore. iReport 26
UTDCore. iResults 26
UTDCore.iScore 26
UTDCore. iSelection 27
UTDCore.iUnitTimer 27
UTDP.StandardTimer 28
UTDP.cBrowserDisplay. 28
UTDP.cLinearNavigate 28
UTDP.cMutliChoiceltem 28
UTDP.cNextPrevious 29
UTDP.cSequentialExhastive 29
UTDP.cSummaryScore 29
UTDP.cUserSelection 29
UTDPE.cChallenge 29
UTDPE.cSLSDeliResult 29
UTDPE.cStandardReview 29
UTDPE.cStandardScoreReport. 29
UTDPE.cWTDResults 29 cActivePlugin 29 cContainerFrameT . 29 cDocumentSiteT 29 cEvent 30 cFormGroup 30 cPluglnT. 30 cPlugin 30 cScreenMinimum 30 cTemplate 31 lAppointment 31 ihaunch 31 iLaunch2 31 iPrint 31
ITransfer 31
TOTALS: 32
LOGICAL PACKAGE STRUCTURE 32
Page 3
W AS HING TON 1 49764V 1 LOGICAL VIEW REPORT
CComControl
Derived from CWindowImpI
Public Operations:
CComControl 0 • CComControl
FireOnRequestEdit (dispID : DISPID) : HRESU T
FireOnChanged (dispID : DISPID) : HRESULT
CreateControlWindow (hWndParent : HWND, rcPos : RECT&) : HWND
ControlQuer lnterface (iid : const IID&, ppv : void**) : HRESULT
CWindowImpI
CWindowImpI allows you to create a new window or subclass an existing window
_ Pu_b__lic_ A_ttributes:
Public Operations:
Create 0 :
Creates a window DefWindowProc 0 :
Provides default message processing
Get ndClassInfo 0 :
Returns a static instance of CWndClassInfo which manages the window class information SubclassWindow 0 ■
Subclasses a window
UnsubcIassWindow 0 :
Restores a previously subclassed window WindowProc 0 '■
Processes messages sent to the window GetWindowProc 0 :
Returns the current window procedure GetCυrrentMessage 0 :
Returns the current message OnFinalMessage 0 :
Called after receiving the last message (typically WM_NCDESTROY)
DSADocs.DocDisplay
The DSA introduction and display screens. Modified to support the new interfaces. DSADocs.docItem
The DSA multichoice item. Does graphic and text distracters. Handles voice-overs and
BSL. Modified to support the new interfaces. DSADocs.docReview
The DSA review screen. Modified to support the new interfaces.
Page 4
WA S H INGTON 1 4976 V I IAdviseSink
Public Operations:
OnViewChange (dw Aspect : DWORD, lindεx : LONG) : void
Advises that view of object has changed OnRename (pmk : IMoniker *) : void
Advises that name of object has changed OnSave 0 '• void
Advises that object has been saved to disk OnClose 0 : void
Advises that object has been closed
IDispatch
Base class for all UTDCore COM Interfaces.
IOleClientSite
Public Operations: SaveObject 0 : HRESULT
Saves embedded object. GetMoniker (dwAssign : DWORD, dwWhichMoniker : DWORD, ppmk : IMoniker **) : HRESULT
Requests object's moniker GetContainer (ppContainer : LPOLECONTAINER *) : HRESULT
Requests pointer to object's container ShowObject 0 : HRESULT
Asks container to display object OnShowWindow (fShow : BOOL) : HRESULT
Notifies container when object becomes visible or invisible RequestNewObjectLayout 0 : HRESULT
Asks container to resize display site.
IOleDocumentlmpl
Public Operations:
CreateView (plPSite : lOlelnPlaceSite *, pstm : IStream *, dwReserved : DWORD, ppView :
IOIeDocumentView **) : EnumViews (ppEnnm : lEnumOlePocument Views**, ppView : IOIeDocumentView **) : GetDocMiscStatus (pdwStatus : DWORD *) :
IOleDocumentSite
Public Operations:
ActivateMe (pViewToActivate : IOIeDocumentView *) : HRESULT Activates an OLE Document Object
IOleDocumentViewI pI
Public Operations:
SetlnPlaceSite (plPSite : lOlelnPlaceSite *) :
GetlnPlaceSite (ppIPSite : lOlelnPlaceSite **) :
GetDocument (ppunk : IUnknown **) :
SetRect (prcView : LPRECT) :
GetRect (prcView : LPRECT) :
SetRcctComplex (prcView : LPRECT, prcHScroll : LPRECT, prcVScroll : LPRECT, prcSizeBox : LPRECT)
Show (fShow : BOOL) :
UIActivate (fUIActivate : BOOL) :
Open O :
CloseView (dwReserve : DWORD) :
SaveViewState (pstm : LPSTREAM) :
Apply ViewState (pstm : LPSTREAM) :
Clone (pIPSiteNew : lOlelnPlaceSite*, ppViewNew : IOIeDocumentView**) :
ActiveXOocActivate (iVerb : LONG) -.
Page 5
WASHINGTON U9764V I IOlelnPlaceFrame
Derived from IOlelnPlaceUIWindow
Public Operations:
InsertMenus (hmenuShared : HMENU, IpMenu Widths : LPOLEMENUGROUPWIDTHS) : HRESULT
Allows container to insert menus SetMenu (hmenuShared : HMENU, holemenu : HOLEMENU, hwndActiveObject : HWND) : HRESULT
Adds a composite menu to window frame RemoveMenus (hmenuShared : HMENU) : HRESULT
Removes a container's menu elements SetStatusText (pszStatusText : LPCOLESTR) : HRESULT
Sets and displays status text about EnableModeless (.Enable : BOOL) : HRESULT
Enables or disables modeless dialog boxes TranslateAccelerator (Ipmsg : LPMSG, wID : WORD) : HRESULT
Translates keystrokes
IOlelnPlaceObjectWindowIessImpl
/////////////////////// //////////////////////////////////////////////////// IOIeInPlaceObjectWindowlessIrr.pl
Public Operations:
AddRef ( : void) : ULONG
Querylnterface (riid : const IID&, ppvObject : void**) : HRESULT
Release ( : void) : ULONG
UIDeactivate ( : void) : HRESULT
GetWindow (phwnd : HWND*) : HRESULT
ContextSensitiveHelp ( : BOOL) : HRESULT
InPlaceDeactivate ( : void) : HRESULT
SetObjectRects (prcPos : LPCRECT, prcClip : LPCRECT) : HRESULT
ReactivateAndUndo ( : void) : HRESULT
OnWindowMessage (msg : UINT, wParam : WPARAM, IPara : LPARAM, plResult : LRESULT*) :
HRESULT GetDropTarget ( : IDropTarget**) : HRESULT
lOlelnPlaceSite
Derived from IOleWindow
Public Operations: CanlnPlaceActivate 0 HRESULT
Determines if the container can activate the object in place. OnlnPlaceActivate 0 : HRESULT
Notifies the container that one of its objects is being activated in place. OnUIActivate 0 : HRESULT
Notifies the container that the object is about to be activated in place, and that the main menu will be replaced by a composite menu GetWindowContext (ppFrame : IOlelnPlaceFrame **, ppDoc : IOlelnPlaceUIWindow **, IprcPosRect :
LPRECT, IprcClipRect : LPRECT, IpFramelnfo : LPOLEINPLACEFRAMEINFO) : HRESULT
Enables an in-place object to retrieve window interfaces that form at the window object hierarchy, and the position in the parent window to locate the object's in-place activation window Scroll (scrolIExtent : SIZE) : HRESULT
Specifies the number of pixels by which the container is to scroll the object OnUIDeactivate (fUndoable : BOOL) : HRESULT
Notifies the container to reinstall its user interface and take focus. OnlnPlaceDeactivate 0 : HRESULT
Notifies the container that the object is no longer active in place
Page 6
WASHINGTON I 49764V J DiscardUndoState 0 HRESULT
Instructs the container to discard its undo state. DeactivateAndUndo 0 : HRESULT
Deactivate the object and revert to undo state. OnPosRectChange (IprcPosRect : LPCRECT) : HRESULT
Object's extents have changed
IOlelnPlaceUIWindow
Derived from IOleWindow
Public Operations:
GetBorder (IprectBorder : LPRECT) : HRESULT
Specifies a RECT structure for toolbars and controls. RequestBorderSpace (pborderwidths : LPCBORDERW1DTHS) : HRESULT
Determines if tools can be installed around object's window frame. SetBorderSpace (pborderwidths : LPCBORDERWIDTHS) : HRESULT
Allocates space for the border. SetActiveObject (pActiveObject : lOlelnPlaceActiveObject *, pszObjName : LPCOLESTR) : HRESULT
Provides for direct communication between the object and each document and frame window.
IOleObject
Public Operations:
AddRef ( : void) : ULONG
Querylnterface (riid : const πD&, ppvObject : void**) : HRESULT
Release ( : void) : ULONG
SetExtent (dwDraw Aspect : DWORD, psizel : SIZEL*) : HRESULT
GetExtent (dwDraw Aspect : DWORD, psizel : SIZEL*) : HRESULT
SetClientSite (pClientSite : IOleClientSite*) : HRESULT
GetClientSite (ppClientSite : IOleClientSite**) : HRESULT
SetHostNames ( .- LPCOLESTR, : LPCOLESTR) : HRESULT
Close (dwSaveOption : DWORD) : HRESULT
SetMoniker ( : DWORD, : IMoniker*) : HRESULT
GetMoniker ( : DWORD, : DWORD, : IMoniker**) : HRESULT
InitFromData ( : IDataObject*, : BOOL, : DWORD) : HRESULT
GetCIipboardData ( : DWORD, : IDataObject**) : HRESULT
DoVerb (iVerb : LONG, : LPMSG, : IOleClientSite*, : LONG, hwndParent : HWND, IprcPosRect :
LPCRECT) : HRESULT EnumVerbs (ppEnumOleVerb : EEnumOLEVERB**) : HRESULT Update ( : void) : HRESULT IsUpToDate ( : void) : HRESULT GetUserClassID (pClsid : CLSID*) : HRESULT
GetUserType (dwFormOfType : DWORD, pszUserType : LPOLESTR*) : HRESULT Advise (pAdvSink : IAdviseSink*, pdwConnection : DWORD*) : HRESULT Unadvise (dwConnection : DWORD) : HRESULT EnumAdvise (ppenumAdvise : IEπumSTATDATA**) : HRESULT GetMiscStatus (dwAspect : DWORD, pdwStatus : DWORD*) : HRESULT SetColorSche e ( : LOGPALETTE*) : HRESULT
Page 7
WA S H INGTON 1 49764V I IOleObjectlmpl
Public Operations:
AddRef ( : void) : ULONG
Querylnterface (riid : const IID&, ppvObject : void**) : HRESULT
Release ( : void) : ULONG
SetExtent (dwDraw Aspect : DWORD, psizel : SIZEL*) : HRESULT
GetExtent (dwDraw Aspect : DWORD, psizel : SIZEL*) : HRESULT
SetClientSite (pClientSite : IOleClientSite*) : HRESULT
GetClientSite (ppCHentSite : IOleClientSite**) : HRESULT
SetHostNames ( : LPCOLESTR, : LPCOLESTR) : HRESULT
Close (dwSaveOption : DWORD) : HRESULT
SetMoniker ( : DWORD, : IMoniker*) : HRESULT
GetMoniker ( : DWORD, : DWORD, : IMoniker**) : HRESULT
InitFromData ( : IDataObject*, : BOOL, : DWORD) : HRESULT
GetCIipboardData ( : DWORD, : IDataObject**) : HRESULT
DoVerbPrimary (prcPosRect : LPCRECT, hwndParent : HWND) : HRESULT
DoVerblnPlaceActivate (prcPosRect : LPCRECT, : HWND) : HRESULT
DoVerbShow (prcPosRect : LPCRECT, : HWND) : HRESULT
DoVerbUIActivate (prcPosRec : LPCRECT, : HWND) : HRESULT
DoVerbHide ( : LPCRECT, : HWND) : HRESULT
DoVerbOpen ( : LPCRECT, : HWND) : HRESULT
DoVerbDiscardUndo ( : LPCRECT, : HWND) : HRESULT
DoVerb (iVerb : LONG, : LPMSG, : IOleClientSite*, : LONG, hwndParent : HWND, IprcPosRect :
LPCRECT) : HRESULT EnumVerbs (ppEnumOleVerb : IEnumOLEVERB**) : HRESULT Update ( : void) : HRESULT IsUpToDate ( : void) : HRESULT GetUserClassID (pClsid : CLSID*) : HRESULT
GetUserType (dwFormOfType : DWORD, pszUserType : LPOLESTR*) : HRESULT Advise (pAdvSink : lAdviseSink*, pdwConnection : DWORD*) : HRESULT Unadvise (dwConnection : DWORD) : HRESULT EnumAdvisc (ppenumAdvise : IEnumSTATDATA**) : HRESULT GetMiscStatus (d Aspect : DWORD, pdwStatus : DWORD*) : HRESULT SetColorScheme ( : LOGPALETTE*) : HRESULT
IOleWindow
Public Operations:
GetWindow (phwnd : HWND *) : HRESULT
Gets a window handle. ContextSensitiveHelp (fEnterMode : BOOL) : HRESULT
Controls enabling of context-sensitive help.
IPersistStoragelmpI
Public Operations:
AddRef ( : void) : ULONG
Querylnterface (riid : const πD&, ppvObject : void**) : HRESULT
Release ( : void) : ULONG
GetClassID (pClassID : CLSID*) : HRESULT
IsDirty ( : void) : HRESULT
Load (pStorage : IStorage*) : HRESULT
Save (pStorage : IStorage*, fSameAsLoad : BOOL) : HRESULT
InitNew ( : IStorage*) : HRESULT
SaveCompleted ( : IStorage*) : HRESULT
HaπdsOfϊStorage ( : void) : HRESULT
Private Operations:
IPSI_GetIPersistStreamInit 0 : IPersistStreamlnit*
UTDCBrowser
The UTD browser control is an Active X control that
Page 8
WASH INGTON I 49764V 1 is a customized web browser control that is locked down for security reasons and supports scripting objects from UTDCore
Public Attributes: oRegistry : IRegistry *
Returns the single instance of the registry object. Read-only.
Public Operations: get_AllowPOLESS 0 : gets AllowPOLESS flag put_AllowPOLESS 0 :
Sets the AllowPOLESS flag get_AllowFile 0 : getsAHowFile flag put_AllowFile 0 :
Sets the AllowFile flag get_AllowURL 0 : gets the AllowURL flag put_AllowURL 0 :
Sets the AllowURL flag get_RelativeFontSize 0 : gets the relative font size put_RelativeFontSize 0 : sets the relaitive font size LoadURL 0 : browses to a URL from a string Load fj : loads the browser from a stream get_oExam () : gets the oExam object put_oExam 0 : sets the oexam object get_oSection 0 '• gets the oSection object put_oSection 0 : sets the oSection object get_oPresentation 0 : gets the opresentation object put_oPresentation 0 : sets the opresentation object get_oItem 0 : gets the oltem object put_oItem 0 : puts the oltem object get_oForm 0 : gets the oFrom object put_oForm 0 : sets the form object get_oAppointment 0 : gets the oAppointment object put_oAppointment 0 : sets the oAppointment object get_ScroUBars 0 : gets the Scrollbars flag
Page 9
W ASH INGTON 1 9764V 1 put_ScrollBars 0 : sets the scrollbars get_Document 0 '•
Gets the document boject Show 0 :
Makes the browser visible Hide O :
Hides the browser control Unload 0 :
Unloads the Browser control Refresh 0 :
Refresh the page in the browser control
UTDCGraphScores
The UTD sections scores control. This control displays bar-graph of a score or scores. This control will run if requested from HTML passed to the UTDC.Browser control.
These can be: single score vs. passing,
Score by section, or score per category.
Bruce UTDCTimeRemaining
This is a visible control showing the remaining time. It can be configured to show the reamining time for the section or exam or the minimum of both. It can be configured to flash or prompt with messages when time is expiring. The time points and the messages are configurable.
It should be scriptable from HTML. UTDComm.cCommentNavigation
The navigation puts the test driver into comment period mode and re-navigates the sections also show any presentation in the comment period section. UTDCore. Collection
Base UTD collection. All other collection classes follow this design. Public Attributes: Count : Long
Returns the number of members in a collection. _NewEnum : lUnknown * *
Returns a Visual Basic enumerator object.
Public Operations:
Item (Index : Variant) : Variant
Returns a specific member of a Collection object either by position or key.
UTDCoreJCompilerServices
Provides nifty features to the plugin. Public Attributes: sResourceDirectory : BSTR
Get the directory of the resource file currently being compiled.
Read-only. sSourceDirectory : BSTR
Get the directory of the source file currently being compiled.
Read-only.
Page 10
WASHINGTON 149764V I Public Operations:
ParseHTML (sSourceFile : BSTR) : BSTR
Loads the source file into mshtml, and removes everything in the <body> except the specified labeled <div>.
C:\EXAM\ITEMBANK.HTM: <div id="Schmackaroo"> ... Some HTML here ... </div>
Parse("C:\EXAM\ITEMBANK.HTM#Schmackaroo") returns "...Some HTML here.
UTDCorccArea
The area in the template for a plugin There is no IArea interface. Derived from cDocumentSiteT, UTDCore.iContainerNotify
Public Attributes: bContained : Boolean
If TRUE the plugin is contained in the area as an active document. If FALSE it is a floating window. Usually full screen. nTop : Integer
The top coordinate. Zero if not contained. nLeft : Integer
The left coordinate. Zero if not contained. nWidth : Integer
The width if the contiainer. Zero if not contained. nHeigth : Integer
The heigth if the contiainer. Zero if not contained. oPluginClass : cPlugin
The plugin class for this window. oPluginActive : UTDCoreJPIugin
If the plugin class is active, this is the iPlugin interface to it. sSize : String
The size of the area on the split axis. Can be a percentage, pixel count or *.
UTDCore.cAttribute
Public Attributes: sName : BSTR
Get the attribute name string, value : VARIANT
Get the attribute value.
UTDCore.cAttributes
Collection of UTDCore.cAttribute Derived from UTDCore.Collection
UTDCore.cCategoiϊes
Collection of UTDCore.cCategory Derived from UTDCore.Collection
Page 1 1
WASH INGTON 1 49764V 1 UTDCore.cCategory
A category of sections or items. For example: "Reporting", "Objecti vies", "Scoring"
Public Attributes: sName : String
The name of the category. bComplete : Boolean
If TRUE the category must contain all sections or items defined at a level. bDuplicate : Boolean
If TRUE the category allows sections or items defined more than once at a level. sDescription : String
The description of the category. colMembers : UTDCore.cEvents
The collection of sections or items that apply to this category . coISubCategories : UTDCorccCategories
Collection of UTDCore.cCategory. (AKA sub-categories). eContents : eCategoryContents
Read-only. The allowable types of members in the colMembers. enum { eCategoryContent_Anything = 0, eCategoryContent_Sections = 1, eCategoryContent_Items = 2, eCategoryContent_Categories = 3,
} eCategoryContents;
Private Attributes: oScore : UTDCore.iScore
The scoring plugin for the form.
UTDCorccEvents
IEvents contains deliverable classes as IDispatch objects. They must be queried individually to determine their exact type.
IEvents can be: cExam, cForm, cSection, cltem, cPresentation, cReport, cResults.
Derived from UTDCore.Collection
UTDCore.cExam
The exam instance. The root object of the driver exposed classes. Derived from cContainerFrameT
Public Attributes: sName : String
Read-only exam name. sLanguage : String
The language of the exam content. Read-only. sVersion : String
The version of the exam. Read-only. oActiveForm : UTDCorexForm
The current form. Only one form can be active for an exam. eStatus : eExa Status
The eExamStatus as follows: eExam_Initializing eExam_Starting eExam_InProgress
Page 12
WAS HIN GTON 1 49764V I eExamJEnded eExamJTerminating bShowResponseComment : Boolean
If TRUE the candidate should not see their response during the comment period. colCustomAttributes : UTDCore-cAttributes
Collection of custom attributes. Read-only. sTitle : String
Exam title. Read-only colCategories : UTDCore.cCategories
The collection of all categories. oResourceRoot : Object
The read-only reference to the root storage of the POLESS resource file. Interfaces:
POLESS. cStorageRoot, iStorage, iRootStorage, iStorageVB. olnstanceRoot : Object
The read-only reference to the root storage of the POLESS exam instance file.
Interfaces: POLESS. cStorageRoot, iStorage, iRootStorage, iStorageVB. co-Templates : UTDCore.cTemplates
Collection of all templates defined in the resource file.
Holds flat collection of all templates at Exam level colItemsChosen : UTDCore Items
Return all items chosen by a selection plugin, at any level of the exam tree, in the order that they were chosen. These are items that have been delivered or will be delivered. colAIISections : UTDCore-cSections
Return all sections chosen by a selection plugin, at any level of the exam tree, in the order that they were chosen. These are items that have been delivered or will be delivered. oRegistry : UTDCore.cRegistry
Returns the single instance of the registry object. Read-only.
Private Attributes: oFormGroup : cFormGroup
The form group to select from.
Public Operations:
Minimize 0 •'
Minimumizes the main windows of UTD. Can be called be simulations to get UTD out of the way. Restore 0 '•
Restore the main windows of UTD. Can be called be simulations to restore UTDto the screen. GetScript 0 : UTDCore.cScr.pt
Returns a script object to be used by a plug-in. Savelnstance 0 :
Save the state of the test driver to the instance file. All active objects are asked to serialize themselves to the instance file.
Any one with a reference to this object can initiate this process. It could be an item plugin that wants to save user interaction and state information.
The driver will save between each presentation. GetTransfer 0 : ITransfer
Returns a UAS Transfer object to transfer results or other files to the data center.
The cExam object does not keep a reference to this object. GetPrint 0 : lUASPrint
Returns a UAS print object to save for reprint score and other reports.
The cExam object does not keep a reference to this object. FatalError ((Number : Long, sSource : String, sDescription : String) :
Reports an error in a plugin to the test driver.
Page 13
WASH INGTON | 976 V 1 Normally errors will be reported back to the test driver via an HRESULT and supportting lErrorlnfo. If the error was generated from something other than test driver initiated call, the plugin should report the error via this method.
An example usage could be the user clicks a button on your plugin item. The resulting actions generate an error. Trace (INumbe : Long, sSource : String, sDescription : String) : Reports an event in a plugin to the test driver.
Warn O :
Writes a Warning' level trace messate to the instance file. Info O :
Writes a Information' level trace messate to the instance file. Start 0 :
Tells the exam that the candidate has accept the exam and started. Normally called by the Navigation plugin in response to the eDirection_Start.
The administration system is informed that the exam has started and the candidate has accepted it. Quit 0 :
Tells the exam that the candidate is quitting the exam. Normally called by the Navigation plugin in response to the eDirection_Quit. The navigation plugin should confirm this decision with the candidate.
This causes the test driver to stop delivery and shutdown execution cleanly. The administration system is informed that the candidate quit the exam. GetMsgBox 0 : UTDCore.cMsgBox Returns a message box object.
UTDCore.cForm
A exam form. Derived from UTDCore.iContainerNotifyHelm
Public Attributes: oCurSection : UTDCore.cSection
The current active section in the exam. Read-only. colChildren : UTDCore Events
The collection of all top level sections of the exam. Read-only. colCustomAttributes : UTDCore.cAttributes
Collection of custom attributes. Read-only. sName : String
Read-only form name. sTitle : String
Form title. Read-only colItemsChosen : UTDCore.cItems
The collection of all items chosen (that is, returned by a selection plugin) in the exam.
This is regardless of their section. colAUSections : UTDCore Sections
The collection of all sections of the exam regardless of their level. datStart : Date
The form start date and time. datFinish : Date
The form finish date and time. oTimer : UTDCore.iTimer
The timer for the form. oScore : UTDCore.iScore
The scoring plugin for the form.
Page 14
W ASHIN GTON 9764V 1 ITimerFactor : Single
The multiple for test timer based on ADA conditions. sVersion : String
The version of the form. colDelivered : UTDCore.cEven
The collection of all delivered top level sections of the form. Read-only. nCurChild : Long = 0
Index of last delivered event in colDelivered. Read-only. eStatus : eScoreStatus
Returns the value of oForm.oScore.eStatus. colAllPresentations : UTDCore.cPresentations
The collection of all presentations of the exam regardless of their level. colPresentationsChosen : UTDCore.cPresentations
The collection of all presentations chosen (that is, returned by a selection plugin) in the exam. This is regardless of their section or group. colSectionsChosen :
The collection pf all sections chosen (that is, returned by a selection plugin) in the exam.
This is regardless of their level. colAUItems :
The collection of all items of the exam regardless of their level. colAHGroup :
The collection of all groups of the exam regardless of their level. colGroupsChosen :
The collection of all groups chosen (that is, the selection plugin for this group has been
Reset()) in the exam. This is regardless of their level.
Public Operations: SectionNext 0 :
Requests that the driver proceeds to the next section or score report or result writing or the comment period. SectionPrevious 0 :
Requests that the driver proceeds to the previous section or score report or result writing or the comment period. This is often illegal and will not be allowed by the defined navigation plugin. SectionGoto (vSection : Variant) :
Requests that the driver proceeds to a named or numbered section or score report or result writing or the comment period. datMinRemaining 0 :
Minimum time remaining for this exam.
UTDCore.cGroup
Public Attributes: sName : String
Read-only group name. sTitle : String
Read-only group title. colCustomAttribu.es : UTDCore.cAttributes
Collection of custom attributes. Read-only. colChildren : UTDCore-cEvents
The collection of presentations, sub-sections, or sub-groups for this group. oScoring : UTDCore.iScore
The scoring plugin for the section. colDelivered :
Collection of presentations or sub-sections containing all delivered events within this group. This will not contain sub-groups. oSelection : UTDCore.iSelection
Pointer to the current selection plugin of this group. oSection : UTDCore.cSection
The section housing this group.
Page 1 5
WASHIN GTON 1 4976 V I UTDCore-cGroups
Collection of UTDCore.cGroup Derived from UTDCore.Collection
UTDCore.cItem
The item's general information. Stuff that is not plugin specific. This class implements Icltem and Htem interface. Derived from UTDCore.iltem
Public Attributes: bPresented : Boolean = False
True if item has been presented to die candidate. Set by item component. sName : String
Read-only item name. sTitle : String
Presentation title. Read-only oScriptStart : UTDCorexScript
The script to execute before this object executes. oScriptFinish : UTDCore.cScript
The script to execute after this object executes. oScriptConditional : UTDCore.cScript
This expression script is evaluated. If TRUE this object is deliveried. -Weight : Double = 1
The item weigthing relative to other items. Normally 1. If multiple weigths per item are required they can be attached in the custom attributes. The weigth can be negative. coICustomAUribu.es : UTDCore.cAttributes
Collection of custom attributes. Read-only. (Seconds : Long
The seconds spent on the item. bScored : Boolean = TRUE
Read-only. TRUE if the item is to be scored.
If false fWeight and fScoreCanidate Weighted will all be 0. bSkipAllowed : Boolean
Read-only property. If TRUE the item can be skipped. If TRUE the bProceed() method on iltem interface will not be called. ScoreCanidateWeighted : Double
The weigthed value of the canidate for the item. The judged value for the item will by multiplied by the weight for this value. sPluginName : String
Read-only plugin name. The prog ID of the plugin. oSection : UTDCore-cSection
Read-only property of the section the item was deliveried in. oPresentation : UTDCore-cPresentation
Read-only property of the presentation the item was deliveried in. datElapsed : DATE
The time spent on the item. Same as lSeconds, but a different COM type.
UTDCore.cItems
Collection of UTDCore.cItem Derived from UTDCore.Collection
Page 16
W A SHINGTON 1 49764V 1 UTDCore.cMsgBox
This class displays a message box with a candidate prompt. This class is returned from IExam::GetMsgBox(). The message in the box is supplied by the called. The title can optionally be supplied. The buttons are supplied by the caller, but the button labels are defined at the exam level in the resource file.
The parent of the message box is the main exam window. The message box can be set to time out after a duration.
Public Attributes: sTitle : String = UTDCore
The title the message box will use. Read/Write. ITimeout : Long = 120
The timeout value on the message box in seconds. Read Write.
The Display will return eMsgBoxResult_Timeout. nParentHWnd : Integer
The parent window handle Read/Write
Default is the main UTDCore window.
Public Operations:
Display (sPrompt : String, eStyle : eMsgBoxStyles) : eMsgBoxResult
Display the message box. Displays the buttons requested and the message supplied.
The return value is enum { eMsgBoxResult_OK = 1, eMsgBoxResult_CanceI = 2, eMsgBoxResult_Abort = 3, eMsgBoxResult_Retry = 4, eMsgBoxResult_Ignore = 5, eMsgBoxResult_Yes = 6, eMsgBoxResult_No = 7, eMsgBoxResult_Timeout = 8 } eMsgBoxResults;
The style of message box. enum { eMsgBoxStyle_OKOnly = 0, eMsgBoxStyle_OKCaπcel = 1, eMsgBoxStyle_AbortRetryIgnore = 2, eMsgBoxStyle_YesNoCancel = 3, eMsgBoxStyle_YesNo = 4, eMsgBoxStyle_RetryCancel = 5, eMsgBoxStyle_Critical = 16, eMsgBoxStyle_Question = 32, eMsgBoxStyle_Exclamation = 48, eMsgBoxStyle_Information = 64, eMsgBoxStyle_ApplicationModal = 0, eMsgBoxStyle_SystemModal = 4096,
} eMsgBoxStyles;
UTDCore.cPresentation
The data and current state of the presentation. Could be a item or a display.
Page 17
WASHINGTON I 49764V I Derived from cEvent, UTDCore.iContainerNotifyHelm
Public Attributes: bCountPresentation : Boolean
If TRUE the presentation should count for the total X or Y for the section, "the item count" colCustomAttributes : UTDCore.cAttributes
Collection of custom attributes. Read-only. sName : String
Read-only presentation name. sTitle : String
Presentation title. Read-only oScriptStart : UTDCore.cScript
The script to execute before this object executes. oScriptFinish : UTDCore.cScript
The script to execute after this object executes. oScriptConditional : UTDCorccScript
This expression script is evaluated. If TRUE this object is deliveried. sComment : String
The user entered comment for this presentation. bMarked : Boolean
TRUE if the item has been mark for later review. Set by the helm. colChildren : UTDCore.cEvents
The items in this presentation.
The collection maybe empty. bSelected : Boolean
True if the presentation or any of its items have been selected for delivery by a selection plugin.
Read-only.
Public Operations:
Findlcltem (plltem : Htem *) : Icltem *
Find the Icltem interface for the given Iltem. Returns failure if the Iltem is not in this presentation.
UTDCore.cPresentations
Collection of UTDCore.cPresentation Derived from UTDCore.Collection
UTDCorccRegistry
Alows registry look up for translating strings
For examples:
<base href="%REG:DSAGRAPGHICS"> into
<base href="V:\DSA\">
Looks at [HKEY_LOCAL_MACHINE\SOFTWARE\Prometric\UTD\Keywords]
Page 18
WASH INGTON 1 49764V I Public Operations:
Lookup (sKey : String) : String
Retrieves the string by finding the key in the registry and returning the value. Translate (slnString : String) : String
Translates each occurance of %REG:x% into the equivilent of "x" in the registry.
UTDCore.cReport
A score report "section".
Derived from cEvent
Public Attributes: oScoreReport : UTDCore-iScore
The score report plugin.
UTDCore.cResults
The results writing "section" Derived from cEvent
Public Attributes: oResults : UTDCore. iResults
The plugin that writes the results.
UTDCore.cScript
A script written in VBscript or JScript. To be executed or evaluated by the ActiveX Scripting Engine. This object will expose UTD objects to the script code. These objects will be based on the scope of the "Parent" object provided-to this object.
All scripts can see oExam, oForm, and oCandidate.
If the parent object is a section object, then the script will also see oCurSection.
If the parent object is a presentation object, then the script will also see oCurSection and oCurPresentation.
If the parent object is an item then the script will also see oCurSection, oCurPresentation, and oCurltem. Public Attributes: oParent : Object
The parent UTD object. It could be a form, a section or a presentation.
Public Operations:
Execute (sScript : BSTR) :
Executes the script code passed in the sScript parameter.
EvaluateBool (sScript : BSTR) : Boolean
Evaluates the expression parsed as the sScript parameter. Returns TRUE or FALSE. Raises an error if the code is not an expression returning a boolean.
EvaluateVar (sScript : BSTR) : Variant
Evaluates the expression passed in the sScript parametert. Returns a variant depending on the return type of the script. Raises an error if the code is not an expression.
SetScope (oSection : ISection *, oPresentation : -Presentation *, oltem : Iltem *) :
Set the parent section, presentation, and item. The script will have access to these as the oCurSection, oCurPresentation, and oCurltem objects. Any/All may be passed as NULL parameters indicating that the script does not have access to that particular object.
Page 19
WAS HINGTON l 4976 V I AddObject (sName : BSTR, pObject : IDispatch *) : HRESULT
Add an object into the global script scope.
Protected Operations:
LoadGlobalScript (oStream : POLESS.IcSlorage) :
Called by the test driver to initialize the global script that is common to all later scripts.
This will be shared by all instances of cScript.
The oStorage parameter is a storage containing an attribute set and a stream containing the global script.
UTDCore.cSection
Represents one exam sections or sub-section. May contain items or sections. Derived from cEvent
Public Attributes: oNavigation : UTDCore-iNavigate
The navigation plugin for this section. colCustomAttributes : UTDCore.cAttributes
Collection of custom attributes. Read-only. sName : String
Read-only section name. sTitle : String
Section title. Read-only. oScoring : UTDCoreJScore
The scoring plugin for the section. oScriptStart : UTDCore.cScript
The script to execute before this object executes. oScriptFinish : UTDCore.cScript
The script to execute after this object executes. oScriptConditional : UTDCorccScript
This expression script is evaluated. If TRUE this object is deliveried. oReview : UTDCore-cPresentation
The review presentation for the section. It can be nothing if the section does not have a review defined for it. oCurChild : Object
The current child being deliveried. This can be a cSection or a cPresentation. datStart : Date
The section start date and time. datFinish : Date
The section finish date and time. colChildren : UTDCore Events
The collection of presentations or sub-sections for the section. oSelection : UTDCore.iSelection
The selection plugin for this section, o Template : UTDCore.cTemplate
The tmeplate for the section. oTimer : UTDCore.iTimer
The timer plugin for the section. diTimeRemaining : DATE
Returns the time remaining in the section. It get the seconds remaining from the iTimer plugin. It then converts it to a variant DATE type (aka VT_DATE). colDelivered : UTDCore.cEvents
The collection of presentations or sub-sections for the section that have been presented.
They are ordered by the order they appeared in. bComment : Boolean = FALSE
The commenting mode of the section. Normally set by the Navigation. bltemsReadOnly : Boolean = FALSE
The read-only mode of the items in the section. Normally set by the Navigation. Read only items cannot have their responses changed.
Page 20
WASH INGTON I 49764V 1 nCurCh..d : .ong = 0
Index of last delivered event in colDelivered. Read-only.
Public Operations:
ChildNext 0 :
Requests that the driver proceeds to the next presentation or child section. ChildPrevious 0 :
Requests that the driver proceeds to the previous presentation or child section. ChildGoto (vPresentation : Variant) :
Requests that the driver proceeds to a named or numbered presentation or child section..
UTDCore.cSections
Collection of UTDCore.cSection Derived from UTDCore.Collection
UTDCore.eDirection
Enumeration of the possible directions to proceed from a presentation.
Values: eDirection_Next eDirection_Previous eDirection_TimeoutImmedidate (now do not wait) eDirectionJfimeoutAfter (after current presentation) eDirection_Review (Goto the review presentation) eDirection_Jump (Goto a specific presentation) eDirection_Flag eDirection_Help eDirection_FirstIncomplete eDirection_FirstSkipped eDirection FirstPresentation
UTDCorcePluginModes
Enumeration of the possible modes that an active document can behave.
Values: ePluginMode_Author (Not supportted) ePluginMode_Compile ePluginMode_DeIivery ePluginMode_Rescore (Not suportted) ePluginMode_ProcessResults (Not suportted)
UTDCore.eScoreStatus
The scoring status for a form/section/group/category, determined by the scoring plugin. eScore_Incomplete eScore_Complete eScore_Failure eScore_Pass eScore_Taken
UTDCore.eStates
The states of the test driver. eStateStart ~ Initializing the test driver & reading the resource file. eStateSectionDelivery - Delivering a section. eStateSectionComment - Commenting all sections.
Page 21
WASHINGTON I 49764V 1 eStateSectionFeedBack ~ Feedback on all sections. eStateScoreReporting — Rendering and printing a score report. eStateResuIts - Writing results. eStateFinish ~ Closing down the test driver. UTDCore.iContainerNotify
All plugins receive a reference to this interface. It allow them to report events to the container.
Public Operations:
Activated 0 :
Call to indicate the plugin has received the presentation change and it fully ready for operations.
This includes: processing its data and if visible displaying.
UTDCore.iContainerNotifyHelm
This interface consumed by plugins to inform the container to navigate.
Public Operations:
RequestMove (eDirect : UTDCore-eDirection, sPresentation : String) :
Requests that the driver proceed in the direction specified. The driver next requests this movement from the navigation plugin.
The second parameter optional specifies the presentation. This is only used for the JUMP.
UTDCore.iDispIay
Interface supportted by plugins that a handle title bars, displays, non-answered items and summaries.
Public Attributes: oSection : UTDCore.cSection
The section we are in.
Public Operations:
PresentationStarting (oPresentation : UTDCore.cPresentalion, oContainerNotify :
UTDCore.iContainerNotify) :
Called by the test driver to inform the plugin that a new presentation is starting. Also who to notify when active. PresentationEnding 0 :
Called by the test driver to inform the plugin thae presentation is ending.
UTDCore.iHelm
Interface supportted by plugins that a handle navigation control or review. Like Next,
Previous or Incomplete. Only User interface. Talks to the iNavigate plug in to perform the actual navigation.
Public Attributes: oSection : UTDCore.cSection
The section we are in.
Public Operations:
PresentationStarting (oPresentation : UTDCore.cPresentalion, oContaiπerNotifyHelm :
UTDCore.iContainerNotifyHelm, oContainerNotify : UTDCore.iContainerNotify, bComment : Boolean)
Called by the test driver to inform the plugin that a new presentation is starting. It also provides a way to notify the test driver of movement. PresentationEnding 0 :
Called by the test driver to inform the plugin thae presentation is ending.
Page 22
WASHINGTON 1 49764V 1 UTDCore-iltem
Interface supportted by plugins that a handle items and simulators.
Public Attributes: oSection : UTDCore.cSection
The section we are in. fScoreMinimum : Double = 0
The lowest possible score. Must be less than or equal to the nominal score. It can be negative. fScoreNomiπal : Double = 0
The score if the candidate takes no actions. Must be less than or equal to the maximum score. Must be greater or equal to the minimum score. Normally zero. CScoreMaximum : Double = 1
The highest possible score. Must be greater than or equal to the nominal score. (ScoreCandidate : Double = 0
The score received by the candidate. Requests that the plugin judges the item. bComplete : Boolean = False
True if Item is complete. bSkipped : Boolean = False
True if item was skipped by the candidate.
Public Operations: bProceed (eDirect : UTDCor -eDirection) : Boolean
Test Driver will call this method to see if it is OK to move off the current presentation.
The parameter indicates the direction. GetCorrectAnswerDisplay 0 : String
Returns the correct answer in a form to be displayed on the screen. GetCorrectAnswerResuIts 0 ! ByteO
Returns the correct answer in a form to be written to the results file. The correct answer is in the form of a safe array of bytes. GetResponseDisplay 0 : String
Returns the candidate response in a form to be displayed on the screen. GetResponseResults 0 : ByteO
Returns the candidate response in a form to be written to the results file. The response is in the form of a safe array of bytes. PresentationStarting (oPresentation : UTDCore.cPresentation, oContainerNotify :
UTDCore.iContainerNotify, bReadOnly : Boolean) :
Called by the test driver to inform the plugin that a new presentation is starting. PresentationEnding 0 :
Called by the test driver to inform the plugin thae presentation is ending.
UTDCore-iNavigate
Interface supportted by plugins that a handle section navigation.
Implementer of iNavigate must also implement a iContainerNotifyHelm Derived from UTDCore.iContainerNotifyHelm
Public Attributes: nPresentations : Integer
The number of counted presented to be presented within the scope of the navigation.
Each item in a compound items count as a one. bltemsDetermined : Boolean
TRUE if the number of items to present within the scope of the navigation can be determined. Adaptive items or sections can prevent this.
Page 23
WASHINGTON 149764V I Public Operations: bNavigateAllowed (eDirect : UTDCore.eDirection) : Boolean
Returns TRUE if navigation in the direction supplied sis allowed. It is the helm's job to query this before every presentation.
This is a combination of the resource file plugin data and the current state of the section.
Example implementation:
The test designer is allowed to configure if the previsous button appears. But if we are on the first presentation of a section it will not appear even if configured. Starting (oSection : UTDCore.cSection) :
The section is starting. Ending 0 :
The section is ending.
UTDCore-iPersistlnstanceSet
Interface for a plugin that wants to be persistant in the instance file as a property set.
Public Attributes:
IsDirty : Boolean
Returns TRUE if the object needs to save state to the instance file.
Public Operations:
Save (oPropSet : POLESS.IPropertyStorageVB) :
Called by the test driver to save the plugin data to the instance file. Reload (oPropSet : POLESS JPropertyStorageVB) :
Called by the test driver to reload the plugin data from the instance file. The
IPersistResource*::Load will be called next.
UTDCore.iPersistlnstanceStore
Interface for a plugin that wants to be persistant in the instance file as a storage.
Public Attributes:
IsDirty : Boolean
Returns TRUE if the object needs to save state to the instance file.
Public Operations:
Save (oStorage : POLESSJStorageVB) :
Called by the test driver to save the plugin data to the instance file. Reload (oStorage : POLESS.iStorageVB) :
Called by the test driver to reload the plugin data from the instance file. The
IPersistResource*::Load will be called next.
UTDCore PersistlnstanceStream
Interface for a plugin that wants to be persistant in the instance file as a property stream. Public Attributes: IsDirty : Boolean
Returns TRUE if the object needs to save state to the instance file.
Public Operations:
Save (oStream : POLESS.IStreamVB) : ~~
Called by the test driver to save the plugin data to the instance file. Reload (oStream : POLESS.IStreamVB) :
Called by the test driver to reload the plugin data from the instance file. The
IPersistResource*::Load will be called next.
Page 24
WAS HINGTON I 49764V I UTDCore PersistResourceSet
Interface for a plugin that wants to be persistant in the resource file as a property set.
Public Operations:
Save (oPropSet : POLESS.IPropertyStorageVB) :
Called by the compiler to save the plugin data to the resource file. Load (oPropSe : POLESS.IPropertyStorageVB) :
Called by the test driver to load the plugin data from the resource file. ValidateResource (oPropSet : POLESS.IPropertyStorageVB, oCompilerServices :
UTDCoreJCompilerServices) :
Validates the compiled resource for the plugin. The plugin must raise an automation error that describes any problems with the source.
The combination of the amalogmated storage for the XXL and the referenced files should resolve all references. For example all HREFs to HTML and JPEGs should resolve.
UTDCore-iPersistResourceStore
Interface for a plugin that wants to be persistant in the resource file as a storage.
Public Operations:
Save (oStorage : POLESS.iStorageVB) :
Called by the compiler to save the plugin data to the resource file. Load (oStorage : POLESSJStorageVB) :
Called by the test driver to load the plugin data from the resource file. ValidateResource (oStorage : POLESS.iStorageVB, oCompilerServices : UTDCoreJCompilerServices) :
Validates the compiled resource for the plugin. The plugin must raise an automation error that describes any problems with the source.
The combination of the amalogmated storage for the XXL and the referenced files should resolve all references. For example all HREFs to HTML and JPEGs should resolve.
UTDCore-iPersistResourceStream
Interface for a plugin that wants to be persistant in the resource file as a stream.
Public Operations:
Save (oStrea : POLESS.IStreamVB) :
Called by the compiler to save the plugin data to the resource file. Load (oStream : POLESS.IStreamVB) :
Called by the test driver to load the plugin data from the resource file.
UTDCorciPlugin
This Interface will be supportted by all UTD plugins.
All plugins must implement the following interfaces: iPlugin
All plugin must implement one of the following to persist into the resource file at compile time:
IPersistResourceSet, IPersistResourceStore or IPersistResourceStream
All plugin may implement one of the following to persist into the instance file during delivery time:
IPersistlnstanceSet, IPersistlnstanceStore or IPersistlnstanceStream if you are visible one of these: iHelm, iltem or iDisplay. It can be an iHelm and an iltem. If you wish to be contained as an active document you must support the following: iDataObject, IolelnPlaceObject, lOlelnPlaceActiveObject , IoleDocument, IOIeDocumentView and IoleCommandTarget.
Page 25
W AS HINGTON I 49764V I or one of these if you are invisible: iScore, iReport, iResults, iNavigate, iTimer.
.Navigate must also implement a iContainerNotifyHelm
Public Attributes: eMode : ePluginModes
The mode the plugin should operate in.
Set by the consumer of the plugin (the driver or compiler).
Public Operations:
VaiidateSource (oSource : POLESS.IStreamVB, oCompilerServices : UTDCore-ICompilerServices) :
Validates the source XXL for the plugin. The plugin must raise an automation error that describes any problems with the source. The source is not required to be complete only the portions provided should be verified.
If the contents of the stream is Unicode it will be marked with the BOM (byte order mark) as defined be Unicode standard (www.unicode.org). The BOM is normally FFFE.
If the stream contains ASCII or UTF-8, no BOM will be included.
The oCompilerServices interface is provided to offer additional features and information to the plugins. Unload 0 :
Unload data and references to UTD objects. Load (oExam : UTDCore.cExam, oCandidate : -Attendance) :
Load with references to UTD objects. Only called during exam delivery.
UTDCore-iReport
Interface supported by plugins that handle printing of the score reports and other other material like stylus ("hand-outs"). Public Operations: PrintReport 0 :
Prints the score report
UTDCore.iResults
Interface supported by plugins that handles writing of the candidate results. Public Operations: WriteResults 0 :
Write the exam results.
UTDCore.iScore
Interface supportted by plugins that a handle scoring of sections or exams.
Public Attributes: eStatus : UTDCore.eScøreStatus
The eScoreStatus as follows. fReportedScoreCut : Double = -1
The cut score to report to the candidate and results. Usually scaled. If this does not apply to the plugin return -1. fReportedScoreMa : Double = -1
The maximum score to report to the candidate and results. Usually scaled. If this does not apply to the plugin return -1. fReportedScoreCaπdidate : Double = -1
The achivied score to report to the candidate. Usually scaled. If this does not apply to the plugin return -1.
Page 26
W ASHIN GTON I 49764V 1 oE vents : IEvents *
Tthe collection of children to score. If the plugin is on a section, then this collection can include ISection and IPresentation objects. If the plugin is on a category, then this collection can include ISection, and Iltem objects.
Public Operations: bStop 0 : Boolean
Test Driver will call this method to see if delivery should be stopped from current delivery unit (section or form).
For example an adaptive exam may return false when it is clear if the candidate has master or not mastered the material. GetScoreDisplay 0 : String
Returns the score in a form to be displayed on the screen. GetScoreResults 0 : ByteO
Returns the score in a form to be written to the results file. The score is in the form of a safe array of bytes.
UTDCore-iSelection
Selects items, sections or forms from a category.
Public Attributes: bMore : Boolean
Returns TRUE if there are more objects to get. Read-only. nCountTotal : Integer
Returns the total count of objects (items, sections or forms) that it has in its pool to select from. It will return -1 if this is not quantifiable. nCountSelected : Integer
Returns the selected count of objects (items, sections or forms) that it will return from the pool. It will return -1 if this is not quantifiable. Like items in an adaptive section. oGroup : UTDCore.cSection
Read- write.
The group that contains this selection plugin. This is not set for a form selection plugin. oEvents : UTDCore.IEvents
Read- write.
The events collection to select from.
Public Operations:
Reset 0 :
This gives the selection plug a chance to prepare the data for the first selection. Some selection plugins like "random/exshastive" make pre-determine all the items for selection a store them in its internal memory and the instance file. GetNext () : Object
Gets the next item, section or form. If there are no more, them NULL AKA Nothing is returned.
UTDCore UnitTimer
A plugin to perform timing across testing units.
Public Attributes: lSecondsElasped : Long
The seconds elasped. lSecondsRemaining : Long
The seconds remaining. If the unit is not timed, this will return -1. lSecondsTota) : Long
The total seconds available for the unit. datElasped : DATE
The time elasped. Same as lSecondsElasped.
Page 27
W ASHINGTON 1 4976 V 1 datRemaining : DATE
The time remaining. If the unit is not timed, this will return -1. Same as lSecondsRemaining datTotal : DATE
The total time available for the unit. Same as lSecondsTotal.
Public Operations: ^__^___
PresentatioπActive (oPresentation : UTDCore.cPresentation) :
This is called by the test driver to inform the timer that the presentation is completely active. All plugins have loaded their data, displayed and are operational.
Presentationlnactive 0 :
This is called by the test driver to inform the timer that the presentation is about to become inactive. The user has selected a direction to go from the item. The Item and the navigation has cleared it for movement.
After this call the plugins will be hidden and unloaded. AddTime (nMinutes : Integer) :
Adds time to the current form or section. Time is specified in minutes. Starting (oChildren : UTDCore.cEvents, oContainerNotifyHelm : UTDCore.iContainerNotifyHelm) :
Informs the plugin that the section or form is starting. If the plugin is defined at the form level then this is called when the form starts. If the plugin is defined at the section level then this is called when the section starts.
It gives it the collection of children to time across. This collection can include cSection objects and cPresentation objects.
Ending 0 :
Informs the plugin that the section or form is ending. If the plugin is defined at the form level then this is called when the form ends. If the plugin is defined at the section level then this is called when the section ends.
Pause 0 :
Pauses the timer Resume 0 :
Resumes the timer. (After a Pause() )
UTDP.StandardTimer
Standard timing pluggin works in three basic modes:
Non-timing, the unit's time does not contribute
Timed, the unit is timed and will timeout
Contributing, the unit time is contributed to the parent unit.
In the case of timeout it will be handled mroe than one ways: Immediate end of unit, End after current presentation, Warnings. UTDP.cBrowserDisplay
This display plugin renders HTML to the full area of the active document. It also interpets scripts in the HTML. The active object model of UTD is exposed to the script.
UTDP.cLinearNavigate
Performs linear navigation through a sections until the items or sections are exshasted.
UTDP.cMutliChoiceltem
The standard multichoice item. Uses HTML presentation layout.
Page 28
W ASH INGTON U9764V 1 UTDP.cNextPrevious
Standard Next/Previous type "toolbar" helm.
UTDP.cSequentialExhastive
This plugin performs sequenial exahstive selections. All objects (sections or items) in the order from the resource file.
UTDP.cSummaryScore
This plug performs scoring by summarizing items or section scores. The scores it summarizes can be weighted. The overall score can be scaled.
UTDP.cUserSelection
Allows a user to select a section from a list. The list is the sub-sections in the section. UTDPE.cChallenge
This display is used to challenge the user to identify themselves. It is used to introduce the exam and during a break. UTDPE.cSLSDeliResult
This result processor creates results for the SLSDC result process that looks like an SLSDeli result. UTDPE.cStandardReview
This is the standard review screen. It displays all the items and their titles. It allows the candidate to:
Jump to any item
Goto the first incomplete
Goto the first skipped
Goto the first flags
End the section.
UTDPE.cStandardScoreReport
The standard score report. Will print an HTML page. Public Operations: get_AllowPOLESS 0 : returns AllowPOLESS flag putAllowPOLESS 0 :
UTDPE.cWTDResults
Writes a standard WTD/TASC results file. cActivePlugin
Helper class for plugins. cContainerFrameT
Template class for a container frame. (The main program window.) Derived from IOlelnPlaceFrame. CWindowImpI
Public Operations:
Create 0 : Destroy 0 :
cDocumentSiteT
Template class for a document site. (Child window of the main program window.)
Page 29
W AS H INGTON I 49764 V 1 Derived from lOlelnPlaceSite, IOleDocumentSite, IAdviseSink, IOleClientSite. CWindowImpI, IOlelnPlaceUIWindow
Public Operations:
Create 0 : Destroy 0 :
cEvent
The base class is used for all events deliveried in the exam. Like: sections, reports, results & presentations. cFormGroup
The contain a from group. Used to select the form.
Public Attributes: colForm : UTDCore-cEvents
Collection of UTDCore.cForm objects. oForm : UTDCore.cForπ_
The selected form.
Private Attributes: oSelection : UTDCore-iSelection
The selection plugin to use to select the form. cPluglnT
Template class for a plug in. Gives active document functionality Derived from lOleObjectlmpl, I01eInPlaceObjectWindowlessIn.pl, IPersistStoragelmpl, CComControl. lOleDocumentlmpl, IOleDocumentViewImpl
cPlugin
This class holds the plugin information.
Public Attributes: sName : String
Read-only name of the plugin XXL name, (read-only) sProgID : String
The ProgID of the plugin. (read-only) guidCLSID : GUID
The class ID of the plugin (read-only).
Public Operations: Create () : UTDCoreJPlugin
CoCreates an instance of the plugin and returns it to the caller. This class does not retain any reference to this instance. The caller must release it. cScreenMinimum
The minimum screen resolution
Public Attributes: nWidth : Long = 800
The minimum screen width. nHeigth : Long = 600
The minimum screen heigth. nBitDepth : Integer - 8
The minimum screen color bit depth. 4 = 16 colors, 8 = 256 colors, 16 = 65768 colors, etc..
Page 30
WASHINGTON 1 49764V 1 bValid : Boolean
If TRUE then current video configuration meets the mimimum requirements. cTemplate
A template that defines the presentation areas on the screen. Referenced by a presentation.
Public Attributes: sName : String
Read-only presentation template name. colAreas : UTDCore-cAreas
Collection of areas in the template. colTempIate : UTDCore.cTemplates
Nested templates for THIS template.
Public Operations: Activate 0 •
Active this template. All areas instanciate their plugins and contain them. Not available outside the driver. Deativate 0 :
Deactive this template. All areas destroy their plugins. Not available outside the driver. iAppointment
This interface is part of the Unified Administration System. It allows access to the candidate information for the candidate taking this exam.
It is also emulated by the UTDTOETS component that emulates the UAS. iLaunch
The interface to the DSA Administration system. It is how the admiinistration system controls a component in the center. iLaunch2
The interface to the Unified Administration system. It is how the admiinistration system controls a component in the center. iPrint
The administration system interface for save score reports for reprint. Also handles initial printing.
.Transfer
Administration interface to transfer results and other files back to the data center. It includes routing.
Page 3 1
W AS HINGTON 1 4976 V 1 TOTALS:
2 Logical Packages 95 Classes
LOGICALPACKAGESTRUCTURE
Logical View
Page 32
WASHINGTON 149764VI Table of Contents
SELECTED COMPONENT VIEW REPORT 4
COMPONENT VIEW 4
UTDCore 4
UTDCore.e sgBoxResults (Interfaces) 5
UTDCore.IPresentation (Interfaces) 6
UTDCore.iContainerNotify (Interfaces) 7
UTDCore.lCategories (Interfaces) 8
UTDCore.lMsgBox (Interfaces) 8
UTDCore.IPersistResourceSet (Interfaces) 9
UTDCore.IGroup (Interfaces; 9
UTDCore.lScr.pt (Interfaces) 10
UTDCore.IUnitTimer (Interfaces) 12
UTDCore.INavigate (Interfaces) 13
UTDCore.Iltems (Interfaces) 15
UTDCore.ICategory (Interfaces) 15
UTDCore.lPersistlnstanceStore (Interfaces) 16
UTDCore.IcResults (Interfaces) 16
UTDCore-Iclte (Interfaces) 17
UTDCore.lScore (Interfaces) 19
UTDCore.lExam (Interfaces) 20
UTDCore.IRegistry (Interfaces) 23
UTDCore.IAttribute (Interfaces) 24
UTDCore.IHelm (Interfaces) 24
UTDCore.IcReport (Interfaces) 25
UTDCore.eDirec.ion (Interfaces) 25
UTDCore.iPersistlnstanceStream (Interfaces) 26
UTDCore.ePluginModes (Interfaces) 26
UTDCore.iPersistResourceStream (Interfaces) 27
UTDCore.ISelection (Interfaces) 27
UTDCore-IGroups (Interfaces) 28
UTDCore.Collection (Interfaces) 28
UTDCore.eScoreStatus (Interfaces) 29
UTDCore.eStates (Interfaces) 29
UTDCore.IPersistResourceStore (Interfaces) 29
UTDCore.eExamStatus (Interfaces) 30
UTDCore.lI.em (Interfaces) 30
UTDCore.IPlugin (Interfaces) 32
UTDCore.iContainerNotifyHelm (Interfaces) 33
UTDCore.lReport (Interfaces) 34
UTDCore.IForm (interfaces) 34
UTDCore.IAttributes (Interfaces) 37
UTDCore.IPresentations (Interfaces) 37
UTDCore.lSections (interfaces) 37
UTDCore.IDisplay (Interfaces) 37
UTDCore.ISection (Interfaces) 38
UTDCore.IResults (Interfaces) 41
UTDCore.eCategoryContent (Interfaces) 4]
UTDCore.lEvents (Interfaces) 41
UTDCore.IPersistlnstanceSet (Interfaces) 42
Administration System 42
UAS.ITransfer (Interfaces) 42
UAS.IPrint (Interfaces) 42
UAS. I Appointment (Interfaces) 43
TOTALS: 44
COMPONENT PACKAGE STRUCTURE 44
Page 2
WASHINGTON I 53548V 1
Page 3
WASHINGTON 153548VI PHYSICAL VIEW REPORT
SELECTED COMPONENT VIEW REPORT
Figure imgf000099_0001
UTDCore.lPersist ResourceStream
UTDCore
Page 4 PHYSICAL VIEW REPORT
Figure imgf000100_0001
UTD version 2
UTDCore.eMsgBoxResuIts (Interfaces)
Enumeration of possible results from IMsgBox::Display
Page 5 eMsgBoxResult_Yes eMsgBoxResult_No eMsgBoxResult_Timeout (Candidate clicked nothing, timeout occurred)
UTDCore.IPresentation (Interfaces)
The data and current state of the presentation. Could be a item or a display.
Derived from cEvent, UTDCore.iContainerNotifyHelm
Public Attributes:
bCountPresentation : Boolean
If TRUE the presentation should count for the total X or Y for the section, "the item count"
colCustomAttributes : UTDCore.IAUributes
Collection of custom attributes. Read-only.
sName : String
Read-only presentation name.
sTitle : String
Presentation title. Read-only
oScriptStart : UTDCoreJScript
The script to execute before this object executes.
oScriptFinish : UTDCore.IScript
The script to execute after this object executes.
oScriptConditional : UTDCore.IScript
This expression script is evaluated. If TRUE this object is deliveried.
Page 6
WASHINGTON 153548V I sCo ment : String
The user entered comment for this presentation.
bMarked : Boolean
TRUE if the item has been mark for later review. Set by the helm.
coIChildren : UTDCoreJEvents
The items in this presentation. The collection will be empty for a non-item presentation.
bSelected : Boolean
True if the presentation or any of its items have been selected for delivery by a selection plugin. Read-only.
Public Operations:
Findlcltem (plltem : Iltem *) : Icltem *
Find the Icltem interface for the given Iltem. Returns failure if the Iltem is not in this presentation.
UTDCore.iContainerNotify (Interfaces)
All plugins receive a reference to this interface. It allow them to report events to the container.
Public Operations:
Activated 0 ■
Call to indicate the plugin has received the presentation change and it fully ready for operations. This includes processing its data and painting on the screen (if visible). This object is passed as a parameter on the PresentationStarting method of visible plug-in types.
Page 7
W AS H I NGTON I 535 8 V 1 Failure to call this method from a visible plug-in will cause the driver to wait infinitely. The candidate will not be able to navigate or respond. Do not call this method more than once for each time PresentationStarting is called.
UTDCore.ICategories (Interfaces)
Collection of UTDCore.ICategory Derived from UTDCore.Collection
UTDCore MsgBox (Interfaces)
This class displays a message box with a candidate prompt. This class is returned from IExam::GetMsgBox(). The message in the box is supplied by the called. The title can optionally be supplied. The buttons are supplied by the caller, but the button labels are defined at the exam level in the resource file.
The parent of the message box is the main exam window. The message box can be set to time out after a duration.
Public Attributes'
sTitle : String = UTDCore
The title the message box will use. Read/Write.
ITimeout : Long = 120
The timeout value on the message box in seconds. Read/Write. The Display will return eMsgBoxResult_Timeout.
nParentHWnd : Integer
The parent window handle Read/Write Default is the main UTDCore window.
Page 8
WASHINGTON 15_ 548V 1 Public Operations:
Display (sPrompt : String, eStyle : UTDCore.eMsgBoxStyle) : UTDCore.eMsgBoxResulU
Display the message box. Displays the buttons requested and the message supplied.
UTDCore PersistResourceSet (Interfaces)
Interface for a plugin that wants to be persistant in the resource file as a property set.
Public Operations:
Save (oPropSet : POLESS.IPropertyStorageVB) :
Called by the compiler to save the plugin data to the resource file. Load (oPropSet : POLESS.IPropertyStorageVB) :
Called by the test driver to load the plugin data from the resource file.
ValidateResource (oPropSet : POLESS.IPropertyStorageVB, oCompilerServices : UTDCore CompilerServices) :
Validates the compiled resource for the plugin. The plugin must raise an automation error that describes any problems with the source.
The combination of the amalogmated storage for the XXL and the referenced files should resolve all references. For example all HREFs to HTML and JPEGs should resolve.
UTDCore.IGroup (Interfaces)
Groups are the holders for events in the section. A group may hold presentations or sections.
Public Attributes:
sName : String
Page 9 WASHINGTON 1535 8V 1 PHYSICAL VIEW REPORT
Read-only group name.
sTitle : String
Read-only group title.
colCustomAttributes : UTDCore.IAttribu.es
Collection of custom attributes. Read-only.
colChildren : UTDCore.IEvents The collection of presentations, sub-sections, or sub-groups for this group.
oScoring : UTDCorelScore
The scoring plugin for the section.
colDelivere :
Collection of presentations or sub-sections containing all delivered events within this group. This will not contain sub-groups.
oSelection : UTDCor e.lSelection Pointer to the current selection plugin of this group.
oSection : UTDCore-ISection
The section housing this group.
UTDCore.IScript (Interfaces)
A script written in VBscript or JScript. To be executed or evaluated by the ActiveX Scripting Engine. This object will expose UTD objects to the script code. These objects will be based on the scope of the "Parent" object provided to this object.
All scripts can see oExam, oForm, and oCandidate.
Page 10
WASHINGTON 1535 8V 1 If the parent object is a section object, then the script will also see oCurSection.
If the parent object is a presentation object, then the script will also see oCurSection and oCurPresentation.
If the parent object is an item then the script will also see oCurSection, oCurPresentation, and oCurltem.
Public Attributes:
oParent : Object
The parent UTD object. It could be a form, a section or a presentation.
Public Operations'
Execute (sScript : BSTR) :
Executes the script code passed in the sScript parameter. EvaluateBool (sScript : BSTR) : Boolean
Evaluates the expression parsed as the sScript parameter. Returns TRUE or FALSE. Raises an error if the code is not an expression returning a boolean. EvaluateVar (sScript : BSTR) : Variant
Evaluates the expression passed in the sScript parametert. Returns a variant depending on the return type of the script. Raises an error if the code is not an expression.
SetScope (oSection : ISection *, oPresentation : IPresentation *, oltem : Iltem *) :
Set the parent section, presentation, and item. The script will have access to these as the oCurSection, oCurPresentation, and oCurltem objects. Any/All may be passed as NULL parameters indicating that the script does not have access to that particular object.
AddObject (sName : BSTR, pObject : IDispatch *) : HRESULT
Add an object into the global script scope.
Protected Operations.
Page 1 1
WASHINGTON 1 53548V 1 LoadGlobalScript (oStream : POLESSJcStorage) :
Called by the test driver to initialize the global script that is common to all later scripts. This will be shared by all instances of cScript. The oStorage parameter is a storage containing an attribute set and a stream containing the global script.
UTDCore.IUnitTimer (Interfaces)
A plugin to perform timing across testing units. Public Attributes.
ISecondsElapsed : Long
The seconds elapsed.
ISecondsRemainin : Long
The seconds remaining. If the unit is not timed, this will return -1.
ISecondsTotal : Long
The total seconds available for the unit.
datElapsed : DATE
The time elapsed. Same as ISecondsElapsed.
datRemaining : DATE
The time remaining. If the unit is not timed, this will return -1. Same as ISecondsRemaining
datTotal : DATE
The total time available for the unit. Same as ISecondsTotal.
Public Operations
Page 12
W AS H I NGTON I 53548 V I PHYSICAL VIEW REPORT
PresentationActive (oPresentation : UTDCore.IPresentation) :
This is called by the test driver to inform the timer that the presentation is completely active. All plugins have loaded their data, displayed and are operational.
Prestntationlnactive 0 :
This is called by the test driver to inform the timer that the presentation is about to become inactive. The user has selected a direction to go from the item. The Item and the navigation has cleared it for movement.
After this call the plugins will be hidden and unloaded.
AddTime (nMinutes : Integer) :
Adds time to the current form or section. Time is specified in minutes. Surting (oChildren : UTDCore.IE vents, oContainerNotifyHelm : UTDCore.iContainerNotifyHelm) :
Informs the plugin that the section or form is starting. If the plugin is defined at the form level then this is called when the form starts. If the plugin is defined at the section level then this is called when the section starts.
It gives it the collection of children to time across. This collection can include cSection objects and cPresentation objects.
Ending 0 :
Informs the plugin that the section or form is ending. If the plugin is defined at the form level then this is called when the form ends. If the plugin is defined at the section level then this is called when the section ends.
Pause 0 :
Pauses the timer
Resume 0 :
Resumes the timer. (After a PauseQ )
UTDCore Navigate (Interfaces)
Page 13
W ASHINGTON I 53548V 1 Interface supportted by plugins that a handle section navigation.
Implementer of .Navigate must also implement a iContainerNotifyHelm
Derived from UTDCore.iContainerNotifyHelm
Public Attributes:
.-Presentations : Integer
The number of counted presented to be presented within the scope of the navigation. Each item in a compound items count as a one.
bltemsDetermined : Boolean
TRUE if the number of items to present within the scope of the navigation can be determined. Adaptive items or sections can prevent this.
Public Operations:
bNavigateAllowed (eDirect : UTDCore.eDirection) : Boolean
Returns TRUE if navigation in the direction supplied sis allowed. It is the helm's job to query this before every presentation.
This is a combination of the resource file plugin data and the current state of the section.
Example implementation:
The test designer is allowed to configure if the previsous button appears. But if we are on the first presentation of a section it will not appear even if configured.
Starting (oSection : UTDCore.ISection) :
The section is starting.
Ending () :
The section is ending.
Page 14
WASHINGTON I 53548V I UTDCore.IItems (Interfaces)
Collection of UTDCore.IcItem. Note this is not the UTDCore.IItem interface implemented by an item plug-in.
Derived from UTDCore.Collection
UTDCore.ICategory (Interfaces)
A category of sections or items. For example: "Reporting", "Objectivies", "Scoring"
Public Attributes:
sName : String
The name of the category.
bCompIete : Boolean
If TRUE the category must contain all sections or items defined at a level.
bDuplicate : Boolean
If TRUE the category allows sections or items defined more than once at a level.
sDescription : String
The description of the category.
colMembers : UTDCoreJEvents
The collection of sections or items that apply to this category .
colSubCategories : UTDCore.ICategories
Collection of UTDCore.cCategory. (AKA sub-categories).
eContents : eCategoryContents
Read-only. The allowable types of members in the colMembers.
Page 15
W A S H I N GTON ! 5 _ 548V ] enum { eCategoryContent_Anything = 0, eCategoryContent_Sections = 1 , eCategoryContent_Items = 2, eCategoryContent_Categories = 3,
} eCategoryContents;
oScoring : UTDCoreJScore
The scoring plugin for the form.
UTDCore.IPersistlnstanceStore (Interfaces)
Interface for a plugin that wants to be persistant in the instance file as a storage.
Public Attributes
IsDirty : Boolean
Returns TRUE if the object needs to save state to the instance file.
Public Operations:
Save (oStorage : POLESS.iStorageVB) :
Called by the test driver to save the plugin data to the instance file. Reload (oStorage : POLESS.iStorageVB) :
Called by the test driver to reload the plugin data from the instance file. The IPersistResource*::Load will be called next.
UTDCore.IcResults (Interfaces)
Page 16
WASHINGTON 153548V I A collection of Events will contain an IcResults interface for every result. It is a container for the IResults plug-in. This is not the IResults interface that the plug-in implements.
Derived from cE\ent
Public Attributes:
oResults : UTDCore ! Results
The plugin that writes the results.
UTDCore.IcItem (Interfaces)
A collection of Items or Events will contain an Icltem interface for every item. This interface provides general item information including attributes specified in the <item> tag of the XXL. This is not the UTDCore.IItem interface that the plug-in implements.
It is possible to Querylnterface this Icltem for the Iltem interface. This allows a report, results, or scoring plug-in to call methods on the Iltem interface.
Derived from UTDCore Iltem
Public Attributes'
bPresented : Boolean = False
True if item has been presented to the candidate. Set by item component.
sName : String
Read-only item name.
sTitle : String
Presentation title. Read-only
fWeight : Double = 1
The item weigthing relative to other items. Normally 1. If multiple weigths per item are required they can be attached in the custom attributes. The weigth can be negative.
Page 17
WAS HINGTON I 53548V I colCustomAttribu.es : UTDCore.IAttribu.es
Collection of custom attributes. Read-only.
ISecondsElapsed : Long
The seconds spent on the item.
bScored : Boolean = TRUE
Read-only. TRUE if the item is to be scored.
If false fWeight and fScoreCandidate Weighted will all be 0.
bSkipAUowcd : Boolean
Read-only property. If TRUE the item can be skipped. If TRUE the bProceed() method on iltem interface will not be called.
fScoreCandidate Weighted : Double
The weighted value of the candidate for the item. The judged value for the item will by multiplied by the weight for this value.
sPluginName : String
Read-only plugin name. The prog ID of the plugin.
oSection : UTDCore.ISec.ion
Read-only property of the section the item was deliveried in.
oPresentation : UTDCore.IPresentation
Read-only property of the presentation the item was deliveried in.
datElapsed : DATE
The time spent on the item. Same as ISecondsElapsed, but sent as a COM DATE type.
Page 1 8
WASHINGTON 1 53548V I UTDCoreJScore (Interfaces)
Interface supportted by plugins that a handle scoring of sections or exams.
Public Attributes.
eStatus : UTDCore.eScoreStatus
The eScoreStatus as follows.
fReportedScoreCut : Double = -1
The cut score to report to the candidate and results. Usually scaled. If this does not apply to the plugin return -1.
fReportedScoreMax : Double = -1
The maximum score to report to the candidate and results. Usually scaled. If this does not apply to the plugin return -1. fReportedScoreCandidate : Double = -1
The achivied score to report to the candidate. Usually scaled. If this does not apply to the plugin return -1.
oE vents : IEvents *
Tthe collection of children to score. If the plugin is on a section, then this collection can include ISection and IPresentation objects. If the plugin is on a category, then this collection can include ISection, and Iltem objects.
Public Operations:
bStop 0 : Boolean
Test Driver will call this method to see if delivery should be stopped from current delivery unit (section or form).
For example an adaptive exam may return false when it is clear if the candidate has master or not mastered the material.
Page 19
WA S HIN GTO N I 53548V 1 GetScoreDisplay 0 : String
Returns the score in a form to be displayed on the screen. GetScoreResults () : ByteO
Returns the score in a form to be written to the results file. The score is in the form of a safe array of bytes.
UTDCoreJExam (Interfaces)
The exam instance. The root object of the driver exposed classes. Deπved from cContamerFrameT
Public Attributes
sName : String
Read-only exam name. sLanguage : String
The language of the exam content. Read-only. (UTDCore 2.0.0.47 - This currently returns a blank string)
sVersion : String
The version of the exam. Read-only.
oActiveForm : UTDCore.IForm
The current form. Only one form can be active for an exam.
eStatus : UTDCore.eExamStatus
The eExamStatus as follows:
eExam_Initializing eExam_Starting eExam_InProgress
Page 20
WASHINGTON 1 53548V 1 eExam_Ended eExamJTerminating
bShowResponseComment : Boolean
If TRUE the candidate should not see their response during the comment period.
colCustomAttributes : UTDCore.IAttribu.es
Collection of custom attributes. Read-only.
sTitle : String
Exam title. Read-only
colCategories : UTDCore.ICategories
The collection of all categories.
oResourceRoot : Object
The read-only reference to the root storage of the POLESS resource file. Interfaces: POLESS.cStorageRoot, iStorage, iRootStorage, iStorageVB.
olnstanceRoot : Object
The read-only reference to the root storage of the POLESS exam instance file. Interfaces: POLESS.cStorageRoot, IStorage, IRootStorage, IStorageVB.
oRegistry : UTDCore-IRegistry
Returns the single instance of the registry object. Read-only.
Public Operations:
Minimize () :
Minimumizes the main windows of UTD. Can be called be simulations to get UTD out of the way.
Page 21
WASHINGTON I 53548V 1 Restore 0 :
Restore the main windows of UTD. Can be called be simulations to restore UTDto the screen.
GetScript 0 : UTDCore JScript
Returns a script object to be used by a plug-in.
Savelnstance 0 :
Save the state of the test driver to the instance file. All active objects are asked to serialize themselves to the instance file.
Any one with a reference to this object can initiate this process. It could be an item plugin that wants to save user interaction and state information.
The driver will save between each presentation. It may also save at various other events or time intervals.
GetTransfer 0 : UASJTransfer
Returns a UAS Transfer object to transfer results or other files to the data center.
The IExam object does not keep a reference to this object.
GetPrint 0 : IUASPrint
Returns a UAS print object to save for reprint score and other reports. The IExa object does not keep a reference to this object.
FatalError (INumber : Long, sSource : String, sDescription : String) :
Reports an error in a plugin to the test driver.
Normally errors will be reported back to the test driver via an HRESULT and supportting lErrorlnfo. If the error was generated from something other than test driver initiated call, the plugin should report the error via this method.
An example usage could be the user clicks a button on your plugin item. The resulting actions generate an error.
Warn 0 :
Writes a Warning' level trace messate to the instance file.
Info () :
Writes a Information' level trace messate to the instance file.
Page 22
WASHINGTON 1 535 8V 1 Quit 0 :
Tells the exam that the candidate is quitting the exam. Normally called by the Navigation plugin in response to the eDirection_Quit. The navigation plugin should confirm this decision with the candidate.
This causes the test driver to stop delivery and shutdown execution cleanly. The administration system is informed that the candidate quit the exam.
GetMsgBox 0 : UTDCoreJMsgBox
Returns a message box object. Any plug-ins that wish to display a message box should use this object whenever possible. It provides language independence, and a consistent look and feel.
StartExam 0 :
Tells the exam that the candidate has accept the exam and started. Normally called by the Navigation plugin in response to the eDirection_Start.
The administration system is informed that the exam has started and the candidate has accepted it.
UTDCore Registry (Interfaces)
Alows registry look up for translating strings For examples:
<base href="%REG:DSAGRAPGHICS"> into
<base href="V:\DSA\">
Looks at
[HKEY_LOCAL_MACHINE\SOFTWARE\Prometric\UTD\Keywords
]
Public Operations:
Page 23
WASHI NGTON I 53548V 1 Lookup (sKey : String) : String
Retrieves the string by finding the key in the registry and returning the value.
Translate (slnString : String) : String
Translates each occurance of %REG:x% into the equivilent of "x" in the registry.
UTDCore.IAttribute (Interfaces)
Public Attributes-
sName : BSTR
Get the attribute name string.
value : VARIANT
Get the attribute value.
UTDCorcIHelm (Interfaces)
Interface implemented by plug-ins that a handle navigation control or review. Like Next, Previous or Incomplete. Only User interface. Talks to the .Navigate plug in to perform the actual navigation.
Public Attributes:
oSection : UTDCore.ISection
The section this helm is in.
Public Operations'.
Page 24
WASHINGTON 1 535 8V 1 PresentationStarting (oPresentation : UTDCore.IPresentation, oContainerNotifyHelm : UTDCore.iContainerNotifyHelm, oContainerNotify : UTDCore.iContainerNotify, bComment : Boolean)
Called by the test driver to inform the plugin that a new presentation is starting. The plug-in may hold the IPresentation, IContainerNotify, and IContainerNotifyHelm objects until the PresentationEnding. See IContainerNotify::Activate. The IContainerNotifyHelm interface as a way to notify the test driver of movement.
PresentationEnding 0 '■
Called by the test driver to inform the plug-in the presentation is ending. After calling this method, the driver expects the plug-in will release the IPresentation, IContainerNotify, and IContainerNotifyHelm objects passed in PresentationStarting.
UTDCore.IcReport (Interfaces)
A collection of Events will contain an IcReport interface for every report. It is a container for the IReport plug-in. This is not the IReport interface that the plug-in implements.
Derived from cEvent
Public Attributes:
oScoreReport : UTDCore.IReport
The score report plugin.
UTDCore.eDirection (Interfaces)
Enumeration of the possible directions to proceed from a presentation. eDirection_Next eDirection_Previous eDirection_TimeoufImmedidate (now do not wait) eDirection_TimeoutAfter (after current presentation) eDirection_Review (Goto the review presentation) eDirection_Jump (Goto a specific presentation)
Page 25
WASHINGTON 153548V 1 eDirection_Flag eDirection_Help eDirection_FirstIncomρlete eDirection_FirstSkipped eDirection_FirstPresentation eDirection_FirstMarked eDirection_End eDirection_Comment eDirection_Start eDirection_Quit
UTDCore.iPersistlnstanceStream (Interfaces)
Interface for a plugin that wants to be persistant in the instance file as a property stream.
Public Attributes:
IsDirty : Boolean
Returns TRUE if the object needs to save state to the instance file.
Public Operations.
Save (oStream : POLESS -IStreamVB) :
Called by the test driver to save the plugin data to the instance file. Reload (oStream : POLESSJStrea VB) :
Called by the test driver to reload the plugin data from the instance file. The IPersistResource*::Load will be called next.
UTDCore.ePluginModes (Interfaces)
Enumeration of the possible modes that an active document ePluginMode_Author (Not supported)
Page 26
WASH INGTON I 53548V I ePluginMode_CompiIe ePluginMode_Delivery ePluginMode_Rescore (Not suportted) ePluginMode_ProcessResults (Not suportted)
UTDCore PersistResourceStream (Interfaces)
Interface for a plugin that wants to be persistant in the resource file as a stream.
Public Operations:
Save (oStream : POLESS.IStreamVB) :
Called by the compiler to save the plugin data to the resource file. Load (oStream : POLESS.IStreamVB) :
Called by the test driver to load the plugin data from the resource file.
UTDCore Selection (Interfaces)
Selects items, sections or forms from a category. Public Attributes:
bMore : Boolean
Returns TRUE if there are more objects to get. Read-only.
nCountTotal : Integer
Returns the total count of objects (items, sections or forms) that it has in its pool to select from. It will return -1 if this is not quantifiable.
nCountSelected : Integer
Returns the selected count of objects (items, sections or forms) that it will return from the pool. It will return -1 if this is not quantifiable. Like items in an adaptive section.
Page 27
W ASHINGTON 1 53548V 1 oGroup : UTDCore-ISection
Read-write.
The group that contains this selection plugin. This is not set for a form selection plugin.
oEvents : UTDCoreJEvents
Read-write.
The events collection to select from.
Public Operations:
Reset 0 :
This gives the selection plug a chance to prepare the data for the first selection. Some selection plugins like "random/exshastive" make predetermine all the items for selection a store them in its internal memory and the instance file.
GetNext 0 : Object
Gets the next item, section or form. If there are no more, them NULL AKA Nothing is returned.
UTDCore.IGroups (Interfaces)
Collection of UTDCore.IGroup Derived from UTDCore.Collection
UTDCore.CoIlection (Interfaces)
Base UTD collection. All other collection classes follow this design. Public Attributes:
Count : Long
Page 28
WASH INGTON 1 53548V I Returns the number of members in a collection.
NewEπum : IUn known * *
Returns a Visual Basic enumerator object.
Public Operations:
Item (Index : Variant) : Variant
Returns a specific member of a Collection object either by position or key.
UTDCore.eScoreStatus (Interfaces)
The scoring status for a form/section/group/category, determined by the scoring plugin. eScore_NonGraded eScore_Failure eScore Pass
UTDCore.eStates (Interfaces)
The states of the test driver. eStateStart -- Initializing the test driver & reading the resource file. eStateSectionDelivery -- Delivering a section. eStateSectionComment — Commenting all sections. eStateSectionFeedBack — Feedback on all sections. eStateScoreReporting - Rendering and printing a score report. eStateResults - Writing results. eStateFinish — Closing down the test driver.
UTDCore.IPersistResourceStore (Interfaces)
Page 29
W ASH I N GTON 1 53548 V I PHYSICAL VIEW REPORT
Interface for a plugin that wants to be persistant in the resource file as a storage.
Public Operations:
Save (oStorage : POLESS.iStorageVB) :
Called by the compiler to save the plugin data to the resource file. Load (oStorage : POLESS.iStorageVB) :
Called by the test driver to load the plugin data from the resource file. ValidateResource (oStorage : POLESS.iStorageVB, oCompilerServices : UTDCore.ICoπ_pilerServices) :
Validates the compiled resource for the plugin. The plugin must raise an automation error that describes any problems with the source.
The combination of the amalogmated storage for the XXL and the referenced files should resolve all references. For example all HREFs to HTML and JPEGs should resolve.
UTDCore.eExamStatus (Interfaces)
Enumeration of possible exam delivery states. eExam_Initializing (loading resource file) eExam_Starting (delivering, candidate not yet "started") eExam_InProgress (delivering, candidate officially "started") eExam_Ended (delivering, candidate officially "ended") eExam_Terminating (cleaning up from memory)
UTDCore.IItem (Interfaces)
Interface supportted by plugins that a handle items and simulators. Public Attributes.
oSection : UTDCore.ISection
The section we are in.
Page 30
WAS H INGTON I 53548V I fScoreMinimum : Double = 0
The lowest possible score. Must be less than or equal to the nominal score. It can be negative.
fScoreNominal : Double = 0
The score if the candidate takes no actions. Must be less than or equal to the maximum score. Must be greater or equal to the minimum score. Normally zero.
fScoreMaximum : Double = 1
The highest possible score. Must be greater than or equal to the nominal score.
fScoreCandidate : Double = 0
The score received by the candidate. Requests that the plugin judges the item.
bComplcte : Boolean = False
True if Item is complete.
bSkipped : Boolean = False
True if item was skipped by the candidate.
Public Operations:
bProceed (eDirect : UTDCore.eDirection) : Boolean
Test Driver will call this method to see if it is OK to move off the current presentation. The parameter indicates the direction.
GetCorrectAnswerDisplay 0 : String
Returns the correct answer in a form to be displayed on the screen. GetCorrectAnswerResults () : ByteO
Returns the correct answer in a form to be written to the results file. The correct answer is in the form of a safe array of bytes.
GetResponseDisplay 0 : String
Returns the candidate response in a form to be displayed on the screen.
Page 3 1
WAS HI NGTON I 53548V ! GetResponseResults () : ByteO
Returns the candidate response in a form to be written to the results file. The response is in the form of a safe array of bytes.
PresentationStarting (oPresentation : UTDCoreJPresentation, oContainerNotify : UTDCore.IContainerNot.fy, bReadOnly : Boolean) :
Called by the test driver to inform the plugin that a new presentation is starting.
PresentationEnding 0 :
Called by the test driver to inform the plugin thae presentation is ending.
UTDCore.IPlugin (Interfaces)
This Interface will be supportted by all UTD plugins.
All plugins must implement the following interfaces: iPlugin
All plugin must implement one of the following to persist into the resource file at compile time:
IPersistResourceSet, IPersistResourceStore or IPersistResourceStream
All plugin may implement one of the following to persist into the instance file during delivery time:
IPersistlnstanceSet, IPersistlnstanceStore or IPersistlnstanceStream if you are visible one of these: iHelm, iltem or iDisplay. It can be an iHelm and an iltem.
If you wish to be contained as an active document you must support the following: iDataObject, IolelnPlaceObject, lOlelnPlaceActiveObject , IoleDocument, IOIeDocumentView and IoleCommandTarget. or one of these if you are invisible: iScore, iReport, iResults, .Navigate, iTimer. iNavigate must also implement a iContainerNotifyHelm
Public Attributes:
eMode : ePluginModes
The mode the plugin should operate in.
Page 32
W ASHI N GTON 1 53548 V I Set by the consumer of the plugin (the driver or compiler).
Public Operations:
ValidateSource (oSource : POLESS StreamVB, oCompilerServices : UTDCore-ICompilerServices) :
Validates the source XXL for the plugin. The plugin must raise an automation error that describes any problems with the source. The source is not required to be complete only the portions provided should be verified.
If the contents of the stream is Unicode it will be marked with the BOM (byte order mark) as defined be Unicode standard (www.unicode.org). The BOM is normally FFFE.
If the stream contains ASCII or UTF-8, no BOM will be included.
The oCompilerServices interface is provided to offer additional features and information to the plugins.
Unload 0 :
Unload data and references to UTD objects. Load (oExa : UTDCoreJExam, oCandidate : iAttendance) :
Load with references to UTD objects. Only called during exam delivery.
UTDCore ContainerNotifyHelm (Interfaces)
This interface consumed by plugins to inform the container to navigate. Public Operations:
RequestMove (eDirect : UTDCore.eDirection, sPresentation : String) :
Requests that the driver proceed in the direction specified. The driver next requests this movement from the navigation plugin.
The second parameter optional specifies the presentation. This is only used for the JUMP.
Page 33
WASHI NGTON I 5.548V I UTDCore.IReport (Interfaces)
Interface supported by plugins that handle printing of the score reports and other other material like stylus ("hand-outs").
Public Operations:
PrintReport 0 :
Prints the score report
UTDCore For (Interfaces)
A exam form. Derived from UTDCore.iContainerNotifyHelm
Public Attributes.
colChildren : UTDCore.IEvents
The collection of all top level sections of the exam. Read-only.
colCustomAttributes : UTDCore.IAttributes
Collection of custom attributes. Read-only.
sName : String
Read-only form name.
sTitle : String
Form title. Read-only
colItemsChosen : UTDCore Items
Page 34
WASHINGTON 153548V I The collection of all items chosen (that is, returned by a selection plugin) in the exam. This is regardless of their section.
colAIISections : UTDCoreJSections
The collection of all sections of the exam regardless of their level.
datStart : Date
The form start date and time.
datFinish : Date
The form finish date and time.
oTimer : UTDCoreJUnitTimer
The timer for the form.
oScorin : UTDCore.IScore
The scoring plugin for the form.
sVersion : String
The version of the form.
colDelivered : UTDCoreJEvents
The collection of all delivered top level sections of the form. Read-only.
nCurlndex : Long = 0
Index of last delivered event in colDelivered. Read-only.
eStatus : UTDCore.eScorcStatus
Returns the value of oForm.oScore.eStatus.
colAHPresentations : UTDCore.IPresenta.ions
The collection of all presentations of the exam regardless of their level.
Page 35
WASHINGTON I 5_ 5 8V ! colPresentationsChosen : UTDCore Presentations
The collection of all presentations chosen (that is, returned by a selection plugin) in the exam. This is regardless of their section or group.
colSectionsChosen : UTDCore.ISections
The collection of all sections chosen (that is, returned by a selection plugin) in the exam. This is regardless of their level.
colAIIItems : UTDCore.IItems
The collection of all items of the exam regardless of their level.
colAUGroups : UTDCore-IGroups
The collection of all groups of the exam regardless of their level.
colGroupsChosen : UTDCore.IGroups
The collection of all groups chosen (that is, the selection plugin for this group has been ResetO) in the exam. This is regardless of their level.
Public Operations
ChildNext 0 :
Requests that the driver proceeds to the next event on the form. This can be a section, a result, or a score report.
ChildPrevious () :
Requests that the driver proceeds to the previous event on the form. This can be a section, a result, or a score report.
ChildGoto (vSection : Variant) :
Requests that the driver proceeds to a named or numbered event on the form. This can be a section, a result, or a score report. datMinRemaining :
Minimum time remaining for this exam.
Page 36
WASHINGTON I 5.548V I UTDCore Attributes (Interfaces)
Collection of UTDCore.IAttribute Derived from UTDCore.Collection
UTDCore.IPresentations (Interfaces)
Collection of UTDCore.cPresentation Derived from UTDCore.Collection
UTDCore.ISections (Interfaces)
Collection of UTDCore.cSection Derived from UTDCore.Collection
UTDCoreJDisplay (Interfaces)
Interface supported by plugins that a handle title bars, displays, non- answered items and summaries.
Public Attributes:
oSection : UTDCore.ISection
The section this display is in.
Public Operations:
Page 37
WASHINGTON I 535 8V I PHYSICAL VIEW REPORT
PresentationStarting (oPresentation : UTDCore Presentation, oContainerNotify : UTDCore.iContainerNotify) :
Called by the test driver to inform the plugin that a new presentation is starting. The plug-in may hold the IPresentation and IContainerNotify objects until the PresentationEnding. See IContainerNotify :Activate.
PresentationEnding ■
Called by the test driver to inform the plug-in the presentation is ending. After calling this method, the driver expects the plug-in will release the IPresentation and IContainerNotify objects passed in PresentationStarting.
UTDCorcISection (Interfaces)
Represents one exam sections or sub-section. May contain presentations or sections.
Derived from cEvent Public Attributes- oNavigation : UTDCore.INavigate The navigation plugin for this section.
colCustomAttributes : UTDCore.IAttributes Collection of custom attributes. Read-only.
sName : String
Read-only section name.
sTille : String Section title. Read-only.
oScorin : UTDCore.IScore
The scoring plugin for the section.
Page 38
WAS H IN GTON 1 535 8V I oScriptStart : UTDCoreJScript
The script to execute before this object executes.
oScriptFinish : UTDCore.IScript
The script to execute after this object executes.
oScriptConditional : UTDCore.IScript
This expression script is evaluated. If TRUE this object is deliveried.
oReview : UTDCore.IPresentation
The review presentation for the section. It can be nothing if the section does not have a review defined for it.
oCurChild : Object
The current child being deliveried. This can be a cSection or a cPresentation.
datStart : Date
The section start date and time.
datFinish : Date
The section finish date and time.
colChildren : UTDCoreJEvents
The collection of presentations or sub-sections for the section.
oSelection : UTDCore.ISelection
The selection plugin for this section.
oTemplate : UTDCore.cTemplate
The tmeplate for the section.
Page 39
W ASH I N GTON 1 53548V I oTiπter : UTDCore.iTin.er
The timer plugin for the section.
dtTimeRemaining : DATE
Returns the time remaining in the section. It get the seconds remaining from the iTimer plugin. It then converts it to a variant DATE type (aka VT_DATE).
colDelivered : UTDCore.IEvents
The collection of presentations or sub-sections for the section that have been presented. They are ordered by the order they appeared in.
bComment : Boolean = FALSE
The commenting mode of the section. Normally set by the Navigation.
bltemsReadOnly : Boolean = FALSE
The read-only mode of the items in the section. Normally set by the Navigation. Read only items cannot have their responses changed.
nCurChild : long = 0
Index of last delivered event in colDelivered. Read-only.
Public Operations:
ChildNext 0 ■
Requests that the driver proceeds to the next presentation or child section.
ChildPrevious () :
Requests that the driver proceeds to the previous presentation or child section.
ChildGoto (vPresentation : Variant) :
Requests that the driver proceeds to a named or numbered presentation or child section..
Page 40
W AS H INGTON 1 53548 V 1 /043255
135
UTDCore Results (Interfaces)
Interface supported by plugins that handles writing of the candidate results.
Public Operations:
WriteResults 0 :
Write the exam results.
UTDCore.eCategoryContent (Interfaces)
Enumeration of possible values for ICategory::eContents
// These types of categories that correspond to the XXL categorycontents eCategoryContent_Anything eCategoryContent_Sections eCategoryContent_Items eCategoryContent_Categories
UTDCore E vents (Interfaces)
IEvents contains deliverable classes as IDispatch objects. They must be queried individually to determine their exact type. Which types of events are returned depends on where the collection comes from.
IEvents can contain: IExam, IForm, ISection, Icltem, IPresentation, Icltem, IcReport, and IcResults.
Derived from UTDCore.Collection
Page 41
WASHINGTON I 535 8V I , „,,_.,
WO 2003/043255
136
UTDCore.IPersistlnstanceSet (Interfaces)
Interface for a plugin that wants to be persistant in the instance file as a property set.
Public Attributes:
IsDirty : Boolean
Returns TRUE if the object needs to save state to the instance file.
Public Operations:
Save (oPropSet : POLESS.IPropertyStorageVB) :
Called by the test driver to save the plugin data to the instance file. Reload (oPropSet : POLESS.IPropertyStorageVB) :
Called by the test driver to reload the plugin data from the instance file. The IPersistResource*::Load will be called next.
Administration System
This is one or more components that make up the administration system. This will vary by channel and lime.
UAS.ITransfer (Interfaces)
Administration interface to transfer results and other files back to the data center. It includes routing.
It is also emulated by UTD2ETS and Launchtest components that emulate the UAS.
UAS.IPrint (Interfaces)
Page 42
WASHINGTON 1 53548V I The administration system interface for save score reports for reprint. Also handles initial printing.
It is also emulated by UTD2ETS and Launchtest components that emulate the UAS.
UAS.IAppointment (Interfaces)
This interface is part of the Unified Administration System. It allows access to the candidate information for the candidate taking this exam.
It is also emulated by UTD2ETS and Launchtest components that emulate the UAS.
Page 43
WAS H I NG TON I 53548 V 1 TOTALS:
2 Components
51 Classes
COMPONENT PACKAGE STRUCTURE
Component View
Page 44
WASHINGTON 15-548V!
APPENDIX B-
POLESS CLASSES AND INTERFACES
TABLE OF CONTENTS
TABLE OF CONTENTS 2
SELECTED LOGICAL VIEW REPORT 3
LOGICAL VIEW 3
CAlgorithm 3
CEnumAlgorithm 3
CEnumProviders 3
CProvider 3 lAlgorithm 4 lCrypto 4
IEnumAlgorithms 5 lEnumProviders 5
IFileRoot 7
POLESS.lPropertyStorageAmalgamated 8
POLESS.IPropertyStorageVB 8
POLESS.lStorageAmalgamated 9
POLESS.iStorageVB 10
POLESS.IStreamVB 12
POLESS.kStorage 13
POLESS.lcStorageRoot. 13
POLESS.cCrypto 14
POLESS.cFileRoot 14
POLESS.cPropertyStorage 15
POLESS. cPropertyStorageAmalgamated 75
POLESS.cStorage 15
POLESS.cStorageAmalgamated 15
POLESS.cStorageRoot 75
POLESS.cStream 75
POLESS.iStorageVB 75
POLESS.iStreamVB 75
TOTALS: 16
LOGICAL PACKAGE STRUCTURE 16
Page 2
WAS H I N G TO N 1 5_ 55 V ]
Figure imgf000142_0001
IProvider fflgβt_ι ()
Hget_TypeO get_lengt O Hget_Name() get_minLength() _maxLength() Jjget_num P rotocolsO
Hge nameO
{ggeUongNameO iϋget_class0
SELECTED LOGICAL VIEW REPORT
_ ^^m61.cS_l]|g^Vie,wiM*«%Φ5-3-.»^ϊ|Sl&
CAlgorithm
CEnumAlgorithm
CEnumProviders
CProvider
Page 3
WASHINGTON 153554V1 IAIgorithm
Public Operations:
get_id (pAlgld : LONG*) : HRESULT
getjength (pdwLength : LONG*) : HRESULT
g- minLength (dwMinLength : LONG*) : HRESULT
get_maxLength (dwMaxLength : LONG*) : HRESULT
get_n-ιmProtocols (dwNumProtocols : LONG*) : HRESULT
geUiame (sName : BSTR*) : HRESULT
getJongName (sName : BSTR*) : HRESULT
get_clas„ (peAlgClass : eAlgorithmClass) : HRESULT
ICrypto
Encryption/Decryption interface. Public Operations:
get_Pπ>viderName (psProviderName : BSTR*) : HRESULT
Returns the name of the Cyrto provider. put_ ProviderName (sProviderName : BSTR) : HRESULT
Sets the name of the Cyrto provider. get_Password (sPassword : BSTR* = "") : HRESULT
Used for Sponsor resource files only. putJPassword (sPassword : BSTR) : HRESULT
Used for Sponsor resource files only. get_FileType (eFileType : eFILE_TYPE) : HRESULT
pu._FileType (eFileType : eFILE IΥPE) : HRESULT
Page 4
W AS H ING TON 1 53554 V 1 LOGICAL VIEW'REPORT
get_Algorithm (cAlgorith : eALGORITHM) : HRESULT
put_Algorithm (eAIgorithm : eALGORITHM) : HRESULT
EmimProviders (eProvType : eProviderType, ppenum : IEnumProviders**) : HRESULT
EnumAIgorithms (sProvName : BSTR, eAlgClass : eAlgorithmClass, ppenum : lEnumAlgorithms**) : HRESULT
lEnumAlgorithms
Public Operations:
Next (ppProv : .Algorithm**) : HRESULT
Returns the next algorithm interface, or NULL if there are no more algorithms. Skip (celt : ULONG) : HRESULT
Skips over the next specified number of algorithms. Reset 0 : HRESULT
Resets the enumerator to the beginning. Clone (ppenum : lEnumAlgorithms) : HRESULT
Creates another enumerator that contains the same enumeration state as the current one.
IEnumProviders
Public Operations:
Next (ppProv : -Provider**) : HRESULT
Returns the next provider interface, or NULL if there are no more providers. Skip (celt : ULONG) : HRESULT
Skips over the next specified number of providers. Reset 0 : HRESULT
Resets the enumerator to the beginning.
Page 5
WASHINGTON I 5355<tV I LOGICAL VIEW REPORT
Figure imgf000145_0001
Clone (ppenum : lEnumProvider**) : HRESULT
Creates another enumerator that contains the same enumeration state as the current one
Page 6
WASHINGTON I5355 V] IFileRoot
The root POLESS interface. This interface should only be used to open one file ata a time, and should not be released until all other storage, stream and propertystorage interfaces and released and the file is ready to be closed.
Public Operations:
StorageFileCreate (sFileName : String, eBlockSize : eBLOCK_SIZE, eAccessMode : eACCESS_MODE, bCo pression : Boolean, bEncrypt : Boolean, oCrypto : POLESS.cCrypto, bCRC : Boolean) : iStorage
Creates a new storage file. Returns the root storage interface. Marks the new structured storage file as a POLESS file by storing the CLSID of this class in a stream in the root storage.
StorageFileOpcn (sFileName : String, eAccessMode : eACCESS_MODE) : iStorage
Open an existing storage file. Returns the root storage interface.
The CRC is checked if enabled and read access mode is selected. An error is returned if the CRC fails to match.
CryptoGet 0 : POLESS.cCrypto
Gets a default configured crypto class. It should be set and used on the open or create of of the storage file. bStorageFile (sFileName : String) : Boolean
Return TRUE if the file provided is an OLE structured sotrage file and a POLELESS storage file.
StorageAmalgamatedGet 0 : POLESS.cStorageAmalgamated
Gets an empty cStorageAmalgamted. PropertyStorageAmalgamatedGet 0 : POLESS.cPropertyStorageAmalgamated
DeltaFileCreate (sFileNameOriginal : String, sFileNameUpdate : String, sFileNameDelta : String, bEncrypt : Boolean, oCrypto : POLESS.cCrypto, bCRC : Boolean) :
Create a POLESS difference file. It compares the original poless file to the updated poless file and create a delta poless file.
The delta file contains branch additions and branch deletions to the orginal poless tress to create the updated poless tree. it contains the CRC of the original file and the CRC of the update file.
DeltaFileApply (sFileNameOriginal : String, sFileNameUpdate : String, sFileNameDelta : String) :
Page 7
W AS H INGTON 1 5355 V I Applies a POLESS delta file. It applies to the original poless file to the delta poless file and create an updated poless file.
The CRC in the delta file for the original poless file in compared to the original file's calculated CRC.
If they match then the deltas are allied to create the update poless file. The CRC of the update file is calculated and compared to the update file
CRC in the delta file.
GetObjectFromPath (sFullPath : BSTR, eAccessMode : eACCESS_MODE, ppDisp : IDispatch **) : HRESULT
Uses monikers to retrieve the object named by the path. Returns a IDispatch pointer to the object retrieved.
CreateStreamFromFile (sName : BSTR, ppDisp : IDispatch**) : HRESULT
Creates a structured storage stream and populates it with the contents of the file. CreateStreamFromBSTR (sin : BSTR, ppDisp : IDispatch**) : HRESULT
Creates a structured storage stream and fills it with the specified BSTR. MemoryStreamFromStream (pStreamln : IStream*, ppDisp : IDispatch**) : HRESULT
Used to copy a stream to a newly created memory stream object, seek pointers for both streams are reset to beginning of stream after operation.
GetBindCtx (ppBindCtx : IBindCtx**) : HRESULT
Returns the static bind context that is used for creating monikers.
POLESS.IPropertyStorageAmalgamated
Public Operations:
PropertyStorageAdd (oPropertySet : IDispatch*, bEnd : VARIANT.BOOL = TRUE) : HRESULT
Add a PropertySet to the collection of PropertySets. ClearStorage () : HRESULT
Clears the collection of PropertySets.
POLESS.IPropertyStorageVB
Manages the persistent properties of a single property set.
Page 8
W A SHINGTON I 53554 V 1 Public Operations:
ReadVB (sName : BSTR, ppvVal : VARIANT**) : HRESULT
Read the value of a specified property from the property set. WriteVB (sName : BSTR, pvVal : VARIANT*) : HRESULT
Write a value for a specified property to the property set. If the property does not exist the property/value pair will be created. If the property already exist the value will be updated if open in eAccess_Write mode.
Delete (sName : BSTR) : HRESULT
Remove a property from the property set. CommitVB (grfFlags : DWORD) : HRESULT
RevertVB 0 : HRESULT
SetClass (sProgld : BSTR) : HRESULT
get_nCount (nCount : short*) : HRESULT
Returns the count of properties in the property set. Copy To (pDest : IPropertyStorageVB *) : HRESULT
Copies the contents of the source property set to a destination property set. GetName (nlndex : short, sName : BSTR*) : HRESULT
Returns the name of the specified property.
POLESS.IStorageAmalgamated
Public Operations:
StorageAdd (oStorage : IStorage, bEnd : VARIANT_BOOL = TRUE) :
Adds a new storage to the collection of storages. ClcarStorage () :
Clears all the storage objects from the collection.
Page 9
W ASHINGTON 1 53554V 1 POLESS.iStorageVB
Public Operations:
Clear 0 :
Clears the storage of all elements: sub-storages and streams. CommitVB 0 :
Ensures that any changes made to a storage object open in transacted mode are reflected in the parent storage. For nonroot storage objects in direct mode, this method has no effect. For a root storage, it reflects the changes in the actual device, for example, a file on disk. For a root storage object opened in direct mode, always call the Commit method prior to Releasing the object. Commit flushes all memory buffers to the disk for a root storage in direct mode and will return an error code upon failure. Although Releasing the object also flushes memory buffers to disk, it has no capacity to return any error codes upon failure. Therefore, calling Releasing without first calling Commit causes indeterminate results.
RevertVB () :
Discards all changes that have been made to the storage object since the last commit operation.
sElementName (vElement : Variant) : String
Returns the name of the element. bStoragc (vElement : Variant) : Boolean
Returns TRUE if the element is a sub-storage oElement (vElement : Variant) : Object
Returns either POLESS.iStreamVB or POLESS.iStorageVB for the selected element. CreateStream (sName : String, eAccess : eACCESS_MODE, bCompression : Boolean) : POLESS.iStreamVB
Creates and opens a stream object with the specified name contained in this storage object. All elements within a storage object — both streams and other storage objects — are kept in the same name space.
Nothing is return if the stream cannot be created.
Page 10
W AS H I NGTON I 53554V 1 5
149
LOGICAL VIEW REPORT
OpenStream (sName : String, eAccess : eACCESS_MODE) : POLESS.iStreamVB
Opens an existing stream object within this storage object in the specified access mode.
If the stream name is not found, we will look for a stream name prefixed with "\03". This is a linked stream. The contents of this stream is the file and storage where to find this stream.
Nothing is return if the stream cannot be opened.
CreateStorage (sName : String, eAccess : eACCESSJMODE) : POLESSJStorageVB
Creates and opens a new storage object nested within this storage object. Nothing is return if the storage cannot be created.
OpenStorage (sName : String, eAccess : eACCESSJVIODE) : POLESS.iStorageVB
Opens an existing storage object with the specified name in the specified access mode. Nothing is return if the storage cannot be opened.
get_sName (sName : BSTR*) : HRESULT
Returns the name of the storage. get_oStorage (ppDisp : IDispatch**) : HRESULT
Returns the IDispatch interface. get.nCount (pnCount : short*) : HRESULT
Returns the count of elements in the storage. GetCompression () : Boolean
Determine if streams may be compressed in the file. If enabled streams may optionally be compressed when created.
GetEncryptioπ 0 : Boolean
Determine if encryption is enabled for the file. If enabled all streams will be encrypted. GetCRC 0 : Boolean
Indicates whether a CRC check is to be performed on the file.
Page 1 J
WASHINGTON 1 53554V 1 CreateStreamLinked (sName : BSTR, sLocation : BSTR, sFile : BSTR, eAccess : eACCESS_MODE, ppStreamVB : IStreamVB**) : HRESULT
CreatePropertyStg (sName : BSTR, grfFlags : DWORD, bCompress : VARIANT_BOOL, ppPropStg : IPropertyStorageVB) : HRESULT
Create a property storage.
OpenPropertyStg (sName : BSTR, grfFlags : DWORD, dwReserved : DWORD, ppPropStg : IPropertyStorageVB) : HRESULT
SetCIass (sProgld : BSTR) : HRESULT
RegisterAIias (sName : BSTR) : HRESULT
Destroy (sName : BSTR) : HRESULT
Destroys the specified element.
POLESS.IStreamVB
Public Operations:
ReadVB (bytData : byteO, nBytes : Integer) : HRESULT
Read data from the stream. Write VB (bytData : byteO) : HRESULT
Write data to the stream. The entire byte array is written. Clear 0 : HRESULT
Clears the stream of all data. Reset 0 : HRESULT
Reset the position in the stream to the begining. get_sName (sName : BSTR*) : HRESULT
Returns the name of the stream. get_oStream (ppDisp : IDispatch**) : HRESULT
Returns the IDispatch interface. Copy To (pDest : IStreamVB*) : HRESULT
Page 12
W ASH IN GTO N I 53554 V 1 Copies the contents of a source stream to a destination stream.
POLESS.IcStorage
Derived from IStorage
Public Operations:
CreatePropertyStg (sName : BSTR, grfFlags : DWORD, bCompress : Boolean, ppPropStg : IPropertyStorage**) : HRESULT
Creates and opens a property set in a stream object.
OpenPropertyStg (sName : BSTR, grfFlags : DWORD, dwReserved : DWORD, ppPropStg : IPropertyStorage**) : HRESULT
Opens an existing property set in a specified stream object.
CreateStreamLinked (sName : BSTR, sLocation : BSTR, sFile : BSTR, eAccess : eAccess_MODE, ppStream : IStream**) : HRESULT
RegisterAlias (sName : BSTR) : HRESULT
POLESS cStorageRoot
Derived from POLESS.IcStorage
Public Operations:
get_Compression (pbCompress : VARIANT_BOOL*) : HRESULT
Determine if streams may be compressed in the file. If enabled streams may optionally be compressed when created. get_Encryption (pbEπcrypt : VARIANT_BOOL*) : HRESULT
Determine if encryption is enabled for the file. If enabled all streams will be encrypted. get_CRC 0 : HRESULT
Indicates whether a CRC check is to be performed on the file.
GetObjectFromPath (sltemPath : BSTR, eAccessMode : eACCESS_MODE, refiid : REFIID, ppUnk : IUnknown**) : HRESULT
Page 1 3
WAS HINGTON 1 53554V 1 POLESS.cCrypto
The class control the configuration of the encyφtion/decryption of the structured storage. Public Attributes:
sProviderName : String = MS_DEF_PROV
The name of the Cyrto provider. eProviderType : ePROVTDER TYPE = PROV_RSA_FULL
The type of crypto provider.
Cryptographic Provider Types
The field of cryptography is large and growing. There are many different standard data formats and protocols. These are generally organized into groups or families, each of which has its own set of data formats and way of doing things. Even if two families use the same algorithm (for example, the RC2 block cipher), they will often use different padding schemes, different key lengths, and different default modes. Microsoft® CryptoAPI is designed so that a CSP provider type represents a particular family. ePROV_RSA_FULL ePROV_RSA_SIG ePROV_RSA_SCHANNEL ePROV_DSS ePROV_DSS_DH ePROV_DH_SCHANNEL ePROV_FORTEZZA ePROV_MS_EXCHANGE ePROV SSL
sContainerName : String
Key name. No default, must be provided by used. sPassword : String
Optional password on the public/private key pair. Only for entery by a human. Can be used for Review disks and their resource files.
POLESS.cFileRoot
The root POLESS class. Must be instanced to perform any POLESS functions. Handles creating and opening POLESS files.
Page 14
W AS H I NGTO N I 53554V I POLESS.cPropertyStorage
POLESS.cPropertyStorageAmalgamated
POLESS-cStorage
POLESS implementation of iStorage. It handles anything POLESS specific and then deligates work to OLE2 compound document Storage class.
POLESS.cStorageAmalgamated
POLESS implementation of iStorage. The class hold references to an order collection of iStorage objects.
When a stream is opened it search the collection of storage objects in order to find the first storage object that has the requested stream. It returns this stream.
It handles compund stream resolution and deligates all other work to POLESS.cStorage. This storage is read-only. It will not allow stream or storages to be created. It primaryly for reading the exam resource file.
Note: This has nothing to do with compund documents. POLESS.cStorageRoot
POLESS implementation of iStorage and iStorageRoot. It handles anything root POLESS specific and then deligates work to the standard POLESS Storage class.
POLESS.cStream
POLESS implementation of iStream. It handles anything POLESS specific and then deligates work to OLE2 compound document stream class. The specific work includes compression/decompression and encryption/decryption of the stream.
POLESS.iStorageVB
A VB friendly storage interface POLESS.iStreamVB
A VB friendly stream interface
Page 1 5
WAS HI N GTON 1 53554 V I LOGICAL VIEW REPORT
TOTALS:
1 Logical Packages 26 Classes
LOGICALPACKAGE STRUCTURE
Logical View OLE2SS
Page 16
WASHINGTON I 53554V1

Claims

What is claimed is:
1. A system for computer based testing for at least one test, the at least one test having a presentation format and data content, comprising: a storage device comprising a first storage location and a second storage location, the first storage location storing a first segment of a test definition language and the second storage location storing a second segment of the test definition language, wherein the first segment and the second segment define information comprising at least one of the data content, the presentation format, progression, scoring, printing, timing, and results reporting of the at least one test; a validation expansion module, in operative data communication with the storage device, that validates the first segment and the second segment of the test definition language by determining whether the first segment and the second segment are correctly formatted, and stores the first segment to one of the first storage location and the second storage location and the second segment to another one of the first storage location and the second storage location in tϊie storage device; a test packager, in operative data communication with the storage device and the validation expansion module, that, during production of the at least one test, transmits the first segment and the second segment of the test definition language to the validation expansion module, determines to which of the first storage location and the second storage location in the storage device the first segment and the second segment are stored by the validation expansion module, amalgamates the first storage location and the second storage location and stores an amalgamated segment of the test definition language in a first virtual storage location, and transmits the amalgamated segment to the validation expansion module such that the validation expansion module is capable of determining whether the amalgamated segment forms a complete and valid set of the first segment and second segment of the test definition language; and a test driver, in operative data communication with the storage device and the validation expansion module, comprising an executable code that controls functionality that enables the test driver to deliver the at least one test to an examinee using a display device, manage the at least one test, control the progression of the at least one test, control the scoring of the at least one test, control the printing of the at least one test, control the timing of the at least one test, and control the results reporting of the at least one test based on the test definition language, the test driver, during delivery of the at least one test, amalgamating the first storage location and the second storage location into a second virtual storage location such that the validation expansion module is capable of retrieving the amalgamated segment from the second virtual storage location to enable the functionality of the test driver.
2. The system of claim 1 , wherein the test definition language comprises extensible markup language format and wherein the test packager comprises a compiler.
3. The system of claim 1 , wherein the first segment and the second segment of the test definition language further comprise a same category of information, the same category of information comprising at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and the results reporting of the at least one test.
4. The system of claim 2, wherein the same category of information further comprises at least one of non-interactive display material, test navigation, test navigation controls, items, timing, selection, scoring, results, and reporting.
5. The system of claim 1 , wherein the validation expansion module comprises a plug-in, wherein the first segment of the test definition language defines the plug-in, and wherein the second segment of the test definition language defines a usage of the plug-in on at least one unit of the at least one test.
6. The system of claim 5, wherein the similar feature defined by the first segment and the second segment of the test definition language further comprises at least one of test navigation, timing, selection, scoring, results, and reporting.
7. The system of claim 1 , further comprising a function interface that enables communication between the test packager and the validation expansion module, wherein the function interface enables the test packager to transmit the first segment and the second segment of the test definition language to the validation expansion module such that the validation expansion module is capable of validating the first segment and the second segment.
8. The system of claim 1, further comprising a persistence interface that enables communication between the validation expansion module and the storage device such that the validation expansion module is capable of storing the first segment to one of the first storage location and the second storage location and the second segment of the test definition language to another one of the first storage location and the second storage location in the storage device.
9. The system of claim 8, wherein the persistence interface enables storing of the first segment and the second segment as one of a set of data and a directory.
10. The system of claim 8, wherein the persistence interface further enables the test packager to transmit the amalgamated segment from the storage device to the validation expansion module such that the validation expansion module is capable of validating the amalgamated segment.
1 1. The system of claim 8, wherein the persistence interface further enables the validation expansion module to retrieve the amalgamated segment from the storage device such that the validation expansion module is capable of enabling the functionality of the test driver.
12. The system of claim 1, further comprising a function interface that enables communication between the test packager and the validation expansion module such that the test packager is capable of transmitting the amalgamated segment to the validation expansion module and such that the validation expansion module is capable of determining whether the amalgamated segment forms a complete and valid set of the first segment and second segment of the test definition language.
13. The system of claim 1 , further comprising a function interface that enables communication between the test driver and the validation expansion module such that such that the test packager is capable of instructing the validation expansion module to retrieve the amalgamated segment from the second virtual storage location to enable the functionality of the test driver.
14. The system of claim 1 , wherein the storage device further comprises a third storage location and the test definition language further comprises a third segment: wherein the validation module validates the third segment and stores the first segment to one of the first storage location, the second storage location, and the third storage location, stores the second segment to another one of the first storage location, the second storage location, and stores the third storage location, and the third segment of the test definition language to another one of the first storage location, the second storage location, and the third storage location, wherein the test packager, during production of the at least one test, transmits the third segment to the validation expansion module such that the validation expansion module is capable of validating the third segment, determines to which of the first storage location, the second storage location, and the third storage location in the storage device the first segment, the second segment, and the third segment of the test definition language are stored by the validation expansion module, amalgamates the first storage location, the second storage location, and the third storage location and stores an amalgamated segment of the test definition language in a first virtual storage location, and transmits the amalgamated segment to the validation expansion module such that the validation expansion module is capable of determining a complete and valid set of the first segment, the second segment, and the third segment of the test definition language, and wherein the test driver, during delivery of the at least one test, amalgamates the first storage location, the second storage location, and the third storage location into the second virtual storage location such that the validation expansion module is capable of retrieving the amalgamated segment from the second virtual storage location to enable the functionality of the test driver.
15. The system of claim 14: wherein the validation expansion module comprises a plug-in, wherein the first segment of the test definition language defines the plug-in, wherein the second segment of the test definition language defines an area in a template in which the plug-in is to be used, the template determining a visual presentation of the at least one test on the display device, and wherein the third segment of the test definition language defines a presentation in which the plug-in is to be used, the presentation determining the visual presentation of the at least one test on the display device at a particular instance during the at least one test.
16. The system of claim 14, wherein the first segment, the second segment, and the third segment of the test definition language further comprise a same category of information, the same category of information comprising at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and the results reporting of the at least one test.
17. The system of claim 16, wherein the same category of information further comprises at least one of non-interactive display material, test navigation controls, and items.
18. The system of claim 14: wherein the validation expansion module comprises a plug-in, the first segment of the test definition language defines the plug-in; wherein the second segment of the test definition language defines an area in a template in which the plug-in is to be used, the template determining a visual presentation of the at least one test on the display device, wherein the third segment of the test definition language defines an item, which includes at least one question delivered to the examinee during the at least one test, and wherein the similar feature defined by the first segment, the second segment, and the third segment of the test definition language is items.
19. The system of claim 14, further comprising a function interface that enables communication between the test packager and the validation expansion module, wherein the function interface enables the test packager to transmit the first segment, the second segment, and the third segment of the test definition language to the validation expansion module such that the validation expansion module is capable of validating the first segment, the second segment, and the third segment.
20. The system of claim 14, further comprising a persistence interface that enables communication between the validation expansion module and the storage device such that the validation expansion module is capable of storing the first segment to one of the first storage location and the second storage location and the second segment of the test definition language to another one of the first storage location and the second storage location in the storage device.
21. The system of claim 20, wherein the persistence interface enables storing of the first segment, the second segment, and the third segment as one of a set of data and a directory.
22. The system of claim 20, wherein the persistence interface further enables the test packager to transmit the amalgamated segment from the storage device to the validation expansion module such that the validation expansion module is capable of validating the amalgamated segment.
23. The system of claim 20, wherein the persistence interface further enables the validation expansion module to retrieve the amalgamated segment from the storage device such that the validation expansion module is capable of enabling the functionality of the test driver.
24. The system of claim 14, further comprising a function interface that enables communication between the test packager and the validation expansion module such that the test packager is capable of transmitting the amalgamated segment to the validation expansion module and such that the validation expansion module is capable of determining whether the amalgamated segment forms a complete and valid set of the first segment and second segment of the test definition language.
25. The system of claim 14, further comprising a function interface that enables communication between the test driver and the validation expansion module such that such that the test packager is capable of instructing the validation expansion module to retrieve the amalgamated segment from the second virtual storage location to enable the functionality of the test driver.
26. A system for computer based testing for at least one test, the at least one test having a presentation format and data content, comprising: a storage device comprising a first storage location, a second storage location, and a third storage location, the first storage location storing a first segment of a test definition language, the second storage location storing a second segment of the test definition language, and the third storage location storing a third segment of the test definition language, wherein the first segment, the second segment, and the third segment comprise at least one of the data content, the presentation format, progression, scoring, printing, timing, and results reporting of the at least one test; a validation expansion module, in operative data communication with the storage device, that validates the first segment, the second segment, and the third segment of the test definition language, and stores the first segment to one of the first storage location, the second storage location, and the third storage location, the second segment to another one of the first storage location, the second storage location, and the third storage location, and the third segment to another one of the first storage location, the second storage location, and the third storage location; a test packager, in operative data communication with the storage device and the validation expansion module, that, during production of the at least one test, transmits the first segment, the second segment, and the third segment of the test definition language to the validation expansion module such that the validation expansion module is capable of validating the first segment, the second segment, and the third segment, determines to which of the first storage location, the second storage location, and the third storage location in the storage device the first segment, the second segment, and the third segment are stored by the validation expansion module, amalgamates the first storage location, the second storage location, and the third storage location and stores an amalgamated segment of the test definition language in a first virtual storage location, and transmits the amalgamated segment to the validation expansion module such that the validation expansion module is capable of determining a complete and valid set of the first segment, the second segment, and the third segment of the test definition language; and a test driver, in operative data communication with the storage device and the validation expansion module, comprising an executable code that controls functionality performed by the test driver that enables the test driver to deliver the at least one test to an examinee using a display device, manage the at least one test, control progression of the at least one test, control scoring of the at least one test, control printing of the at least one test, control timing of the at least one test, and control reporting of test results based on the test definition language, the test driver, during delivery of the at least one test, amalgamating the first storage location, the second storage location, and the third storage location into a second virtual storage location such that the validation expansion module is capable of retrieving the amalgamated segment from the second virtual storage location to enable the functionality of the test driver.
27. The system of claim 26, wherein the validation expansion module comprises a plug-in, wherein the first segment of the test definition language defines the plug-in, wherein the second segment of the test definition language defines an area in a template in which the plug-in is to be used, the template determining a visual presentation of the at least one test on the display device, and wherein the third segment of the test definition language defines a presentation in which the plug-in is to be used, the presentation determining the visual presentation of the at least one test on the display device at a particular instance during the at least one test.
28. The system of claim 26, wherein the first segment, the second segment, and the third segment of the test definition language further comprise a same category of information, the same category of information comprising at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and the results reporting of the at least one test.
29. The system of claim 28, wherein the same category of information further comprises at least one of non-interactive display material, test navigation controls, and items.
30. The system of claim 26: wherein the validation expansion module comprises a plug-in, wherein the first segment of the test definition language defines the plug-in, wherein the second segment of the test definition language defines an area in a template in which the plug-in is to be used, the template determining a visual presentation of the at least one test on the display device wherein the third segment of the test definition language defines an item, which includes at least one question delivered to the examinee during the at least one test, and wherein the first segment, the second segment, and the third segment further comprise a same category of information, the same category of information comprising items.
31. The system of claim 26, further comprising a function interface that enables communication between the test packager and the validation expansion module, wherein the function interface enables the test packager to transmit the first segment, the second segment, and the third segment of the test definition language to the validation expansion module such that the validation expansion module is capable of validating the first segment, the second segment, and the third segment.
32. The system of claim 26, further comprising a persistence interface that enables communication between the validation expansion module and the storage device such that the validation expansion module is capable of storing the first segment to one of the first storage location and the second storage location and the second segment of the test definition language to another one of the first storage location and the second storage location in the storage device.
33. The system of claim 32, wherein the persistence interface enables storing of the first segment, the second segment, and the third segment as one of a set of data and a directory.
34. The system of claim 32, wherein the persistence interface further enables the test packager to transmit the amalgamated segment from the storage device to the validation expansion module such that the validation expansion module is capable of validating the amalgamated segment.
35. The system of claim 32, wherein the persistence interface further enables the validation expansion module to retrieve the amalgamated segment from the storage device such that the validation expansion module is capable of enabling the functionality of the test driver.
36. The system of claim 26, further comprising a function interface that enables communication between the test packager and the validation expansion module such that the test packager is capable of transmitting the amalgamated segment to the validation expansion module and such that the validation expansion module is capable of determining whether the amalgamated segment forms a complete and valid set of the first segment and second segment of the test definition language.
37. The system of claim 26, further comprising a function interface that enables communication between the test driver and the validation expansion module such that such that the test packager is capable of instructing the validation expansion module to retrieve the amalgamated segment from the second virtual storage location to enable the functionality of the test driver.
38. A system for computer based testing for at least one test, the at least one test having a presentation format and data content, comprising: first storage means for storing a first segment of a test definition language; second storage means for storing a second segment of the test definition language, wherein the first segment and the second segment define information comprising at least one of the data content, the presentation format, progression, scoring, printing, timing, and results reporting of the at least one test; validation means, in operative data communication with the first storage means and the second storage means, for validating the first segment and the second segment of the test definition language, and storing the first segment to one of the first storage means and the second storage means and the second segment to another one of the first storage means and the second storage means; test packager means, in operative data communication with the first storage means, the second storage means, and the validation means, for, during production of the at least one test, transmitting the first segment and the second segment of the test definition language to the validation means such that the validation means is capable of validating the first segment and the second segment, determining to which of the first storage means and the second storage means the first segment and the second segment are stored by the validation means, amalgamating the first storage means and the second storage means and storing an amalgamated segment of the test definition language in a third storage means, and transmitting the amalgamated segment to the validation means such that the validation means is capable of determining a complete and valid set of the first segment and second segment of the test definition language; and test driver means, in operative data communication with the first storage means, the second storage means, and the validation means, for amalgamating the first storage means and the second storage means into a third storage means such that the validation means is capable of retrieving the amalgamated segment from the third storage means, the test driver means comprising an executable code that controls functionality performed by the test driver means that enables the test driver means to deliver the at least one test to an examinee using a display device, manage the at least one test, control progression of the at least one test, control scoring of the at least one test, control printing of the at least one test, control timing of the at least one test, and control reporting of test results based on the test definition language, wherein the validation means enables the functionality of the test driver means.
39. The system of claim 38, wherein the first segment and the second segment of the test definition language further comprise a same category of information, the same category of information comprising at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and the results reporting of the at least one test.
40. The system of claim 38, further comprising function interface means that enables communication between the test packager means and the validation means.
41. The system of claim 38, further comprising persistence interface means that enables communication between the validation means and the first storage means and the second storage means.
42. The system of claim 38, further comprising function interface means that enables communication between the test packager means and the validation means such that the test packager means is capable of transmitting the amalgamated segment to the validation means and such that the validation means is capable of determining whether the amalgamated segment forms a complete and valid set of the first segment and second segment of the test definition language.
43. The system of claim 38, further comprising function interface means that enables communication between the test driver means and the validation means such that such that the test packager means is capable of instructing the validation means to retrieve the amalgamated segment from the second virtual storage location to enable the functionality of the test driver.
44. A system for computer based testing for at least one test, the at least one test having a presentation format and data content, comprising: first storage means for storing a first segment of a test definition language; second storage means for storing a second segment of the test definition language; third storage means for storing a third segment of the test definition language, wherein the first segment, the second segment, and the third segment define information comprising at least one of the data content, the presentation format, progression, scoring, printing, timing, and results reporting of the at least one test; validation means, in operative data communication with the first storage means, the second storage means, and the third storage means, for validating the first segment, the second segment, and the third segment of the test definition language, and storing the first segment to one of the first storage means, the second storage means, and the third storage means, the second segment to another one of the first storage means, the second storage means, and the third storage means, and the third segment to another one of the first storage means, the second storage means, and the third storage means; test packager means, in operative data communication with the first storage means, the second storage means, the third storage means, and the validation means, for, during production of the at least one test, transmitting the first segment, the second segment, and the third segment of the test definition language to the validation means such that the validation means is capable of validating the first segment, the second segment, and the third segment, determining to which of the first storage means, the second storage means, and the third storage means the first segment, the second segment, and the third segment are stored by the validation means, amalgamating the first storage means, the second storage means, and the third storage means and storing an amalgamated segment of the test definition language in a fourth storage means, and transmitting the amalgamated segment to the validation means such that the validation means is capable of determining a complete and valid set of the first segment, the second segment, and the third segment of the test definition language; and test driver means, in operative data communication with the first storage means, the second storage means, the third storage means, and the validation means, for, during delivery of the at least one test, amalgamating the first storage means, the second storage means, and the third storage means into a fourth storage means such that the validation means is capable of retrieving the amalgamated segment from the fourth storage means, the test driver means comprising an executable code that controls functionality performed by the test driver means that enables the test driver means to deliver the at least one test to an examinee using a display device, manage the at least one test, control progression of the at least one test, control scoring of the at least one test, control printing of the at least one test, control timing of the at least one test, and control reporting of test results based on the test definition language, wherein the validation means enables the functionality of the test driver means.
45. The system of claim 44, wherein the first segment, the second segment, and the third segment of the test definition language further comprise a same category of information, the same category of information comprising at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and the results reporting of the at least one test.
46. The system of claim 44, further comprising function interface means that enables communication between the test packager means and the validation means.
47. The system of claim 44, further comprising persistence interface means that enables communication between the validation means and the first storage means, the second storage means, and the third storage means.
48. The system of claim 44, further comprising function interface means that enables communication between the test packager means and the validation means such that the test packager means is capable of transmitting the amalgamated segment to the validation means and such that the validation means is capable of determining whether the amalgamated segment forms a complete and valid set of the first segment and second segment of the test definition language.
49. The system of claim 44, further comprising function interface means that enables communication between the test driver means and the validation means such that such that the test packager means is capable of instructing the validation means to retrieve the amalgamated segment from the second virtual storage location to enable the functionality of the test driver.
50. A method for computer based testing for at least one test, the at least one test having a presentation format and data content, the at least one test being controlled by a test driver, the test driver having an executable code that controls the test driver and functionality that enables the test driver to deliver the at least one test to an examinee using a display device, manage the at least one test, control progression of the at least one test, control scoring of the at least one test, control printing of the at least one test, control timing of the at least one test, and control results reporting of the at least one test based on a test definition language, the test definition language comprising a plurality of segments, the method comprising the steps of: validating a first segment of the test definition language during a test production cycle; validating a second segment of the test definition language during the test production cycle, wherein the first segment and the second segment define information comprising at least one of the data content, the presentation format, progression, scoring, printing, timing, and results reporting of the at least one test; amalgamating the first segment and the second segment of the test definition language during the test production cycle, wherein an amalgamated segment is formed; validating the amalgamated segment during the test production cycle, wherein a validated amalgamated segment is created and wherein the amalgamated segment is valid if the amalgamated segment forms a complete and valid set; and amalgamating the first segment and the second segment of the test definition language during a test delivery cycle, wherein the validated amalgamated segment is reformed and retrieved by a validation expansion module to enable the functionality of the test driver.
51. The method of claim 50, wherein the first segment and the second segment of the test definition language comprise a same category of information, the same category of information being at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and the results reporting of the at least one test.
52. The method of claim 51, wherein the same category of information further comprises at least one of test navigation, timing, selection, scoring, printing, timing, results, and reporting.
53. The method of claim 50, further comprising the step of instantiating the validation expansion module during the test production cycle, wherein validating the first segment and the second segment of the test definition language is performed by the validation expansion module.
54. The method of claim 53, wherein instantiating the validation expansion module is facilitated by standard Microsoft object instantiation using a component object model server.
55. The method of claim 53, wherein the first segment and the second segment of the test definition language are transmitted to the validation expansion module by a test packager and wherein transmitting to the validation expansion module is facilitated by a function interface.
56. The method of claim 55, wherein the test packager comprises a compiler.
57. The method of claim 50, wherein the test definition language comprises extensible markup language format, validating the first segment and the second segment of the test definition language further comprising the step of determining whether the first segment and the second segment are correctly formatted.
58. The method of claim 57, wherein a correct format for the first segment and the second segment of the test definition language is defined in a schema.
59. The method of claim 57, further comprising the step of instantiating the validation expansion module during the test production cycle, wherein validating the first segment and the second segment of the test definition language is performed by the validation expansion module, instantiating the validation expansion module further comprising the step of calling the validation expansion module using a product identification defined in extensible markup language in the test definition language.
60. The method of claim 50, further comprising the step of storing the first segment to one of a first storage location and a second storage location and the second segment to another one of the first storage location and the second storage location in a storage device after the first segment and the second segment have been validated.
61. The method of claim 60, wherein the first segment and the second segment of the test definition language are stored the storage device by the validation expansion module, and wherein a test packager determines into which of the first storage location and the second storage location the first segment and the second segment are stored.
62. The method of claim 60, wherein storing the first storage location and the second storage location into the storage device is facilitated by a persistence interface.
63. The method of claim 60, wherein the test packager comprises a compiler.
64. The method of claim 50, amalgamating the first segment and the second segment of the test definition language further comprising the steps of: combining the first segment and the second segment of the test definition language according to an amalgamation rule; and storing the amalgamated segment in a virtual storage location in a storage device.
65. The method of claim 50, further comprising the step of instantiating the validation expansion module during the test production cycle, wherein validating the amalgamated segment is performed by the validation expansion module.
66. The method of claim 65, wherein instantiating the validation expansion module is facilitated by standard Microsoft object instantiation using a component object model server.
67. The method of claim 65, wherein an amalgamated interface is retrieved from a virtual storage object in a storage device by the validation expansion module and wherein communication between the virtual storage location and the validation expansion module is facilitated by a function interface.
68. The method of claim 65, wherein the first segment of the test definition language is stored to one of a first storage location and a second storage location and the second segment of the test definition language to another one of the first storage location and the second storage location in the storage device, the first segment and the second segment forming the amalgamated segment, wherein the first segment and the second segment are retrieved from the storage device by the validation expansion module in amalgamated form using a persistence interface.
69. The method of claim 65, wherein the validation expansion module determines what requirements are necessary for the amalgamated segment to form a complete and valid set.
70. The method of claim 50, further comprising the step of instantiating the validation expansion module during the test production delivery cycle, wherein validating the amalgamated segment is performed by the validation expansion module.
71. The method of claim 70, further comprising the step of loading the validated amalgamated segment into the validation expansion module, wherein the validated amalgamated segment is retrieved from a virtual storage location in a storage device by the validation expansion module and wherein communication between the virtual storage location and the validation expansion module is facilitated by a function interface.
72. The method of claim 70, wherein the first segment of the test definition language is stored to one of a first storage location and a second storage location and the second segment of the test definition language to another one of the first storage location and the second storage location in the storage device, the first segment and the second segment forming the amalgamated segment, wherein the first segment and the second segment are retrieved from the storage device by the validation expansion module in amalgamated form using a persistence interface.
73. A method computer based testing for at least one test, the at least one test having a presentation format and data content, the at least one test being controlled by a test driver, the test driver having an executable code that controls the test driver and functionality that enables the test driver to deliver the at least one test to an examinee using a display device, manage the at least one test, control progression of the at least one test, control scoring of the at least one test, control printing of the at least one test, control timing of the at least one test, and control results reporting of the at least one test based on a test definition language, the test definition language comprising a plurality of segments, the method comprising the steps of: validating a first segment of the test definition language during a test production cycle; validating a second segment of the test definition language during the test production cycle; validating a third segment of the test definition language, wherein the first segment, the second segment, and the third segment define information comprising at least one of the data content, the presentation format, progression, scoring, printing, timing, and results reporting of the at least one test; amalgamating the first segment, the second segment, and the third segment of the test definition language during the test production cycle, wherein an amalgamated segment is formed; validating the amalgamated segment during the test production cycle, wherein a validated amalgamated segment is created and wherein the amalgamated segment is valid if the amalgamated segment forms a complete and valid set; and amalgamating the first segment, the second segment, and the third segment of the test definition language during a test delivery cycle, wherein the validated amalgamated segment is reformed and retrieved by a validation expansion module to enable the functionality of the test driver.
74. The method of claim 73, further comprising the step of instantiating the validation expansion module during the test production cycle, wherein validating the first segment, the second segment, and the third segment of the test definition language is performed by the validation expansion module.
75. The method of claim 74, wherein instantiating the validation expansion module is facilitated by standard Microsoft object instantiation using a component object model server.
76. The method of claim 74, wherein the first segment, the second segment, and the third segment of the test definition language are transmitted to the validation expansion module by a test packager and wherein transmitting to the validation expansion module is facilitated by a function interface.
77. The method of claim 76, wherein the test packager comprises a compiler.
78. The method of claim 73, wherein the test definition language comprises extensible markup language format, validating the first segment, the second segment, and the third segment of the test definition language further comprising the step of determining whether the first segment, the second segment, and the third segment are correctly formatted.
79. The method of claim 78, wherein a correct format for the first segment, the second segment, and the third segment of the test definition language is defined in a schema.
80. The method of claim 78, further comprising the step of instantiating the validation expansion module during the test production cycle, wherein validating the first segment, the second segment, and the third segment of the test definition language is performed by the validation expansion module, instantiating the validation expansion module further comprising the step of calling the validation expansion module using a product identification defined in extensible markup language in the test definition language.
81. The method of claim 73, further comprising the step of storing the first segment to one of a first storage location, a second storage location, and a third storage location, storing the second segment to another one of the first storage location, the second storage location, and the third storage location, and storing the third segment of the test definition language to another one of the first storage location, the second storage location, and the third storage location after the first segment, the second segment, and the third segment have been validated.
82. The method of claim 81 , wherein the first segment is stored to one of a first storage location, a second storage location, and a third storage location, the second segment is stored to another one of the first storage location, the second storage location, and the third storage location, and the third segment is stored to another one of the first storage location, the second storage location, and the third storage location by the validation expansion module, and wherein a test packager determines into which of the first storage location, the second storage location, and the third storage location the first segment, the second segment, and the third segment are stored.
83. The method of claim 82, wherein storing the the first storage location, the second storage location, and the third storage location into the storage device is facilitated by a persistence interface.
84. The method of claim 82, wherein the test packager comprises a compiler.
85. The method of claim 73, amalgamating the first segment, the second segment, and the third segment of the test definition language further comprising the steps of: combining the first segment, the second segment, and the third segment of the test
definition language according to an amalgamation rule; and
storing the amalgamated segment in a virtual storage location in a storage device.
86. The method of claim 73, further comprising the step of instantiating the validation expansion module during the test production cycle, wherein validating the amalgamated segment is performed by the validation expansion module.
87. The method of claim 86, wherein instantiating the validation expansion module is facilitated by standard Microsoft object instantiation using a component object model server.
88. The method of claim 86, wherein an amalgamated interface is retrieved from a virtual storage object in a storage device by the validation expansion module and wherein communication between the virtual storage location and the validation expansion module is facilitated by a function interface.
89. The method of claim 86, wherein the first segment is stored to one of a first storage location, a second storage location, and a third storage location, the second segment is stored to another one of the first storage location, the second storage location, and the third storage location, and the third segment is stored to another one of the first storage location, the second storage location, and the third storage location, and the first segment, the second segment, and the third segment of the test definition language forming the amalgamated segment, and wherein the first segment, the second segment, and the third segment are retrieved from the storage device by the validation expansion module in amalgamated form using a persistence interface.
90. The method of claim 86, wherein the validation expansion module determines what requirements are necessary for the amalgamated segment to form a complete and valid set.
91. The method of claim 73, further comprising the step of instantiating the validation expansion module during the test delivery cycle, wherein validating the amalgamated segment is performed by the validation expansion module.
92. The method of claim 91 , further comprising the step of loading the validated amalgamated segment into the validation expansion module, wherein the validated amalgamated segment is retrieved from a virtual storage location in a storage device by the validation expansion module and wherein communication between the virtual storage location and the validation expansion module is facilitated by a function interface.
93. The method of claim 91 , wherein the first segment is stored to one of a first storage location, a second storage location, and a third storage location, the second segment is stored to another one of the first storage location, the second storage location, and the third storage location, and the third segment is stored to another one of the first storage location, the second storage location, and the third storage location, the first segment, the second segment, and the third segment of the test definition language forming the validated amalgamated segment, and wherein the first segment, the second segment, and the third segment are retrieved from the storage device by the validation expansion module in amalgamated form using a persistence interface.
94. The system of claim 73, wherein the first segment, the second segment, and the third segment of the test definition language further comprise a same category of information, the same category of information being at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and the results reporting of the at least one test.
95. The method of claim 94, wherein the same category further comprises at least one of non-interactive display material, test navigation controls, and items.
96. A method for computer based testing for at least one test, the at least one test having a presentation format and data content, comprising: defining the presentation format and the data content in at least two locations comprising a plugin element and actual usage of the plugin element on at least one unit of the at least one test, the at least one unit comprising form, section and group associated with the at least one test; and amalgamating the presentation format and the data content defined in the two locations by at least one test driver to deliver the at least one test to an examinee.
97. A method for computer based testing for at least one test, the at least one test having a presentation format and data content, comprising: validating by a plugin at least partial exam source information that is received; amalgamating exam resource data associated with the at least one test and validating the exam resource data that has been amalgamated to provide a substantially complete amalgamated exam specification and content; and delivering the substantially complete amalgamated exam specification and content validated by the validating step.
98. The method of claim 97, wherein the validating step further comprises the step of validating by the plugin the at least partial exam source information that is received during compilation of the at least one test.
99. The method of claim 97, wherein the validating step further comprises the step of amalgamating the exam resource data associated with the at least one test, and validating the exam resource data that has been amalgamated to provide the substantially complete amalgamated exam specification and content during test compilation.
100. The method of claim 97, wherein the delivering further comprises the step of delivering the substantially complete amalgamated exam specification and content during delivery of the at least one test to an examinee.
101. A method for computer based testing for at least one test, the at least one test having a first presentation format, a second presentation format, a first data content, and second data content, comprising: defining the first presentation format and the second presentation format in at least two locations comprising a plugin element and actual usage of the plugin element on at least one unit of the at least one test, the at least one unit comprising form, section and group associated with the at least one test; defining the first data content and the second data content in at least two locations comprising the plugin element and the actual usage of the plugin element on at least one unit of the at least one test, the at least one unit comprising form, section and group associated with the at least one test; and amalgamating at least one of the first and second presentation format and the first and second data content defined in the two locations by at least one test driver to deliver the at least one test to an examinee.
PCT/US2002/036220 2001-11-13 2002-11-13 Method and system for computer based testing using an amalgamated resource file WO2003043255A2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
AU2002360371A AU2002360371A1 (en) 2001-11-13 2002-11-13 Method and system for computer based testing using an amalgamated resource file

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US33122801P 2001-11-13 2001-11-13
US60/331,228 2001-11-13

Publications (3)

Publication Number Publication Date
WO2003043255A2 WO2003043255A2 (en) 2003-05-22
WO2003043255A9 true WO2003043255A9 (en) 2004-05-13
WO2003043255A3 WO2003043255A3 (en) 2004-11-25

Family

ID=23293104

Family Applications (5)

Application Number Title Priority Date Filing Date
PCT/US2002/036264 WO2003043157A1 (en) 2001-11-13 2002-11-13 Method and system for computer based testing using plugins to expand functionality of a test driver
PCT/US2002/036288 WO2003042786A2 (en) 2001-11-13 2002-11-13 Extensible exam language (xxl) protocol for computer based testing
PCT/US2002/036286 WO2003042956A1 (en) 2001-11-13 2002-11-13 Method and system for computer based testing using a non-deterministic exam extensible language (xxl) protocol
PCT/US2002/036220 WO2003043255A2 (en) 2001-11-13 2002-11-13 Method and system for computer based testing using an amalgamated resource file
PCT/US2002/036287 WO2003042785A2 (en) 2001-11-13 2002-11-13 Method and system for computer based testing using customizable templates

Family Applications Before (3)

Application Number Title Priority Date Filing Date
PCT/US2002/036264 WO2003043157A1 (en) 2001-11-13 2002-11-13 Method and system for computer based testing using plugins to expand functionality of a test driver
PCT/US2002/036288 WO2003042786A2 (en) 2001-11-13 2002-11-13 Extensible exam language (xxl) protocol for computer based testing
PCT/US2002/036286 WO2003042956A1 (en) 2001-11-13 2002-11-13 Method and system for computer based testing using a non-deterministic exam extensible language (xxl) protocol

Family Applications After (1)

Application Number Title Priority Date Filing Date
PCT/US2002/036287 WO2003042785A2 (en) 2001-11-13 2002-11-13 Method and system for computer based testing using customizable templates

Country Status (6)

Country Link
US (12) US7494340B2 (en)
EP (1) EP1444763A4 (en)
CN (1) CN100486068C (en)
AU (3) AU2002360371A1 (en)
CA (1) CA2466683C (en)
WO (5) WO2003043157A1 (en)

Families Citing this family (146)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020182579A1 (en) * 1997-03-27 2002-12-05 Driscoll Gary F. System and method for computer based creation of tests formatted to facilitate computer based testing
US7845950B2 (en) * 1997-03-27 2010-12-07 Educational Testing Service System and method for computer based creation of tests formatted to facilitate computer based testing
WO2003043157A1 (en) * 2001-11-13 2003-05-22 Prometric, A Division Of Thomson Learning, Inc. Method and system for computer based testing using plugins to expand functionality of a test driver
US20040076930A1 (en) * 2002-02-22 2004-04-22 Steinberg Linda S. Partal assessment design system for educational testing
US7496845B2 (en) 2002-03-15 2009-02-24 Microsoft Corporation Interactive presentation viewing system employing multi-media components
US8769517B2 (en) * 2002-03-15 2014-07-01 International Business Machines Corporation Generating a common symbol table for symbols of independent applications
US7127641B1 (en) * 2002-03-29 2006-10-24 Cypress Semiconductor Corp. System and method for software testing with extensible markup language and extensible stylesheet language
US20040001106A1 (en) * 2002-06-26 2004-01-01 John Deutscher System and process for creating an interactive presentation employing multi-media components
US7216340B1 (en) * 2002-08-19 2007-05-08 Sprint Communications Company L.P. Analysis data validation tool for use in enterprise architecture modeling with result based model updating
US7424702B1 (en) 2002-08-19 2008-09-09 Sprint Communications Company L.P. Data integration techniques for use in enterprise architecture modeling
US8554129B2 (en) * 2002-11-13 2013-10-08 Educational Testing Service Systems and methods for testing over a distributed network
AU2003282439A1 (en) * 2002-11-27 2004-06-18 Samsung Electronics Co., Ltd. Apparatus and method for reproducing interactive contents by controlling font according to aspect ratio conversion
CA2414047A1 (en) * 2002-12-09 2004-06-09 Corel Corporation System and method of extending scalable vector graphics capabilities
CA2414378A1 (en) * 2002-12-09 2004-06-09 Corel Corporation System and method for controlling user interface features of a web application
KR100526181B1 (en) * 2003-05-13 2005-11-03 삼성전자주식회사 Test-Stream Generating Method And Apparatus Providing Various Standards And Testing Level
CN100585662C (en) 2003-06-20 2010-01-27 汤姆森普罗梅特里克公司 System and method for computer based testing using cache and cacheable objects to expand functionality of a test driver application
US20050282133A1 (en) * 2004-06-18 2005-12-22 Christopher Crowhurst System and method for facilitating computer-based testing using traceable test items
WO2005008440A2 (en) * 2003-07-11 2005-01-27 Computer Associates Think, Inc. System and method for common storage object model
US8745222B2 (en) * 2003-08-15 2014-06-03 Blackboard Inc. Content system and associated methods
US7523447B1 (en) * 2003-09-24 2009-04-21 Avaya Inc. Configurator using markup language
US20050125196A1 (en) * 2003-12-09 2005-06-09 Len Swanson Method and system for computer-assisted test construction performing specification matching during test item selection
CA2651461A1 (en) * 2003-12-17 2005-06-17 Ibm Canada Limited - Ibm Canada Limitee Relationship management for data modeling in an integrated development environment
US8136094B2 (en) * 2004-01-07 2012-03-13 International Business Machines Corporation Relationship management for data modeling in an integrated development environment
US7506332B2 (en) * 2004-03-24 2009-03-17 Sap Ag Object set optimization using dependency information
US7793262B2 (en) * 2004-07-29 2010-09-07 International Business Machines Corporation Method and apparatus for facilitating software testing and report generation with interactive graphical user interface
EP1782180A4 (en) * 2004-08-02 2007-10-31 Justsystems Corp Document processing and management approach for creating a tag or an attribute in a markup language document, and method thereof
US7688723B1 (en) 2004-09-16 2010-03-30 Avaya Inc. Procedural XML-based telephony traffic flow analysis and configuration tool
US7830813B1 (en) 2004-09-30 2010-11-09 Avaya Inc. Traffic based availability analysis
US20060099563A1 (en) * 2004-11-05 2006-05-11 Zhenyu Lawrence Liu Computerized teaching, practice, and diagnosis system
US8117591B1 (en) 2005-01-07 2012-02-14 Interactive TKO, Inc. Graphical model for test case viewing, editing, and reporting
US8146057B1 (en) * 2005-01-07 2012-03-27 Interactive TKO, Inc. Instrumentation system and method for testing software
US20060160057A1 (en) * 2005-01-11 2006-07-20 Armagost Brian J Item management system
FR2882448B1 (en) * 2005-01-21 2007-05-04 Meiosys Soc Par Actions Simpli METHOD OF MANAGING, JOURNALIZING OR REJECTING THE PROGRESS OF AN APPLICATION PROCESS
WO2006096133A1 (en) * 2005-03-10 2006-09-14 Knowledge Director Pte. Ltd. System and method for generation of multimedia learning files
US8834173B2 (en) * 2005-04-08 2014-09-16 Act, Inc. Method and system for scripted testing
US8326659B2 (en) * 2005-04-12 2012-12-04 Blackboard Inc. Method and system for assessment within a multi-level organization
US7493520B2 (en) * 2005-06-07 2009-02-17 Microsoft Corporation System and method for validating the graphical output of an updated software module
US7543188B2 (en) * 2005-06-29 2009-06-02 Oracle International Corp. Browser based remote control of functional testing tool
US20070038894A1 (en) * 2005-08-09 2007-02-15 Microsoft Corporation Test Data verification with different granularity levels
US20070111182A1 (en) * 2005-10-26 2007-05-17 International Business Machines Corporation Method and system for distributing answers
US8083675B2 (en) * 2005-12-08 2011-12-27 Dakim, Inc. Method and system for providing adaptive rule based cognitive stimulation to a user
BRPI0707211A2 (en) * 2006-01-23 2011-04-26 Procter & Gamble laundry treatment compositions with thiazole dye
JP2008009715A (en) * 2006-06-29 2008-01-17 Hitachi Ltd Program distribution method and computer system
JP4902282B2 (en) * 2006-07-12 2012-03-21 株式会社日立製作所 Business system configuration change method, management computer, and business system configuration change program
US9390629B2 (en) 2006-09-11 2016-07-12 Houghton Mifflin Harcourt Publishing Company Systems and methods of data visualization in an online proctoring interface
US9892650B2 (en) 2006-09-11 2018-02-13 Houghton Mifflin Harcourt Publishing Company Recovery of polled data after an online test platform failure
US9230445B2 (en) 2006-09-11 2016-01-05 Houghton Mifflin Harcourt Publishing Company Systems and methods of a test taker virtual waiting room
US10861343B2 (en) 2006-09-11 2020-12-08 Houghton Mifflin Harcourt Publishing Company Polling for tracking online test taker status
US9142136B2 (en) 2006-09-11 2015-09-22 Houghton Mifflin Harcourt Publishing Company Systems and methods for a logging and printing function of an online proctoring interface
US20080124696A1 (en) * 2006-10-26 2008-05-29 Houser Ronald L Empirical development of learning content using educational measurement scales
GB2443446B (en) * 2006-10-30 2011-11-30 Hewlett Packard Development Co A method of identifying an extractable portion of a source machine-readable document
GB2443444A (en) * 2006-10-30 2008-05-07 Hewlett Packard Development Co Remotely editing a template document
GB2443447A (en) * 2006-10-30 2008-05-07 Hewlett Packard Development Co A method of constructing an output document by adding data from a variable data document to a template document
GB2443443A (en) * 2006-10-30 2008-05-07 Hewlett Packard Development Co method of defining editable portions within the template document
GB2443438A (en) * 2006-10-30 2008-05-07 Hewlett Packard Development Co Method of constructing and storing a document
US7831195B2 (en) * 2006-12-11 2010-11-09 Sharp Laboratories Of America, Inc. Integrated paper and computer-based testing administration system
US8239478B2 (en) * 2006-12-18 2012-08-07 Fourier Systems (1989) Ltd. Computer system
US20080229288A1 (en) * 2007-03-13 2008-09-18 Steve Nelson Software Plugin Modules for Device Testing
US20080293033A1 (en) * 2007-03-28 2008-11-27 Scicchitano Anthony R Identity management system, including multi-stage, multi-phase, multi-period and/or multi-episode procedure for identifying and/or authenticating test examination candidates and/or individuals
US8881105B2 (en) * 2007-04-11 2014-11-04 Patrick J. Quilter, Jr. Test case manager
US8156149B2 (en) * 2007-07-24 2012-04-10 Microsoft Corporation Composite nested streams
US8191045B2 (en) * 2007-09-04 2012-05-29 Nec Laboratories America, Inc. Mining library specifications using inductive learning
US8037163B1 (en) 2008-01-08 2011-10-11 Avaya Inc. Alternative methodology in assessing network availability
US8387015B2 (en) * 2008-01-31 2013-02-26 Microsoft Corporation Scalable automated empirical testing of media files on media players
WO2009137389A1 (en) 2008-05-05 2009-11-12 Eveready Battery Company, Inc. Razor blade and method of manufacture
US9111019B2 (en) 2008-09-30 2015-08-18 Interactive TKO, Inc. Modeling and testing interactions between components of a software system
KR20100041447A (en) * 2008-10-14 2010-04-22 삼성전자주식회사 Apparatus and method for automatic testing of softwares or digital divices
US10943030B2 (en) 2008-12-15 2021-03-09 Ibailbonding.Com Securable independent electronic document
US9081881B2 (en) 2008-12-18 2015-07-14 Hartford Fire Insurance Company Computer system and computer-implemented method for use in load testing of software applications
US9141513B2 (en) 2009-10-01 2015-09-22 Kryterion, Inc. Maintaining a secure computing device in a test taking environment
US9280907B2 (en) * 2009-10-01 2016-03-08 Kryterion, Inc. Proctored performance analysis
US8915744B2 (en) * 2009-11-06 2014-12-23 Tata Consultancy Services, Ltd. System and method for automated competency assessment
US9640085B2 (en) * 2010-03-02 2017-05-02 Tata Consultancy Services, Ltd. System and method for automated content generation for enhancing learning, creativity, insights, and assessments
US20120135388A1 (en) * 2010-03-14 2012-05-31 Kryterion, Inc. Online Proctoring
US10672286B2 (en) 2010-03-14 2020-06-02 Kryterion, Inc. Cloud based test environment
US9317572B2 (en) 2010-03-31 2016-04-19 Cloudera, Inc. Configuring a system to collect and aggregate datasets
US8874526B2 (en) 2010-03-31 2014-10-28 Cloudera, Inc. Dynamically processing an event using an extensible data model
US9082127B2 (en) 2010-03-31 2015-07-14 Cloudera, Inc. Collecting and aggregating datasets for analysis
US9081888B2 (en) 2010-03-31 2015-07-14 Cloudera, Inc. Collecting and aggregating log data with fault tolerance
EP2556451A4 (en) * 2010-04-06 2016-09-07 Webtrends Inc Method and system for defining and populating segments
US9243476B2 (en) * 2010-05-19 2016-01-26 Schlumberger Technology Corporation System and method for simulating oilfield operations
US8739150B2 (en) * 2010-05-28 2014-05-27 Smartshift Gmbh Systems and methods for dynamically replacing code objects via conditional pattern templates
US8855961B2 (en) 2010-06-01 2014-10-07 United States Of America As Represented By The Secretary Of The Navy Binary definition files
US8713130B2 (en) 2010-08-04 2014-04-29 Kryterion, Inc. Peered proctoring
US9137163B2 (en) 2010-08-04 2015-09-15 Kryterion, Inc. Optimized data stream upload
WO2012024352A2 (en) * 2010-08-16 2012-02-23 Extegrity Inc. Systems and methods for detecting substitution of high-value electronic documents
US9449405B2 (en) * 2010-11-30 2016-09-20 Sap Se Systems and methods to display dependencies within a graph of grouped elements
US8869112B2 (en) * 2010-11-30 2014-10-21 Sap Se System and method for modular business applications
US8429744B1 (en) * 2010-12-15 2013-04-23 Symantec Corporation Systems and methods for detecting malformed arguments in a function by hooking a generic object
US8554797B2 (en) 2010-12-17 2013-10-08 Sap Ag System and method for modular business applications
JP5978597B2 (en) * 2011-03-18 2016-08-24 株式会社リコー Information display device, question input device and system
US20120254722A1 (en) * 2011-03-31 2012-10-04 Cloudera, Inc. Interactive user interface implementation and development environment therefor
FR2976391A1 (en) * 2011-06-08 2012-12-14 Edunovo ELECTRONIC PERSONAL LEARNING DEVICE.
US8909127B2 (en) 2011-09-27 2014-12-09 Educational Testing Service Computer-implemented systems and methods for carrying out non-centralized assessments
US9128949B2 (en) 2012-01-18 2015-09-08 Cloudera, Inc. Memory allocation buffer for reduction of heap fragmentation
US9172608B2 (en) * 2012-02-07 2015-10-27 Cloudera, Inc. Centralized configuration and monitoring of a distributed computing cluster
WO2013126782A1 (en) * 2012-02-24 2013-08-29 National Assoc. Of Boards Of Pharmacy Outlier detection tool
US9405692B2 (en) 2012-03-21 2016-08-02 Cloudera, Inc. Data processing performance enhancement in a distributed file system
US9338008B1 (en) 2012-04-02 2016-05-10 Cloudera, Inc. System and method for secure release of secret information over a network
GB2511668A (en) * 2012-04-12 2014-09-10 Supercell Oy System and method for controlling technical processes
US9842126B2 (en) 2012-04-20 2017-12-12 Cloudera, Inc. Automatic repair of corrupt HBases
US9088463B1 (en) * 2012-05-11 2015-07-21 Amazon Technologies, Inc. Container contract for data dependencies
US9015234B2 (en) 2012-07-25 2015-04-21 Lg Cns Co., Ltd. Automated distributed testing administration environment
US9330164B1 (en) 2012-10-26 2016-05-03 Andrew Wills Edwards Electronic platform for user creation and organization of groups of member profiles to aid in memorization of selected information
KR20140056478A (en) * 2012-10-26 2014-05-12 삼성전자주식회사 Automatic testing apparatus for embedded software, automatic testing method thereof and test scenario composing method
US9116865B2 (en) * 2012-12-05 2015-08-25 Chegg, Inc. Enhancing automated terms listings in HTML document publishing based on user searches
US9595205B2 (en) * 2012-12-18 2017-03-14 Neuron Fuel, Inc. Systems and methods for goal-based programming instruction
US20140214709A1 (en) * 2013-01-07 2014-07-31 Assessment Innovation, Inc. Occupational performance assessment apparatuses, methods and systems
US9342557B2 (en) 2013-03-13 2016-05-17 Cloudera, Inc. Low latency query engine for Apache Hadoop
US9703679B2 (en) 2013-03-14 2017-07-11 International Business Machines Corporation Probationary software tests
US8954456B1 (en) 2013-03-29 2015-02-10 Measured Progress, Inc. Translation and transcription content conversion
WO2014174414A1 (en) * 2013-04-23 2014-10-30 University Of The Witwatersrand, Johannesburg A system for instrument quality assessment
US10102766B2 (en) * 2013-07-16 2018-10-16 Benesse Corporation Portable information processing apparatus, test support system and test support method
US9477731B2 (en) 2013-10-01 2016-10-25 Cloudera, Inc. Background format optimization for enhanced SQL-like queries in Hadoop
US9934382B2 (en) 2013-10-28 2018-04-03 Cloudera, Inc. Virtual machine image encryption
US9690671B2 (en) 2013-11-01 2017-06-27 Cloudera, Inc. Manifest-based snapshots in distributed computing environments
US10025839B2 (en) 2013-11-29 2018-07-17 Ca, Inc. Database virtualization
US20150161900A1 (en) 2013-12-06 2015-06-11 Act, Inc. Methods for improving test efficiency and accuracy in a computer adaptive test (cat)
US9940310B1 (en) * 2014-03-04 2018-04-10 Snapwiz Inc. Automatically converting an electronic publication into an online course
US9727314B2 (en) 2014-03-21 2017-08-08 Ca, Inc. Composite virtual services
US9531609B2 (en) 2014-03-23 2016-12-27 Ca, Inc. Virtual service automation
US9430216B2 (en) * 2014-05-11 2016-08-30 Lumension Security, Inc. Self-contained executable for predetermined software updating
EP3149662A4 (en) * 2014-05-30 2018-05-23 The Regents of The University of Michigan Brain-computer interface for facilitating direct selection of multiple-choice answers and the identification of state changes
US9747333B2 (en) 2014-10-08 2017-08-29 Cloudera, Inc. Querying operating system state on multiple machines declaratively
US9684876B2 (en) 2015-03-30 2017-06-20 International Business Machines Corporation Question answering system-based generation of distractors using machine learning
US20170004723A1 (en) * 2015-06-30 2017-01-05 Act, Inc. Identifying evidence of justification and explanation skills in computer automated scoring
WO2017015955A1 (en) * 2015-07-30 2017-02-02 Hewlett-Packard Development Company,L.P. Application testing
US10033678B2 (en) 2015-10-23 2018-07-24 Paypal, Inc. Security for emoji based commands
US10402049B1 (en) * 2015-12-29 2019-09-03 EMC IP Holding Company LLC User interface development
US11593342B2 (en) 2016-02-01 2023-02-28 Smartshift Technologies, Inc. Systems and methods for database orientation transformation
US10114736B2 (en) 2016-03-30 2018-10-30 Ca, Inc. Virtual service data set generation
US10585655B2 (en) 2016-05-25 2020-03-10 Smartshift Technologies, Inc. Systems and methods for automated retrofitting of customized code objects
US10089103B2 (en) 2016-08-03 2018-10-02 Smartshift Technologies, Inc. Systems and methods for transformation of reporting schema
EP3672478A4 (en) 2017-08-23 2021-05-19 Neurable Inc. Brain-computer interface with high-speed eye tracking features
JP6322757B1 (en) * 2017-09-29 2018-05-09 株式会社ドワンゴ Server and terminal
US10740132B2 (en) * 2018-01-30 2020-08-11 Veritas Technologies Llc Systems and methods for updating containers
US10740075B2 (en) 2018-02-06 2020-08-11 Smartshift Technologies, Inc. Systems and methods for code clustering analysis and transformation
US10698674B2 (en) 2018-02-06 2020-06-30 Smartshift Technologies, Inc. Systems and methods for entry point-based code analysis and transformation
US10528343B2 (en) 2018-02-06 2020-01-07 Smartshift Technologies, Inc. Systems and methods for code analysis heat map interfaces
US10664050B2 (en) 2018-09-21 2020-05-26 Neurable Inc. Human-computer interface using high-speed and accurate tracking of user interactions
US11561997B2 (en) * 2019-03-13 2023-01-24 Oracle International Corporation Methods, systems, and computer readable media for data translation using a representational state transfer (REST) application programming interface (API)
US11188707B1 (en) * 2020-05-08 2021-11-30 Bold Limited Systems and methods for creating enhanced documents for perfect automated parsing
CN112099928A (en) * 2020-08-28 2020-12-18 上海微亿智造科技有限公司 Recovery method, system and medium for accidental stop of Maxwell process
US11481312B2 (en) * 2020-10-15 2022-10-25 EMC IP Holding Company LLC Automation framework for monitoring and reporting on resource consumption and performance bottlenecks
US11574696B2 (en) * 2021-04-12 2023-02-07 Nanya Technology Corporation Semiconductor test system and method
CN113985252B (en) * 2021-10-28 2022-07-15 江苏博敏电子有限公司 Lamp panel jig regional testing method

Family Cites Families (41)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4216539A (en) * 1978-05-05 1980-08-05 Zehntel, Inc. In-circuit digital tester
US5318450A (en) 1989-11-22 1994-06-07 Gte California Incorporated Multimedia distribution system for instructional materials
US5157782A (en) * 1990-01-31 1992-10-20 Hewlett-Packard Company System and method for testing computer hardware and software
US5195033A (en) 1990-06-08 1993-03-16 Assessment Systems, Inc. Testing system including removable storage means for transfer of test related data and means for issuing a certification upon successful completion of the test
US6162060A (en) 1991-08-09 2000-12-19 Texas Instruments Incorporated System and method for the delivery, authoring, and management of courseware over a computer network
US5359546A (en) * 1992-06-30 1994-10-25 Sun Microsystems, Inc. Automatic generation of test drivers
DE69316631T2 (en) * 1992-08-31 1998-07-16 Dow Chemical Co SCREENPLAY BASED SYSTEM FOR TESTING A MULTI-USER COMPUTER SYSTEM
US5565316A (en) 1992-10-09 1996-10-15 Educational Testing Service System and method for computer based testing
CA2149660C (en) 1993-09-30 1996-12-03 Roger C. Kershaw A centralized system and method for administering computer based tests
US5629878A (en) 1993-10-07 1997-05-13 International Business Machines Corporation Test planning and execution models for generating non-redundant test modules for testing a computer system
US5524110A (en) * 1993-11-24 1996-06-04 Intel Corporation Conferencing over multiple transports
US5761684A (en) * 1995-05-30 1998-06-02 International Business Machines Corporation Method and reusable object for scheduling script execution in a compound document
US5743743A (en) 1996-09-03 1998-04-28 Ho; Chi Fai Learning method and system that restricts entertainment
US6138252A (en) 1996-07-01 2000-10-24 Sun Microsystems, Inc. Graphical test progress monitor
US6632248B1 (en) * 1996-12-06 2003-10-14 Microsoft Corporation Customization of network documents by accessing customization information on a server computer using uniquie user identifiers
US6029257A (en) 1996-12-06 2000-02-22 Intergraph Corporation Apparatus and method for testing computer systems
US5854930A (en) * 1996-12-30 1998-12-29 Mci Communications Corporations System, method, and computer program product for script processing
US6134674A (en) * 1997-02-28 2000-10-17 Sony Corporation Computer based test operating system
US6259890B1 (en) * 1997-03-27 2001-07-10 Educational Testing Service System and method for computer based test creation
US6142682A (en) 1997-06-13 2000-11-07 Telefonaktiebolaget Lm Ericsson Simulation of computer processor
US6018617A (en) * 1997-07-31 2000-01-25 Advantage Learning Systems, Inc. Test generating and formatting system
US6289472B1 (en) 1997-08-07 2001-09-11 Texas Instruments Incorporated Method and test system for testing under a plurality of test modes
US6112049A (en) 1997-10-21 2000-08-29 The Riverside Publishing Company Computer network based testing system
US6243835B1 (en) 1998-01-30 2001-06-05 Fujitsu Limited Test specification generation system and storage medium storing a test specification generation program
US6000945A (en) 1998-02-09 1999-12-14 Educational Testing Service System and method for computer based test assembly
US6149441A (en) 1998-11-06 2000-11-21 Technology For Connecticut, Inc. Computer-based educational system
US5987443A (en) * 1998-12-22 1999-11-16 Ac Properties B. V. System, method and article of manufacture for a goal based educational system
US6725399B1 (en) 1999-07-15 2004-04-20 Compuware Corporation Requirements based software testing method
US6431875B1 (en) * 1999-08-12 2002-08-13 Test And Evaluation Software Technologies Method for developing and administering tests over a network
US6934934B1 (en) * 1999-08-30 2005-08-23 Empirix Inc. Method and system for software object testing
US6681098B2 (en) 2000-01-11 2004-01-20 Performance Assessment Network, Inc. Test administration system using the internet
US6712615B2 (en) * 2000-05-22 2004-03-30 Rolf John Martin High-precision cognitive performance test battery suitable for internet and non-internet use
US6594466B1 (en) * 2000-05-24 2003-07-15 Bentley Systems, Incorporated Method and system for computer based training
US6505342B1 (en) * 2000-05-31 2003-01-07 Siemens Corporate Research, Inc. System and method for functional testing of distributed, component-based software
US7269631B2 (en) * 2000-06-02 2007-09-11 Hewlett-Packard Development Company, L.P. Architecture for parallel distributed table driven I/O mapping
US6632174B1 (en) * 2000-07-06 2003-10-14 Cognifit Ltd (Naiot) Method and apparatus for testing and training cognitive ability
US6996816B1 (en) * 2000-10-02 2006-02-07 Hewlett-Packard Development Company, L.P. Utilization of third party legacy data list
US6704741B1 (en) * 2000-11-02 2004-03-09 The Psychological Corporation Test item creation and manipulation system and method
US6694510B1 (en) * 2000-11-03 2004-02-17 Hewlett-Packard Development Company, L.P. Collection driver for collecting system data using record based requests with tag lists and pausing all but one thread of a computer system
US6898712B2 (en) * 2001-02-20 2005-05-24 Networks Associates Technology, Inc. Test driver ordering
WO2003043157A1 (en) 2001-11-13 2003-05-22 Prometric, A Division Of Thomson Learning, Inc. Method and system for computer based testing using plugins to expand functionality of a test driver

Also Published As

Publication number Publication date
AU2002360371A1 (en) 2003-05-26
US6948153B2 (en) 2005-09-20
US20090162829A1 (en) 2009-06-25
US9418565B2 (en) 2016-08-16
US20030138765A1 (en) 2003-07-24
WO2003043255A2 (en) 2003-05-22
US20060107254A1 (en) 2006-05-18
US20090155756A1 (en) 2009-06-18
WO2003042785A3 (en) 2003-10-02
WO2003042786A2 (en) 2003-05-22
WO2003043255A3 (en) 2004-11-25
CN1608338A (en) 2005-04-20
US7080303B2 (en) 2006-07-18
WO2003042786B1 (en) 2003-11-27
US7828551B2 (en) 2010-11-09
WO2003043157A1 (en) 2003-05-22
CA2466683A1 (en) 2003-05-22
US20090155758A1 (en) 2009-06-18
WO2003042956B1 (en) 2003-10-30
US6966048B2 (en) 2005-11-15
WO2003042785A2 (en) 2003-05-22
US20030182602A1 (en) 2003-09-25
EP1444763A4 (en) 2005-11-16
US20060069970A1 (en) 2006-03-30
US7318727B2 (en) 2008-01-15
US20030196170A1 (en) 2003-10-16
WO2003042956A1 (en) 2003-05-22
US20030203342A1 (en) 2003-10-30
AU2002363735A1 (en) 2003-05-26
US7784045B2 (en) 2010-08-24
CN100486068C (en) 2009-05-06
EP1444763A1 (en) 2004-08-11
US8454369B2 (en) 2013-06-04
US7494340B2 (en) 2009-02-24
US8579633B2 (en) 2013-11-12
WO2003042786A3 (en) 2003-10-23
US20030129573A1 (en) 2003-07-10
CA2466683C (en) 2016-01-12
US20110236873A1 (en) 2011-09-29
US8413131B2 (en) 2013-04-02
AU2002361616A1 (en) 2003-05-26
US20110145792A1 (en) 2011-06-16

Similar Documents

Publication Publication Date Title
CA2466683C (en) Method and system for computer based testing using plugins to expand functionality of a test driver
US8798520B2 (en) System and method for computer based testing using cache and cacheable objects to expand functionality of a test driver application
McIver The effect of programming language on error rates of novice programmers.
Kearsley Designing educational software for international use
Staub xLogo online-a web-based programming IDE for Logo
CA2530064C (en) System and method for computer based testing using cache and cacheable objects to expand functionality of a test driver application
Cabral et al. Automated formal specification generation and refinement from requirement documents
Kortenkamp et al. Geometry teaching in wireless classroom environments using Java and J2ME
Kim Structural design of efficient automated GUI testing/operations
Langtangen et al. Introduction to GUI Programming
Michael et al. Practical modeling & simulation for online courses: Proposal for A low level architecture
LaVerriere The Shogun MIDI Control System

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A2

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NO NZ OM PH PL PT RO RU SD SE SG SI SK SL TJ TM TN TR TT TZ UA UG US UZ VN YU ZA ZM ZW

AL Designated countries for regional patents

Kind code of ref document: A2

Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM AT BE BG CH CY CZ DE DK EE ES FI FR GB GR IE IT LU MC NL PT SE SK TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
COP Corrected version of pamphlet

Free format text: PAGES 40/50 AND 42/50, DRAWINGS, ADDED

122 Ep: pct application non-entry in european phase
DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
NENP Non-entry into the national phase

Ref country code: JP

WWW Wipo information: withdrawn in national office

Country of ref document: JP