|Publication number||USRE39435 E1|
|Application number||US 10/653,748|
|Publication date||Dec 19, 2006|
|Filing date||Sep 2, 2003|
|Priority date||Oct 17, 2000|
|Also published as||US6461166|
|Publication number||10653748, 653748, US RE39435 E1, US RE39435E1, US-E1-RE39435, USRE39435 E1, USRE39435E1|
|Inventors||Dennis Ray Berman|
|Original Assignee||Drb Lit Ltd.|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (78), Non-Patent Citations (4), Referenced by (8), Classifications (7), Legal Events (8)|
|External Links: USPTO, USPTO Assignment, Espacenet|
This invention relates to systems and methods for personnel training and, more particularly, to supervised or self-administered computer-based training systems that incorporate a learner-constructed response based testing methodology for improved evaluation of knowledge acquisition.
A variety of systems are available for automated learning and training using computers or other personal electronic devices. In current computer mediated learning and training systems, assessment of the “knowledge” gained by the user is carried out by, for example, true/false questions, matching (paired-associate) type questions, multiple choice questions, and marking questions. A multiple choice question differs from a marking question in that a multiple choice question has one correct answer, while a marking question has multiple correct answers. The foregoing question formats are not fully effective as learning aids, not are they reliable in assessing actual knowledge, for various reasons. For example, in a true/false question, a learner has a fifty-fifty chance of answering correctly by guessing; in a four way multiple choice question, the probability of a correct answer through guessing is twenty five percent. Test results thus are not necessarily indicative of actual knowledge.
What is needed, therefore, is a methodology for use in computer based training that provides for improved learning, improved efficiency, and improved reliability in the assessment of a user's actual knowledge of subject matter.
This invention provides a methodology in which a learner-constructed response is provided in answer to a question presented by the system, the response being evaluated by comparison with pre-defined expected responses and, based upon the evaluation, the system determining whether to proceed to another question or to offer remedial feedback. Such a learner-constructed response based evaluation methodology greatly reduces the potential for “guess-work” based correct responses and improves the training process through remedial feedback and advancement upon demonstration of knowledge.
Evaluation of responses involves identification of pre-defined keyword data pertaining to the subject matter being tested. Examples include passages of text with important keywords (keywords being defined herein to include one or more words, or phases, or related words and phases, or synonyms). Multiple choice questions may also include keywords, such that after the learner completes a sequence of reading material or any kind of current multiple-choice, mix or match, true false questions, the learner is prompted to enter answers to “fill-in-the-blank” or “verbal narrative” questions (a learner-constructed response). The learner entered responses are compared to standard solutions recorded on the system and remedial actions are provided.
The methodology may be used in a specially designed training system or in cooperation with existing computer based training systems. For every “choice” based question (e.g., multiple choice), for example, the methodology may prompt for a “user-constructed response” based upon a question that has associated with it all acceptable correct user-constructed responses to this question, the presentation to the learner being designed to include an area or mechanism for capturing a learner response either in the form of text or spoken words. The correct response is recognized if the response matches the keyword(s), e.g., primary/related keyword(s) or phrase(s) and/or synonym(s).
In one implementation, a computer program is provided for implementing a learning system with a learner-constructed response based methodology, the program including a presentation process for presenting at least one knowledge topic to the learner and for prompting the learner to enter a learner constructed response thereto; an evaluation information process for providing keyword data that corresponds to the knowledge topic; and an evaluation process for determining, based upon entry of a learner-constructed response to the knowledge topic, success or failure of the learner to know the knowledge topic, the success or failure being determined by comparison of the learner-constructed response with the keyword data.
Programming, as discussed in detail below for implementing the present learning methodology, is stored on disc input 26 and/or memory 16 and is executed by the system 10. The learning methodology preferably is practiced using the foregoing system components, although it may be practiced with alternative components.
The presentation information component 28 contains information for presenting the question, and may also include additional instructions, help information and an avenue for capturing learner-constructed responses (e.g., a text area or a record button for voice input). The evaluation information component 30 may include a sequence of phrases and, in one embodiment, these may take the form of standard HTML tags for the display of question information and a sequence of proprietary tags for the encoding of expected key-words or phrases under the “META-DATA” tag in HTML.
The program 34 enables creation of the components 42, 44 for a desired training session. During the creation of the training “content” the authors are prompted to create different key-words and phrases that best describe the “gist” of the content or embody the essence of the knowledge topic under discussion. These key-words and phrases are utilized for the construction of questions. These key-words may also be analyzed to produce additional key-words, phrases or synonyms, and identify negative constructs (wrong answers).
Referring to the process 300, in step 302 the learner is prompted to construct the target knowledge (presented previously, as described above) in his or her own words. One example of the prompt is the fill-in-the-blank format 48, above. In step 304, if the learner's response is verbal, the speech is converted into text data. After the learner's response has been fully entered, a comparison can be triggered automatically in a predetermined manner. For example, the learner can hit a particular key on the keyboard (e.g., an “Enter” key) or activate a particular area on the display screen to start the comparison. In step 306, the comparison is performed of the learner's response with the pre-defined key word data contained in the evaluation information component 30 (FIG. 2A). The comparison may involve a variety of analyses. For example, the comparison may:
(1) check for and correct spelling mistakes in the learner-constructed responses;
(2) determine whether the correct key word (words, phrases) appear in the learner-constructed response;
(3) determine whether synonyms of missing key word(s) appear in the learner-constructed response;
(4) determine whether related phrases that convey the same meaning as the expected key word(s) or phrases appear in the learner-constructed response;
(5) determine whether there are any incorrect key word(s) or phrases in the learner-constructed response or other negative constructs that might indicate a wrong answer.
A variety of logic selections for evaluation are contemplated. In one example, for purposes of improved learning and expediting the testing, a decision is made in step 308 of whether the learner response fails a lexical analysis (described more fully in FIG. 3B), thereby indicating a possible wrong answer or misunderstanding. If yes, then in step 310 the methodology prompts the user for a positive construct. If not, in step 312 a determination is made whether or not expected keyword(s) are found in the response, albeit not necessarily in the exact way or phraseology preferred. If yes, then the methodology proceeds to step 314 and provides a success message to the evaluation control program and execution returns to the program for testing of other target knowledge topics. If not, then in step 316 a determination is made whether expected related phrase(s) are found in the learner's response (thus indicating a correct or partially correct answer). If yes, execution proceeds to step 314. If not, in step 318 a determination is made whether expected synonym(s) appear in the learner response, thereby indicating a correct or partially correct answer. If yes, execution proceeds to step 314. If not, the methodology proceeds to step 320. In step 320, a “failure” message is sent to the evaluation control program 34.
Possible scenarios of a “failure” message to the evaluation control program 34 are that the evaluation control program may:
(1) Proceed to other questions and come back to the question upon which failure is indicated, until a satisfactory answer is received.
(2) Offer remedial questions or target information;
(3) Re-evaluate the learner with a focus on the missed part of the current topic.
Possible scenarios of a “success” message to the evaluation control program 34 are that the evaluation control program may:
(1) Discontinue further questioning on the target knowledge subject;
(2) Question the learner on the target knowledge again or in a different way to confirm understanding.
In step 324, if the response contains negative constructs, the learner is prompted in step 326 for alternative responses. For example, if the learner types “no empathy” or “not empathy” or “don't XXX” or “can't YYY” a parsing algorithm that looks for “empathy” or “XXX” or “YYY” will normally flag this as correct even though the negative construct makes the meaning totally different. Accordingly, step 324 determines that the answer with the negative construct is incorrect and proceeds to step 326.
If in step 324 there are no negative constructs, in step 328 a determination is made whether the user-constructed response contains a “conjunctive” construct and, if so, in step 330 prompts the learner for a single response. As an example, if “and” or “hut” or “or” are included in the answer, to indicate a possible guess or two possible answers, step 328 determines that the user-constructed responses is not correct and prompts the learner in step 330.
If in step 328 there are no conjunctive constructs, a determination in step 332 whether there are non-definite constructs, and if so, prompts the learner for a definite response. Example non-definite constructs include, e.g., “maybe” or “perhaps.”
If in step 332 there are no non-definite constructs, in step 336 execution proceeds to the next phase of the analysis, as further described in step 312 of
It is noted that at any given moment during the execution of the above mentioned learning methodology, various information pertaining to the training session or the performance of the learner is collected by the system 10 for different purposes. In one specific case, at the end of a training session, the collected information gives an in-depth view of how well the learner has been trained. The collected information can be analyzed to generate various reports to be delivered to a predetermined interested party. For instance, the analyzed information will help to identify comparative difficulties of different materials or subjects covered in the training session, or provide information on how the learner has performed on a per question basis, etc. A statistical analysis and report can also be generated in a similar fashion based on the performances of a group of learners with regard to the training session. Therefore, the interested party can evaluate the performance of a group of learners to make various decisions such as to determine whether the training session should be revised, or whether the group of learners can be profiled in a certain manner.
In summary, the system 10 provides a learning methodology that improves the speed and retention of learning, and furthermore provides improved accuracy in assessment of the learner. By requiring, perhaps in addition to traditional multiple choice or other testing techniques, a learner-constructed response in which the learner must use his or her own words in answering a question, greater assurance is provided that the learner indeed knows the subject matter. Also, the system allows for refinement of the testing as the learner gets closer to accurate responses, as enabled by the construction of a key word component associated with the target knowledge component, as enabled by the evaluation process.
Although illustrative embodiments of the invention have been shown and described, other modifications, changes, and substitutions are intended in the foregoing disclosure. Accordingly, it is appropriate that the appended claims be constructed broadly and in a manner consistent with the scope of the invention.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US3408749||Apr 11, 1967||Nov 5, 1968||American Can Co||Branching-instruction teaching device|
|US3566482 *||Oct 24, 1968||Mar 2, 1971||Data Plex Systems||Educational device|
|US3606688 *||Jul 11, 1969||Sep 21, 1971||Associated Research Lab Propri||Method and apparatus for teaching a multiplicity of students|
|US3671668 *||Nov 18, 1968||Jun 20, 1972||Reiffel Leonard||Teaching system employing a television receiver|
|US3715811 *||Oct 13, 1970||Feb 13, 1973||Westinghouse Electric Corp||Remedial branching teaching system|
|US4289313 *||Sep 7, 1979||Sep 15, 1981||Delamontagne Robert P||Management teaching game apparatus and method|
|US4416454 *||Jul 2, 1981||Nov 22, 1983||Delamontagne Robert P||Management teaching game method|
|US4817036||Mar 15, 1985||Mar 28, 1989||Brigham Young University||Computer system and method for data base indexing and information retrieval|
|US4833610||Mar 7, 1988||May 23, 1989||International Business Machines Corporation||Morphological/phonetic method for ranking word similarities|
|US4895518 *||Nov 2, 1987||Jan 23, 1990||The University Of Michigan||Computerized diagnostic reasoning evaluation system|
|US4958284 *||Dec 6, 1988||Sep 18, 1990||Npd Group, Inc.||Open ended question analysis system and method|
|US5002491||Apr 28, 1989||Mar 26, 1991||Comtek||Electronic classroom system enabling interactive self-paced learning|
|US5002865 *||Feb 26, 1990||Mar 26, 1991||Konica Corporation||Silver halide photographic material|
|US5011413 *||Jul 19, 1989||Apr 30, 1991||Educational Testing Service||Machine-interpretable figural response testing|
|US5033969 *||Mar 13, 1990||Jul 23, 1991||Pioneer Electronic Corporation||Support device for resolving questions about reproduced information|
|US5112064 *||Dec 19, 1990||May 12, 1992||Weedman Gail H||Psychology game|
|US5168565||Jan 13, 1989||Dec 1, 1992||Ricoh Company, Ltd.||Document retrieval system|
|US5246375 *||Sep 23, 1991||Sep 21, 1993||Wouter Goede||Memory aiding device|
|US5265065||Oct 8, 1991||Nov 23, 1993||West Publishing Company||Method and apparatus for information retrieval from a database by replacing domain specific stemmed phases in a natural language to create a search query|
|US5307266||Aug 7, 1991||Apr 26, 1994||Hitachi, Ltd.||Information processing system and method for processing document by using structured keywords|
|US5314340 *||Oct 30, 1990||May 24, 1994||Texas Instruments Incorporated||Electronic teaching apparatus having two-part partially and wholly actuated for indication of correct and incorrect answers|
|US5325465||Mar 4, 1992||Jun 28, 1994||Singapore Computer Systems Limited||End user query facility|
|US5384703||Jul 2, 1993||Jan 24, 1995||Xerox Corporation||Method and apparatus for summarizing documents according to theme|
|US5424947||Jun 12, 1991||Jun 13, 1995||International Business Machines Corporation||Natural language analyzing apparatus and method, and construction of a knowledge base for natural language analysis|
|US5441415 *||Dec 6, 1993||Aug 15, 1995||John R. Lee||Interactive computer aided natural learning method and apparatus|
|US5442780||Jul 8, 1992||Aug 15, 1995||Mitsubishi Denki Kabushiki Kaisha||Natural language database retrieval system using virtual tables to convert parsed input phrases into retrieval keys|
|US5463773||May 25, 1993||Oct 31, 1995||Fujitsu Limited||Building of a document classification tree by recursive optimization of keyword selection function|
|US5475588||Jun 18, 1993||Dec 12, 1995||Mitsubishi Electric Research Laboratories, Inc.||System for decreasing the time required to parse a sentence|
|US5511793 *||Jun 8, 1992||Apr 30, 1996||Quantum Development, Inc.||Composite chess game and method|
|US5519608||Jun 24, 1993||May 21, 1996||Xerox Corporation||Method for extracting from a text corpus answers to questions stated in natural language by using linguistic analysis and hypothesis generation|
|US5528491||Aug 31, 1992||Jun 18, 1996||Language Engineering Corporation||Apparatus and method for automated natural language translation|
|US5540589 *||Apr 11, 1994||Jul 30, 1996||Mitsubishi Electric Information Technology Center||Audio interactive tutor|
|US5597312||May 4, 1994||Jan 28, 1997||U S West Technologies, Inc.||Intelligent tutoring method and system|
|US5632624 *||Sep 22, 1993||May 27, 1997||Brainchild, Inc.||Electronic study guide|
|US5689716||Apr 14, 1995||Nov 18, 1997||Xerox Corporation||Automatic method of generating thematic summaries|
|US5694523||May 31, 1995||Dec 2, 1997||Oracle Corporation||Content processing system for discourse|
|US5696962||May 8, 1996||Dec 9, 1997||Xerox Corporation||Method for computerized information retrieval using shallow linguistic analysis|
|US5708822||May 31, 1995||Jan 13, 1998||Oracle Corporation||Methods and apparatus for thematic parsing of discourse|
|US5749736 *||May 31, 1996||May 12, 1998||Taras Development||Method and system for computerized learning, response, and evaluation|
|US5768580||May 31, 1995||Jun 16, 1998||Oracle Corporation||Methods and apparatus for dynamic classification of discourse|
|US5863208 *||Jul 2, 1996||Jan 26, 1999||Ho; Chi Fai||Learning system and method based on review|
|US5885087 *||Mar 3, 1997||Mar 23, 1999||Robolaw Corporation||Method and apparatus for improving performance on multiple-choice exams|
|US5987302 *||Mar 20, 1998||Nov 16, 1999||Educational Testing Service||On-line essay evaluation system|
|US5987443||Dec 22, 1998||Nov 16, 1999||Ac Properties B. V.||System, method and article of manufacture for a goal based educational system|
|US6029043||Jan 29, 1998||Feb 22, 2000||Ho; Chi Fai||Computer-aided group-learning methods and systems|
|US6067538||Dec 22, 1998||May 23, 2000||Ac Properties B.V.||System, method and article of manufacture for a simulation enabled focused feedback tutorial system|
|US6077085 *||May 19, 1998||Jun 20, 2000||Intellectual Reserve, Inc.||Technology assisted learning|
|US6086382 *||Feb 22, 1999||Jul 11, 2000||Robolaw Corporation||Method and apparatus for improving performance on multiple-choice exams|
|US6115683 *||Mar 31, 1997||Sep 5, 2000||Educational Testing Service||Automatic essay scoring system using content-based techniques|
|US6120297 *||Aug 25, 1998||Sep 19, 2000||Lyceum Communication, Inc.||Vocabulary acquistion using structured inductive reasoning|
|US6160987||Dec 29, 1999||Dec 12, 2000||Ho; Chi Fai||Computer-aided group-learning methods and systems|
|US6164974||Mar 27, 1998||Dec 26, 2000||Softlight Inc.||Evaluation based learning system|
|US6168440 *||Aug 28, 1998||Jan 2, 2001||National Computer Systems, Inc.||Multiple test item scoring system and method|
|US6173251||Jul 28, 1998||Jan 9, 2001||Mitsubishi Denki Kabushiki Kaisha||Keyword extraction apparatus, keyword extraction method, and computer readable recording medium storing keyword extraction program|
|US6181909 *||Jul 22, 1998||Jan 30, 2001||Educational Testing Service||System and method for computer-based automatic essay scoring|
|US6199034||Apr 14, 1998||Mar 6, 2001||Oracle Corporation||Methods and apparatus for determining theme for discourse|
|US6208832||Nov 14, 1997||Mar 27, 2001||Sony Corporation||Learning system with response analyzer|
|US6226611 *||Jan 26, 2000||May 1, 2001||Sri International||Method and system for automatic text-independent grading of pronunciation for language instruction|
|US6254395 *||Apr 12, 1999||Jul 3, 2001||Educational Testing Service||System and method for automated testing of writing skill|
|US6256399 *||Sep 27, 1999||Jul 3, 2001||Ncs Pearson, Inc.||Method of distribution of digitized materials and control of scoring for open-ended assessments|
|US6267601 *||Dec 5, 1997||Jul 31, 2001||The Psychological Corporation||Computerized system and method for teaching and assessing the holistic scoring of open-ended questions|
|US6282404 *||Sep 22, 1999||Aug 28, 2001||Chet D. Linton||Method and system for accessing multimedia data in an interactive format having reporting capabilities|
|US6287123 *||Mar 29, 1999||Sep 11, 2001||O'brien Denis Richard||Computer managed learning system and data processing method therefore|
|US6292792 *||Mar 26, 1999||Sep 18, 2001||Intelligent Learning Systems, Inc.||System and method for dynamic knowledge generation and distribution|
|US6295439 *||Mar 21, 1997||Sep 25, 2001||Educational Testing Service||Methods and systems for presentation and evaluation of constructed responses assessed by human evaluators|
|US6302698 *||Feb 16, 1999||Oct 16, 2001||Discourse Technologies, Inc.||Method and apparatus for on-line teaching and learning|
|US6311040 *||Jul 31, 1997||Oct 30, 2001||The Psychological Corporation||System and method for scoring test answer sheets having open-ended questions|
|US6343935 *||Mar 1, 2000||Feb 5, 2002||Castle Hill Learning Company, Llc||Computerized interactive educational method and apparatus for teaching vocabulary|
|US6345270||Mar 19, 1998||Feb 5, 2002||Fujitsu Limited||Data management system|
|US6356864 *||Jul 23, 1998||Mar 12, 2002||University Technology Corporation||Methods for analysis and evaluation of the semantic content of a writing based on vector length|
|US6411924||Jan 23, 1998||Jun 25, 2002||Novell, Inc.||System and method for linguistic filter and interactive display|
|US6461166 *||Oct 17, 2000||Oct 8, 2002||Dennis Ray Berman||Learning system with learner-constructed response based testing methodology|
|US6470170||May 16, 2001||Oct 22, 2002||Hai Xing Chen||System and method for interactive distance learning and examination training|
|US6493690||Feb 10, 2000||Dec 10, 2002||Accenture||Goal based educational system with personalized coaching|
|US6548470||Dec 9, 1999||Apr 15, 2003||The Procter & Gamble Company||Bleaching compositions|
|US6553382||Mar 14, 1996||Apr 22, 2003||Canon Kabushiki Kaisha||Data management system for retrieving data based on hierarchized keywords associated with keyword names|
|US6554618||Apr 20, 2001||Apr 29, 2003||Cheryl B. Lockwood||Managed integrated teaching providing individualized instruction|
|WO1997018698A2 *||Nov 8, 1996||May 29, 1997||Peter Geoffrey Mcdermott||Memory aid|
|1||*||Crowder, Norman A., Arithmetic of Computers, An Introduction to Binary and Octal Mathematics. A Tutor Text, 1958, pp. i-iv and 1-18, Doubleday & Company, Garden City, NY.|
|2||FullRecall. Retrieved from the Internet on Apr. 25, 2005: <URL: http://www.fullrecall.com>.|
|3||Spaced repetition-Wikipedia, Retrieved from the Internet on Apr. 26, 2005: <URL: http://en.wikipedia.org/wiki/Spaced_repetition>.|
|4||Super Memory. Retrieved from the Internet on Apr. 25, 2005 <URL: http://www.supermemo.com>.|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US8506305||Dec 23, 2008||Aug 13, 2013||Deck Chair Learning Systems Inc.||Electronic learning system|
|US8606170||Jan 20, 2012||Dec 10, 2013||Northrop Grumman Systems Corporation||Method and apparatus for interactive, computer-based, automatically adaptable learning|
|US8851900||Jul 24, 2013||Oct 7, 2014||Deck Chair Learning Systems Inc.||Electronic learning system|
|US20040234938 *||Dec 23, 2003||Nov 25, 2004||Microsoft Corporation||System and method for providing instructional feedback to a user|
|US20060141438 *||Dec 23, 2004||Jun 29, 2006||Inventec Corporation||Remote instruction system and method|
|US20070065797 *||Sep 20, 2005||Mar 22, 2007||Ross Elgart||System and method of preparing for essay examinations|
|US20090202969 *||Jan 8, 2009||Aug 13, 2009||Beauchamp Scott E||Customized learning and assessment of student based on psychometric models|
|WO2009089475A1 *||Jan 9, 2009||Jul 16, 2009||Beauchamp Scott E||Customized learning and assessment of student based on psychometric models|
|U.S. Classification||434/323, 434/322, 434/353|
|International Classification||G09B7/04, G09B7/00|
|Sep 2, 2003||AS||Assignment|
Owner name: DRB LIT LTD., TEXAS
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:BERMAN, DENNIS RAY;REEL/FRAME:014480/0687
Effective date: 20030829
|Mar 20, 2007||CC||Certificate of correction|
|Apr 19, 2007||AS||Assignment|
Owner name: BERMAN, DENNIS R, TEXAS
Free format text: SECURITY AGREEMENT;ASSIGNOR:DRB LIT LTD.;REEL/FRAME:019181/0279
Effective date: 20070101
|Mar 22, 2010||FPAY||Fee payment|
Year of fee payment: 8
|Dec 13, 2011||AS||Assignment|
Effective date: 20111201
Free format text: PATENT SECURITY AGREEMENT;ASSIGNOR:DRB LIT LTD.;REEL/FRAME:027371/0283
Owner name: BERMAN, DENNIS R., MR., TEXAS
|Mar 6, 2014||AS||Assignment|
Effective date: 20140131
Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:BERMAN, DENNIS R.;REEL/FRAME:032370/0618
Owner name: DRB LIT LTD., TEXAS
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:TRIVAC LTD.;DRB LIT LTD.;REEL/FRAME:032371/0018
Owner name: LOCKIN, LLC, TEXAS
Effective date: 20140131
|Apr 8, 2014||FPAY||Fee payment|
Year of fee payment: 12
|Jul 22, 2014||AS||Assignment|
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:LOCKIN, LLC;REEL/FRAME:033367/0085
Owner name: MEMORY SCIENCE, LLC, TEXAS
Effective date: 20140721