For a computerized instructional system to be considered "intelligent," it needs to know what, when, and how to teach the subject matter. That is, the tutoring system must have expertise in the domain being instructed as well as the capability to assess how learners are progressing in their knowledge/skill acquisition. This paper focuses on the "what" of tutoring systems (i.e., the domain expertise) Towards this end, we outline a novel approach to cognitive task analysis (CTA) which presumes to automate the knowledge-elicitation process. Heretofore, task analyses have consumed large amounts of time given all that's involved in the various procedures, some of which include: (a) extensive reading, summarizing, and hierarchically structuring of knowledge embedded within domain-specific documents, (b) interviewing experts, and transcribing and abstracting their protocols/answers, and (c) observing experts during the performance of various domain-specific tasks and reducing their actions into low-level rules and skills. Shute and Torreano (1995) proposed an automated cognitive task analysis procedure called DNA (Decompose, Network, Assess) designed to elicit domain specifications for curricula which encompass expert knowledge and skills. This procedure is intended to be broader and more efficient compared to existing CTA methodologies. The current paper reviews background literature in the domain of CTA and discusses the specific DNA procedure.
One major activity that continues to hamper efforts to efficiently develop intelligent instructional software is the derivation of knowledge and skill components to be instructed and assessed (i.e., extracting and organizing domain expertise). This is often referred to as the "bottleneck" in the development process (e.g., Gordon, Schmierer, and Gill, 1993; Durkin, 1994; Hayes-Roth, Waterman, and Lenat, 1983), although technically it is called cognitive task analysis (CTA). In general, CTA represents a collection of approaches united by a common goal-to represent the underlying knowledge, skills, and structures of a particular task performance.
There are typically two phases to CTA: (1) Initial acquisition of knowledge and skills, and (2) Compilation of information into a useful database that can serve as an expert model or curriculum. Regardless of the approach for eliciting and/or structuring knowledge, current CTA procedures are considerably more art than science. Furthermore, many tend to be extremely labor-intensive, unstandardized, potentially incomplete, and difficult to translate into instruction and assessment. Following are listings of some of the typical activities underlying each of the two phases of CTA.
Phase 1-Acquisition. A few of the more commonly-used procedures for extracting knowledge include: document analysis, interviews with experts, observing experts as they perform some task, protocol analysis, retrospective probe questioning, and critical incident methods. Benysh, Koubek, & Calvez (1993) classify the plethora of knowledge elicitation procedures into the following four categories: (a) Verbal reports (e.g., interviewing experts-structured and unstructured, questionnaires, interruption analysis, protocol analysis); (b) Cognitive modeling (e.g., GOMS, task action grammar, production rules); (c) Clustering (e.g., card sorting, ordered recall, closed-curve analysis, spatial reconstruction); and (d) Scaling (e.g., similarity ratings, repertory grid, co-occurrence analysis, proximity in recall). The last two may actually be collapsed into one category, for an even simpler framework.
Phase 2-Organization. How is extracted information optimally arrayed? Conceptual graphs are currently the most popular means of representing hierarchically-structured knowledge. As the name implies, conceptual graphs are the graphical representation of concepts showing, at various grain sizes, relevant concepts (nodes) and their interrelationships (arcs). Conceptual graphs are finite, connected, and typically bipartite (i.e., there are two different types of nodes-concepts and their relationships). These graphs can address declarative or procedural knowledge. And the possibilities for denoting relationships among concepts are innumerable.
In an attempt to somewhat constrain these possibilities, DuBois & Shalin (1995) suggest annotating conceptual graphs with the following information: (a) Concepts and principles explaining why the method works; (b) Procedure-selection knowledge; (c) Pattern-recognition knowledge of contexts required to select or execute task procedures; (d) Procedure-execution knowledge such as steps or subgoals; and (e) Goal-attainment knowledge including priority, sequencing, and standards.
There are many possible ways to elicit, then structure knowledge and skills from a domain expert. One thing to note is that these two phases are almost always performed at different points in time. What would happen if we attempted to perform them concurrently (i.e., extract the knowledge and arrange it during the same session with the expert)?
This paper describes a new methodology embodied in a prototype computer program called DNA (Decompose, Network, Assess; Shute & Torreano, 1995). The goals of DNA are to create an easy-to-use cognitive task analysis procedure that is capable of eliciting comprehensive and valid curricula from experts (and others) across a variety of domains. The method is implemented as a friendly, interactive computer program that performs both the acquisition and the organization phases of CTA in parallel. The product is a hierarchically-organized set of curriculum elements (CEs) that feeds directly into instruction and assessment.
We begin with a brief literature review of cognitive task analysis approaches, then describe our procedure (DNA), providing a sample output (CE records and conceptual graph) from a specific domain. We illustrate how easily the output can prescribe different types of assessment, and conclude with implications about the future of intelligent instructional software.
What Is CTA?
The literature can be quite confusing when viewing the multifarious definitions of CTA. For instance, Olson & Biolsi (1991) broadly contend that CTA is the aggregation of techniques, analyses, and output representations of knowledge and skill. In their view, CTA is substantially more than just a listing of cognitive operations required for successful performance of some task. Dehoney (1995) more circumspectly suggests that CTA models and techniques relate to training, specifically in terms of advancing people to the level of expert performance on complex problem-solving tasks.
Still another way to define CTA is in contrast with behavioral task analysis (BTA). The main focus of BTA is on procedural knowledge while the primary focus of CTA is on the content and structure of declarative knowledge (see Hall, Gott, & Pokorny, 1995; Redding, 1989). CTA also differs from behavioral task analysis in several other ways. First, CTA aids in the understanding of how skills and knowledge are obtained and how the learning process can be improved. Second, CTA typically analyzes the knowledge and skills needed to perform the entire task and examines interrelations among these knowledge and skill components. Finally, CTA emphasizes skill development. That is, attention is paid to how novices become experts by analyzing a range of competency levels (Willis, 1996). We now present our views on CTA, culled from the chaotic literature.
The Main Goal of CTA. As mentioned earlier, we argue that the primary objective of CTA is to acquire a rich body of knowledge about a domain from experts and compile that into a useful model that can serve as the basis or curriculum for something like an intelligent instructional system (Durkin, 1994). Anderson (1988) referred to the expert model as the "backbone" of these systems (p. 21). Unfortunately, it is often the case that the expert model within an intelligent tutoring system is incomplete and thus limited to instructing only a portion of the domain knowledge (Anderson, 1988). Without a rich knowledge base, the system will not provide adequate instruction. Thus, it is important to develop methods which are capable of effectively capturing a complete array of knowledge and skills. This position then relies on specifying both cognitive and behavioral elements.
Conjoining CTA and BTA. CTA can be thought of as the "overt observable behavior and the covert cognitive functions behind it that form an integrated whole." (S. Chipman, Feb 24, 1997, personal communication). Artificially separating BTA and CTA is likely to produce information that is not very useful in understanding, aiding, or training job performance. Essens, Fallesen, McCann, Cannon-Bowers, & Dörfel (1994) similarly argue that CTA should be performed in the context of prior, more traditional behavioral task analyses. Their definition of CTA involves: (a) the knowledge required for the task and the relationships among important concepts (semantic nets); (b) the mental operations for retrieval, storage, transformation, integration, and modeling of information (GOMS); (c) the metacognitive processes that control cognitive effort and attention; and (d) cognitive skill development and progression of knowledge structures form novice to expert.
Representing Knowledge-Level of Expertise. Some researchers limit their definition of CTA to expert performance. For example, Gordon & Gill (1997) define CTA as a "theory or model of how experts in a field perform their tasks" (p. 132). But we are more comfortable with a broader definition that encompasses performance and knowledge representations of experts and novices (as well as intermediate performance levels). According to Kirwan & Ainsworth (1992), "Cognitive task analysis covers a range of approaches used for looking at mental (hence cognitive) internal events or knowledge structures" (p. 392). Cognitive task analysis is appropriate for tasks in which the critical steps are cognitive operations (e.g., hypothesis formation, judgments, problem solving) rather than observable behaviors (Means, 1993, p. 101).
Redding (1992) summarizes this novice-to-expert knowledge capturing well, "Cognitive task analysis determines the mental processes and skills required to perform a task at high proficiency levels and the changes that occur as the skills develop. Cognitive task analysis helps determine how a task is learned, how learning can be facilitated, and what cognitive processes underlie and support effective job performance. The information is then used in the design phase to determine the most effective manner of training employees to achieve high levels of proficiency in job performance." (p. 3).
Outcome Types
There is general agreement in the literature that knowledge is optimally characterized by a declarative/procedural distinction (e.g., Anderson, 1993; Hall, et al., 1995; Shute, 1995), regardless of whether it is from an expert or novice, correct or incorrect, and so on. In the context of discussing a new student modeling paradigm (i.e. SMART for Student Modeling Approach for Responsive Tutoring), Shute (1995) takes the declarative/procedural framework and breaks it out into three outcome types. The most fundamental form of knowledge is termed "symbolic knowledge" (SK) which includes symbols and definitions of terms (e.g., knowing that the symbol "" refers to "the sum of"). Procedural skill (PS) relates to the application of some procedure, such as the ability to calculate the arithmetic mean from a set of data using the formula "(X)/N". PS typically requires some prerequisite SK (e.g., one must possess the knowledge of the formula (X)/N before actually calculating it). Conceptual knowledge (CK) refers to organized sets of concepts, rules, and their linkages. CK is the ability to construct higher-level relationships among concepts, like knowing that the formula to calculate the mean refers to a measure of central tendency which should be used to summarize quantitative data, and has a specific relationship with its underlying distribution (reflecting the "balance point" of the data). Thus, CK is much more complex than SK or PS, and in many cases, requires prerequisite knowledge of SK and PS.
Each of the three outcome types in SMART has its own optimal form of instruction and/or remediation. For example, SK is facilitated via drill (i.e., repeating something until memorized); PS by providing the learner with practice opportunities for solving carefully-crafted problems; and CK via the presentation of functionally-mapped analogies (e.g., using an on-line seesaw as an analogy for the arithmetic Mean). Moreover, these different outcome types should be elicited using different techniques (see Cooke, 1994). DNA seeks to explain the "what, when/how, and why" underlying any subject matter, mapping onto SMART's three outcome types (i.e., SK, PS, and CK).
Experts often find it difficult to verbalize knowledge responsible for perceptual, skill-based processing (Gordon, Schmierer, & Gill, 1993). Knowledge that experts use but are not aware of is referred to as implicit knowledge. Collins (1985) reported that experts may not report parts of their knowledge because they are unaware they have the knowledge, not because they cannot verbalize it. It is important to elicit both explicit knowledge and implicit forms of knowledge and skill when developing a complete curriculum.
After extracting knowledge and skills and arranging them into a learning hierarchy, how can we insure that the information is correct and stable? These issues are now addressed.
Validity & Reliability
Gordon & Gill (1997) claim that two (often overlooked, but equally) important parts of CTA include issues of validity and reliability. Validity asks the questions: (a) Is the domain expertise adequately captured by the procedure? and (b) Are the data psychologically meaningful and practically useful? Issues of validity pertain to models of performance from multiple expert inputs, which can be used to generate testable predictions within the domain. One can also validate the output of the CTA via the success of the artifact it has supported, although this may be confounded by the efficacy of the instructional system.
In addition to using multiple experts, Olson & Biolsi (1991) state that it is imperative to obtain convergent evidence from multiple methods, perhaps even coming up with an orchestration of different approaches. For example, one can use interviews to identify important concepts, indirect methods to explore relations among concepts, and integrate think-aloud protocols into indirect data collection techniques to simplify the interpretation of results. In DNA, we collect data from several different experts (as well as novices, if necessary). We also employ a variety of techniques during the extraction of the three outcome types. These will be discussed later.
Reliability addresses the issue of whether different analysts using identical methods end up with the same results or not. We tend to agree with the assessment by Gordon & Gill (1997) that there is a shortage of relevant documentation of CTA methods/procedures in the literature, thus any one procedure becomes most difficult to replicate. Again, our goal with DNA is to transform the CTA process from an art into more of a science.
CTA Cost-Benefits
Most current CTA methodologies require a great deal of time to complete. For example, Gordon, Babbitt, Sorensen, Bell, and Crane (1993) report that their conceptual graph analyses can easily require a total of 360 hours to perform. They break down their analysis time as follows: (a) 80 hours conducting and organizing information obtained during document analysis, (b) 160 hours conducting and organizing information obtained during interviews with nine F-16 expert pilots, and (c) 120 hours collecting and organizing observation data. For a 40-hour week, this translates into approximately nine months of time to achieve just the CTA.
Similarly, Crandell, Klein, Militello, & Wolf (1994) assert that conducting the necessary interviews related to critical decision methods requires at least a six-month apprenticeship. And cognitive-modeling methods like COGNET and GOMS are even more time-consuming. We believe that DNA can cut the data extracting and arranging time requirement by maybe half, and still end up with the same (if not better) domain expertise. This, of course, is an empirical question that we're preparing to test, fall 1997.
What Constitutes An Effective CTA Method/Procedure?
CTA methods must meet certain requirements in order to be considered effective. First, a CTA method must be domain-independent. Some existing CTA methods are only applicable to certain tasks or specific domains. Second, a CTA method must effectively capture all types of knowledge (i.e., implicit and explicit forms of SK, PS, CK). Third, a CTA method must be able to illustrate relationships among knowledge components (i.e., hierarchically and conceptually). Fourth, a CTA method must be easy to use. Redding (1989) mentions that most people who design curricula (e.g., teachers) do not have training in areas such as statistics and cognitive science, thus those who need it are not skilled to use most current CTA methods. Finally, a CTA method must be efficient. That is, CTA should not require hundreds of hours to complete.
If performed in a careful and principled manner, automating the CTA procedure could certainly expedite the CTA process without sacrificing accuracy for speed. This has been the motivation underlying the original design of DNA (Shute & Torreano, 1995) and our current DNA design.
DNA is being designed to elicit knowledge from subject-matter experts (SMEs) which will Decompose a domain, Network the knowledge into comprehensive structures, and employ other experts in a given domain to Assess the validity, completeness, and reliability of the knowledge structures. Ideally, we want this to be an easy procedure capable of extracting and organizing a full set of knowledge and skills from experts, regardless of domain.
We currently have developed a prototype of DNA which consists of a series of five interactive modules that automate knowledge acquisition and are based on theoretical and empirical support. It is designed to be a running dialog between the computer and the SME.
Five Modules of DNA
Customize. The first module of DNA (i.e., Customize module) is completed by the instructional designer (ID). The ID provides information such as the domain to be decomposed by SMEs and characteristics of the learning population. This information provides the point at which the SME should start decomposing (i.e., superordinate goal) and the point at which the SME should stop decomposing (i.e., the lowest-level subordinate goal). After providing the required information, the Customize module generates a personalized letter explaining the purpose of the project to prospective SMEs and a floppy diskette which will be mailed to prospective SMEs. The diskette contains files for a SME to install on his or her computer which DNA needs to elicit and store knowledge structures.
Orient. Before the SME begins the program, a general orientation to DNA is provided in the second module. This includes a description of the different knowledge types (introduced very simply) along with a summary of how the program will be operating.
Decompose. Once a SME receives and installs the files from the DNA diskette, he or she begins the third module (i.e., Explicit Decompose module). This consists of a fairly structured, interactive dialog between computer and SME, specifically designed to elicit most of the explicit knowledge associated with the domain/topic. The program concurrently attempts to elicit implicit knowledge an expert may recall during the course of the interactive dialog.
DNA utilizes the "What, How, Why" (WHW) questioning procedure which has been shown in the past to successfully elicit knowledge from experts (e.g., Gordon, Schmierer, & Gill, 1993). WHW questions are designed to elicit SK, PS, and CK, respectively. For example, Symbolic knowledge is obtained via questions such as. "What is a definition of ______?". Procedural skills are obtained via questions like, "How do you _____?", or "What is the first step you do when you ______?". And conceptual knowledge is obtained via questions such as, "Why is _______ important?".
In addition to the WHW questioning technique, the program utilizes a structured interviewing technique in a modified funnel sequence. That is, questions are designed to elicit general superordinate goals first, then specific knowledge related to these superordinate goals. However, our system modifies the funnel sequence by allowing experts the choice of decomposing knowledge in depth- or breadth-first manner. That is, experts are not forced to generate all superordinate goals before decomposing into subordinate goals. Rather, experts are allowed to generate a superordinate goal and decompose it at any point during the knowledge acquisition phase. Thus, the funnel sequence is not strictly enforced in order to allow customization of use. Questions are iterated until the expert has exhausted domain knowledge.
In an effort to acquire implicit knowledge (e.g., PS that has become automatic), experts are prompted to list procedures they utilize, answer questions as to why they perform certain steps, and under which conditions these steps are performed. Experts will also be asked to submit with their data, a few common situations, as well as a few novel situations, they have encountered during their careers. These situations will be presented to other experts and solved in a subsequent module. This is similar to the PARI method (Hall, et al., 1995) that employs expert dyads who solve problems for one another. These worked problems then become input for instruction.
All information given by experts will be stored in a database as a list of curriculum elements (CEs) which are components of knowledge needed in order to develop expertise in the domain. By storing information in a CE record, it becomes easy to translate the information into teachable units and generate curricula.
Network/graph. This fourth module loads all of the CEs contained within the CE record (generated during the Explicit Decompose module) and allows experts to arrange and link them into learning hierarchies and conceptual graphs. Each node contains the name of the CE and its contents as defined during the Explicit Decompose module. Links placed between CEs can differ in terms of strength (i.e., weak, moderate, strong-showing the degree to which the items are related), as well as directionality (i.e., uni-, or bi-directional-indicating which CEs are prerequisites for other CEs).
The graphical representation is designed to make relationships among knowledge bits salient. Furthermore, this format can highlight missing knowledge components, which is helpful to both experts and IDs. The procedure is similar to conceptual graph analysis with the exception that with DNA, experts generate the conceptual graphs, as opposed to the ID generating the graph and presenting it to experts. We speculate that DNA will enable experts to recognize gaps in the knowledge and skills they provided earlier. Moreover, they have a chance to correct inadequacies as they can return to the Decompose module and update the CE record with new information.
After SMEs complete the Network module, data are stored on a floppy diskette and returned to the ID. The ID reviews the CE record and conceptual graphs for any glaring omissions in content. If any omissions are present, the ID can ask the expert to expand the inadequate CEs.
Assess. The final module is very important. It is used to validate the CE record and conceptual graph generated by SMEs. This is accomplished by having other experts in a domain review the conceptual graphs generated by the first group of experts. Multiple experts will be used to edit initial conceptual graphs as a method of modifying and validating the externalized knowledge structures.
A peripheral component of the Assess module is the Implicit Decompose module, used to elicit any implicit knowledge that experts have not previously described. This is accomplished by having additional SMEs review common and novel problems generated in the Explicit Decompose module. Experts are instructed to complete these tasks (either mentally or physically) and report their actions as well as their reasoning for each action. In essence, this is similar to a think-aloud concurrent observation technique; however, a knowledge engineer does not need to be present to collect data (i.e., DNA becomes the knowledge engineer).
The modules are repeated until IDs are satisfied with the content of the revised CE records and conceptual graphs. Efforts have been made to elicit SK, PS, and CK, as well as explicit and implicit forms of knowledge; however, the procedures outlined above must be empirically validated.
Following is a description of the dialog and questioning sequence. Also included is the rationale (based on literature and intuition) behind the selection of questions, dialog, and procedures.
Types of Questions
As mentioned earlier, to maintain a dialog between the SME and the computer, we use "What, How, Why" (WHW) questions because they are chatty-easy to interpret and understand. Graesser & Clark (1985) developed this methodology for use on prose comprehension. WHW questions can be used to elicit all our knowledge types: "What" questions relate to "SK", "How" questions to PS, and "Why" questions to CK.
Questions typically move from general to specific. However, our procedure allows the SME to decide whether to decompose in a depth-first or breadth-first manner. For instance, an expert may decompose a series of steps, from first to last, successively. This corresponds to a depth-first approach. Alternatively, an expert may elect not to specify all the low-level steps after identifying some procedure, but continue specifying parallel, superordinate goals. This corresponds to a breadth-first approach.
Conventions in the Following Examples
The questioning sequences described below are accompanied by examples of DNA questions which analyze the domain of Blackjack. General questions generated by DNA are coupled with variables provided within experts' answers to previous probes. General questions are worded such that they can be used across a variety of domains, whereas variables are used to make the questions specific to the domain being analyzed. In the following examples, variables are denoted by italicized words embedded in the question.
A SME responds to DNA's various questions by choosing words and phrases from different list boxes which contain words designed to prompt SME answers (see Figure 3), and also by just typing in text. List boxes are used in an effort to overcome the natural language problem. Free-form typing allows the SME to effectively verbalize knowledge and skills required. In the given examples, prompts selected from list boxes are displayed in underlined type, whereas free-form answers are displayed in normal font.
Sequence
DNA begins by asking a broad questions followed by ones designed to elicit information associated with the specified node. The expert has the choice of answering any question with either "I know how to" do something (procedural), or "I understand" something (conceptual).
Example 1:
DNA: What do you know that makes you good at Blackjack?
SME: I know how to count cards.
The "know how to" path is primarily designed to elicit PS. However, provisions are made to elicit SK and CK associated with PS element described (e.g., "count cards"). The "know how to" path is described in greater detail below. Based on the current response, the program now asks the SME the following query.
Example 2:
DNA: What do you know that makes you good at Blackjack?
SME: I understand card counting strategies.
The "understand" path is designed to get more at CK as well as additional SK and PS. The "understand" path is described in greater detail below.
The "know how to" path is a linearly arranged set of questions primarily designed to elicit PS. However, embedded "why is this step important" questions elicit CK associated with the specified step. Embedded "when do you perform this step" questions elicit additional CK elements such as 'Under what conditions is this step performed?' Experts have the option of defining related terms and concepts during the unpacking of each step. This option elicits SK associated with the specified step. The "know how to" path is linearly arranged because procedures are perceived as being inherently linear (i.e., step-by-step).
Example 3:
DNA: What is the first step you generally do when you count cards?
SME: Initialize my mental card counter.
The SME then associates a name with this step.
Example 4:
DNA: Give it a name.
SME: initialize mental counter
The SME is then directed to a screen which presents a "why" question designed to elicit CK associated with this step.
Example 5:
DNA: Why is step: initialize mental counter important?
SME: Because all the cards are in the shoe, so probabilities are all equal.
Also available on this screen is a button which allows the SME to define any terms that may have been used in the SME's description. When this button is selected, the SME is presented a screen which allows him/her to enter the term and associated definition. SMEs are also prompted to include an example of the term in their definition.
Example 6:
DNA: Term:
SME: Shoe.
DNA: Definition:
SME: a thing in Blackjack where lots of decks of cards are kept.
The SME is then directed to a screen which presents a "when" question designed to elicit additional CK associated with this step.
Example 7:
DNA: When do you perform step: initialize mental counter?
SME: When the shoe has been shuffled.
Upon entering information associated with a step, the SME is allowed to enter additional steps, edit existing steps, or delete steps. When all step revisions are completed, the SME is prompted to place the steps in the order in which they are typically performed. After the steps have been ordered, DNA returns the SME to the "What do you that makes you good at _____?" question which allows the SME to define a new superordinate goal.
The "understand" path is a horizontally arranged set of questions designed to elicit CK, and some additional SK and PS as well. Embedded "why is it important to understand _____?" questions elicit CK. Embedded "How do you apply your understanding of _____?" questions elicit additional PS (see the "know how to" path described above). Embedded "What is a definition of ____?" questions elicit a SK definition. Embedded "What are some additional concepts and definitions related to _____?" elicit additional SK definitions associated with the node.
The "understand" path is horizontally arranged because knowledge is not necessarily linearly arranged. The SME does not have to answer all of the questions (like those listed above) as some may be irrelevant. A "None of the above apply, but here's what I mean..." option is included in the event the SME does not find any of the questions relevant. Selecting the "None of the above" option leads the SME to a screen which will accept free-form typing without any prompt constraints. Note: selecting the "how" question leads the SME to the "know how to" branch to obtain information about steps. Upon completion of the "know how to" branch, the SME is returned to select additional questions.
Example 1:
DNA: Why is it important that you understand card counting strategies?
SME: Because it allows me to estimate the chances of 10s being dealt.
Example 2:
DNA: What is a definition of card counting strategies?
SME: A strategy in Blackjack that involves keeping a count of the number of 10s that have been played in relation to the total number of cards played at a given point in time.
Example 3:
DNA: What are some additional concepts and definitions related to card counting strategies?
DNA: Concept:
SME: The situation with Less than Avg chance of 10 being dealt
DNA: Definition:
SME: A lot of 10s have already been played, so few 10s remain in the deck. Consequently, there's a reduced chance of 10 being dealt.
This interactive process is repeated for additional definitions.
CE Labeling and Numbering
The CE record is a Microsoft Access database, constructed in real-time (i.e., while SMEs enter data). We used this particular database structure because MS Visual Basic, the programming language in which DNA is written, can easily interface with Access. Each CE that is defined by the SME is entered into a new record in the database. CEs are given a name and a unique number. Information related to each CE is also stored in the record (e.g., description of the CE, "why" information [if any], "when" information [if any], and knowledge type).
CE names are typically derived from the name the SME associates with a given knowledge chunk. And CE numbers are calculated based on when the SME chose to decompose a given knowledge chunk. CE numbers are also used to indicate relationships among nodes. For example, suppose the node "count cards" is assigned a CE number of 2. The first step entered for counting cards is assigned a number of 2.001. The second step is assigned the number 2.002. This shows the order in which steps are performed for the main CE (i.e., CE-2). Additional definitions associated with steps are stored in a similar extension format, as described above. For example, a definition associated with step 1 (i.e., CE-2.001) would become CE-2.00101. This shows that the definition is associated with step 1 for the main CE-2.
CE descriptions are typically obtained from the free-form typing that the SME includes to describe the knowledge or skill. Free-form typing is stored as an entry in the database. The ID can look at it later for additional information. The provision for free-form typing was made because there are inevitably instances where what the expert wants to enter doesn't quite fit the list boxes. DNA does not use this information; rather, it is of potential interest to the ID.
"Why" information is provided when an SME enters information in the "why" screen for a step during a trip down the "know how to" path. Thus, it is not always required (i.e., it is supplemental to the primary PS being delineated). "Why" information obtained via the "understand" path is stored in an individual record as it relates to the primary CK. "When" information follows the same logic as the "why" information above.
Knowledge type is collected by DNA as the SME enters data. "What" information is categorized as SK by the system, "how" and "when" data (i.e., steps) are labeled PS, and "why" information is CK. These categorizations may be edited later either by the current SME, a different SME, or the instructional designer.
When is the process/program over? When does the SME know to quit, satisfied that all information he/she possesses has been articulated? Each CE outlined by the SME is entered into a queue. As each CE is unpacked (i.e., explained or delineated), the CE listed in the queue is checked with a "»" symbol. This tells the SME that he/she has already unpacked this CE. When all CEs have been checked, the SME may quit. This doesn't necessarily mean that the SME has entered everything he/she knows about the domain. Rather, all the superordinate goals that have been entered have been unpacked. The SME may exit DNA at virtually any time.
In summary, the process of conducting CTA generally involves: (1) Identifying key job tasks and training issues, (2) Developing visual representations of knowledge structures, (3) Describing cognitive processes underlying performance, (4) Identifying differences between experts & novices, and (5) Determining implications of results for the design phase (Redding, 1992). Historically, this has been an extremely labor-intensive series of activities.
Researchers have proposed that automating the CTA procedure could provide a solution to the bottleneck that CTA presents to automated instructional development (Redding, 1989). The DNA approach to CTA is designed to elicit knowledge from a SME which will Decompose a domain, Network the knowledge into comprehensive structures, and employ other experts in a given domain to Assess the validity and reliability of the knowledge structures. The specific goals of DNA are to create an easy-to-use cognitive task analysis procedure capable of eliciting comprehensive curricula from experts across many domains. The prototype of DNA is a series of interactive modules that automate knowledge acquisition and are based on theoretical and empirical support previously reviewed.
Future research directions for automated cognitive task procedures
DNA can be used as a tool to explore numerous basic and applied issues in cognitive psychology. One important applied issue which must be addressed is whether DNA can obtain knowledge structures comparable to existing CTA methods in a shorter period of time. Furthermore, can knowledge structures obtained by DNA be easily translated into curricula? Research in this area must be conducted to determine the efficacy of DNA compared to other methods of CTA.
Can DNA obtain a comprehensive knowledge structure? The questioning dialog DNA utilizes is designed to elicit explicit forms of knowledge such as SK and CK (i.e., definitions of terms and relationships among terms), as well as implicit knowledge structures (i.e., PS elements that experts cannot easily recall). It is important to validate whether implicit knowledge can be elicited with retrospective observation of novel and common situations.
Existing CTA methodologies are inefficient and often provide
incomplete knowledge of the domain. The proposed automated cognitive task
analysis procedure provides a principled approach to CTA which is based on
theory and research. DNA is designed to elicit a comprehensive knowledge
structure (i.e., both explicit and implicit versions of SK, PS, and CK) which
can easily be translated into a curricula. However, extensive research must be
conducted to assess the effectiveness and efficiency of such a method.
Anderson, J. R. (1988). The expert module. In M. C. Polson and J. J. Richardson (Eds.), Foundations of intelligent tutoring systems, (pp. 21-50). Hillsdale, NJ: Erlbaum.
Anderson, J. R. (1993). Rules of the mind. Hillsdale, NJ: Erlbaum.
Benysh, D. V., Koubek, R. J., & Calvez, V. (1993). A comparative review of knowledge structure measurement techniques for interface design. International Journal of Human-Computer Interaction, 5(3), 211-237.
Collins, H. M. (1985). Changing order: Replication and induction in scientific practice. London: Sage.
Cooke, N.J. (1994). Varieties of knowledge elicitation techniques. International Journal of Human­Computer Studies, 41, 801­849.
Crandell, B., Klein, G., Militello, L. G., & Wolf, S. P. (1994). Tools for applied cognitive task analysis. Contract summary report on N66001-94-C-7008. Fairborn, OH: Klein Associates.
Dehoney, J. (1995). Cognitive task analysis: Implications for the theory and practice of instructional design. Proceedings of the 1995 Annual National Convention of the Association for Educational Communications and Technology, Anaheim, CA.
DuBois, D. A., & Shalin, V. L. (1995). Adapting cognitive methods to real-world objectives: An application to job knowledge testing. In: P. D. Nichols, S. F. Chipman, & R. L. Brennan (Eds.) Cognitively diagnostic assessment. Hillsdale, NJ: Erlbaum.
Durkin, J. (1994). Knowledge acquisition. In J. Durkin (Ed.), Expert systems: Design and development, (pp. 518-599). New York, NY: Macmillan.
Essens, P.J.M.D., Fallesen, J.J., McCann, C.A., Cannon­Bowers, J., & Dörfel, G. (1994). COADE: A framework for cognitive analysis, design, and evaluation. Final Report of RSG.19 on Decision Aids in Command and Control. NATO Defence Research Group AC/243 (Panel 8/RSG.19).
Gordon, S. E., & Gill, R. T. (1997). Cognitive task analysis. In C. Zsambok & G. Klein (Eds.), Naturalistic Decision Making (pp. 131-140). Hillsdale, NJ: Lawrence Erlbaum.
Gordon, S. E., Schmierer, K. A., & Gill, R. T. (1993). Conceptual graph analysis: Knowledge acquisition for instructional system design. Human Factors, 35, 459-481.
Graesser, A. C., & Clark, L. C. (1985). Structures and procedures of implicit knowledge. Norwood, NJ: Ablex.
Hall, E. P., Gott, S. P., & Pokorny, R. A. (1995). A procedural guide to cognitive task analysis: The PARI methodology. (Report No. AL/HR-TR-1995-0108). Air Force Materiel Command, Brooks Air Force Base, TX.
Hayes-Roth, F., Waterman, D. A., & Lenat, D.B. (1983). Building expert systems. Reading, MA: Addison Wesley.
Kirwan, B. & Ainsworth, L.K. (Eds.) (1992). A guide to task analysis. London: Taylor and Francis.
Means, B. (1993). Cognitive task analysis as a basis for instructional design. In M. Rabinowitz (Ed.), Cognitive science foundations of instruction (pp. 97­118). Hillsdale, NJ: Erlbaum.
Olson, J. R., & Biolsi, K.J. (1991). Techniques for representing expert knowledge. In K. Anders Ericsson & J. Smith (Eds.), Toward a general theory of expertise (pp. 240­285). Cambridge, England: Cambridge University Press.
Redding, R. E. (1989). Perspectives on cognitive task-analysis: The state of the state of the art. Proceedings of the Human Factors Society, USA, 33, 1348-1352.
Redding, R. E. (1992). A standard procedure for conducting cognitive task analysis. McLean, VA: Human Technology, Inc. (ERIC Document Reproduction Service No. ED 340 847).
Shute, V. J. (1995). SMART: Student modeling approach for responsive tutoring. User Modeling and User-Adapted Interaction, 5, 1-44.
Shute, V. J., & Torreano, L. A. (1995). Cognitive task analysis made simple: Decompose, network, assess (DNA). Unpublished manuscript.
Willis, R. E. (1996). Automating the cognitive task analysis
procedure. Final report: Graduate Student Research Program at the Armstrong
Laboratory, Brooks AFB, TX.
*Armstrong Laboratory has been reorganized
and is now part of the Air Force Research Laboratory.