Mental Model Assessment for Education

The task of extracting mental models is challenging due to several factors. Models are dynamic structures, changing constantly through experiences. Individuals themselves often have difficulty articulating exactly what is “in their head”. As is true of all cognitive psychology, issues of the mind are difficult at best to understand, but over the years a number of approaches have emerged as acceptable methods for eliciting and representing mental models. These generally fall into one of three categories: content analysis (declarative knowledge), procedural analysis (procedural knowledge), and cognitive mapping (structural knowledge). 

Content analysis examines written text to determine the presence and frequency of certain terms, though it typically does not provide any context of meaning or relationships between recurring concepts. Therefore it can reveal what an individual knows about a subject, but not meaningful connections within this knowledge. Content analysis is suitable for determining declarative knowledge. Social science research has generally utilized content analysis, which is generalizable and easy to administer (Carley & Palmquist, 1992). 

 

Procedural analysis observes how an individual performs a given task, focusing on both implicit and explicit factors as to not only how, but also why certain actions are performed. This method provides good insight on the structure of what the individual is thinking throughout the process. However, this is limited to the task itself and not particularly on the general knowledge the individual may possess. It is suitable for eliciting procedural knowledge within an individual’s mental model. Procedural analysis, task analysis, and similar techniques are often employed as a complement to cognitive assessment methods (Carley & Palmquist, 1992; Jonassen, 1995).

 

Cognitive analysis is ideal for describing knowledge structures (structural knowledge), which includes content as well as relationships between concepts, and comparing them among a group of people. This makes this method ideal for expert/novice comparisons and classroom/training analysis (Carley & Palmquist, 1992). 

Researchers have long theorized about how humans mediate their internal knowledge structures and how to elicit that information. Carley and Palmquist (1992) believe that language is the key to understanding and mediating mental models. They believe that:

• Both the cognitive structure and the text can be modeled using symbols, i.e. concepts.
• The text is a sample of what is known by the individual and hence of the contents of the individual’s cognitive structure.
• The symbolic or verbal structure extracted from the text is a sample of the full symbolic representation of the individual’s cognitive structure. In other words, mental models can be represented as networks. (p. 3)

Others agree that symbol manipulation plays a key role in mental model theory, described by the philosopher Wartofsky as “cognitive constructions in which individuals organize symbols of their experience or of their thought in such a way that they effect a systematic representation of this experience or thought, as a means of subjective understanding” (Seel, 1999, Overview, para. 1). The belief that individuals interact with and represent the world through symbols has received considerable attention over the years from various perspectives (Carley & Palmquist, 1992). The notion that language is central to model formation is the basis for text-based analysis of domain knowledge, where verbal or written text descriptions from subjects is analyzed for frequency of occurrence of key terms and relations to other concepts, resulting in a network diagram showing these relationships (Carley & Palmquist, 1992). This is the basis for eliciting declarative and cognitive (structural) knowledge, discussed next. Procedural knowledge methods will be covered later in this section.

Representing a mental model in terms of the individual’s structural knowledge requires two steps: elicitation of their knowledge structure and representation of this data in some way that reflects that structure. Some of the more common methods for eliciting structural knowledge include pair-wise ratings, card sorts, verbal protocol, and concept maps (concept mapping combines elicitation and representation in one process) (Jonassen et al., 1993; www.tpl.ucf.edu; Jonassen, 1995; Scielzo, Fiore, Cuevas, & Salas, 2002; Evans, Harper, & Jentsch, 2004; Subramani, Nerur, & Mahapatra, 2002). Pair-wise, card sorts, and concept maps are generally scored through quantitative methods, though there are occasional exceptions for concept mapping. Verbal protocols may also be treated similarly to quantitative written text analysis, but is often examined through qualitative procedures. 

 

Pairwise ratings, or similarity ratings, involve having subjects rate the similarity of two paired concepts on a scale of around 7 or 9 degrees, much like a Likert-type scale. A list of significant concepts is compiled, then all possible pairings of these terms are presented to the participant. For each pair, the individual scores either a lower or higher degree of similarity between the two. This method thus determines relationships between concepts and their relative strengths or weaknesses. Pairwise data can then be fed into a statistical package for producing cognitive maps or Pathfinder nets (discussed later) that provide a visual diagram of concepts and relatedness. This method is based on the assumption that cognitive structure can be viewed in a spatial dimension, that “geometric distances between concepts…reflect the psychological proximity of the concepts in the individual’s cognitive structure” (Jonassen et al., 1993). Research has shown similarity ratings to have a high degree of reliability. Care must be taken, however, in selection of concepts presented in the study to avoid “vague definitions of relatedness” and an overly broad range of concepts that might present different levels of relatedness, confusing the subject. Consequently, selection of concepts should be restricted to a fairly narrow content area (Jonassen et al., 1993).

 

The card sort exercise presents the individual with a stack of cards which have concepts written on them. They are to sort these into piles that the individual believes to have some meaningful relationship or similarity. Miller (1969) notes that card sort is good “for identifying distances between concepts that are organized in a hierarchical structure” (cited in Jonassen et al., 1993, p. 50). Jonassen also notes that they are also good for identifying “organization of knowledge in a content area and to identify areas of knowledge deficiency” (p. 51). There are variations in how this activity is performed; sometimes the piles are pre-determined, most often the user can create piles as they wish. The researcher may also have participants label the piles they create. With computer software, this activity is easily administered and scored, even allowing for multimedia files such as video, audio, and graphics. Card sort similarity data can be used to generate Pathfinder nets and cognitive maps. 

 

Concept mapping is a modeling procedure that simultaneously elicits and represents structural knowledge. The subject writes concepts and draws labeled links between them to indicate relationship structures. Another variation is when the researcher provides the concepts and asks the participants to generate links. As opposed to pairwise ratings and card sorts, which probe internal knowledge structures that the individual may not even be aware of, concept maps are user generated and therefore can only reveal what the participant actually recalls from memory. However, one benefit of this is that it reduces researcher intrusiveness, allowing the subject to “explicitly state the relationships they see” (Williams, 1995, p. 3). Concept maps are very popular for educational applications, where teachers have students draw these diagrams to help in assessment of learning, informing further instructional needs, and providing feedback for the student (Freeman & Urbaczewski, 2002; Enger, 1998; Williams, 1995; Kinchin & Hay, 2000). One drawback to concept mapping is the learning curve involved. The “activity of concept mapping also requires instruction and practice to become “fluent” in the act of setting concepts out on paper or a computer platform” (Enger, 1998, p. 2). “Novak (1990) also noted that skill in concept mapping took at least a year to develop” (cited in Enger, 1998, p. 5). Other factors include the lack of consistency in scoring techniques and other issues (Ruiz-Primo & Shavelson, 1996; Jonassen et al., 1997).