Theoretical background
During a sort exercise, participants are presented a list of concepts that are to be sorted into different piles according to their own perception of similarity or other association. This represents their structural organization of the concepts deemed important and relevant to the content area being researched. Cognitive maps derived from card sort data are effective for comparisons between groups, such as between the students and the professional engineers. This is one of the most common applications for cognitive mapping where assessing student knowledge of a particular content area is the objective. Their maps are then compared to an expert map to determine inaccuracies and gaps in their knowledge structure, thereby informing instruction (Jonassen et al., 1993).
Card sorts allow examination into the meanings of words as perceived by the individual (Miller, 1969). The resulting stacks, or categories, indicate the relationships the individual sees among the concepts, and these can then be compared with other individuals or groups. Harper, Jentsch, Berry, Lau, Bowers, and Salas (2003) cite two studies (Fiore, Cuevas, & Oser, 2003; Fiore, Cuevas, Scielzo, & Salas, 2002) where the data consistently shows that “similarity to an expert model is positively related to performance on measures of knowledge acquisition” (p. 579). Therefore the card sort procedure has been utilized extensively for expert/novice comparisons and is good for identifying “organization of knowledge in a content area and to identify areas of knowledge deficiency” (Jonassen et al., 1993, p. 51). Thus while it provides a snapshot of an individual’s cognitive structure, it is also suitable for discovering gaps and misperceptions in students’ understanding of subject area content. Chi, Hutchinson, and Robin (1989) conducted a card sort as part of a multi-method study and discovered that the sort task clearly revealed a higher-level hierarchy as opposed to merely indicating basic similarity among concepts.
Research on the card sort exercise indicates that it is a stable, reliable method. Studies by Evans, Hitt, and Jentsch (2001) and Tessmer, Perrin, and Bennett (1998) all concluded that repeated card sorts by individuals produce reliable, consistent outcomes. Validity concerns have been raised by some researchers; however, many believe that it is still an effective tool for assessing how an individual views conceptual relations (Harper et al., 2003; Fiore, Cuevas, & Oser, 2003; Jonassen et al., 1993), especially if implemented as one component of a multi-method approach. Cheatham and Lane (2002) found that card sorting predicted user performance better than other methods. A comparison study by Fiore, Fowlkes, Martin-Milham, and Oser (2000) showed card sort and similarity ratings to be equally effective. Recommendations regarding the number of subjects desired for satisfactory results generally range between five and thirty (Card sorting, n.d.; Nielsen, 2004; Maurer & Warfel, 2004). The primary issue is balancing accurate, usable data with the effort required to administer and process large numbers of participants. Nielsen describes how a pool of fifteen participants provides an adequate correlation when compared with a much larger group, though general results can be obtained with as few as five subjects.
There are variations on how the card sort is administered, usually consisting of whether to control how the subject sorts the cards and whether they can add their own concepts and/or piles. These options are situation specific and do not impact the integrity of the exercise (Team Performance Laboratory, Card Sort, 2002). The most efficient and reliable method for administering a card sort task is through computer software. This allows automation and accuracy of the exercise procedure as well as simplified data scoring and analysis. Though many methods for scoring card sort data have been implemented, the most common is a simple “hit or miss” comparison, where all concepts in the exercise are paired to see how many pairs were in the same pile. This is then examined to determine characteristics of the sort, such as whether each individual sorted concepts based on content structural features or hierarchical relationships.
Lastly, card sorts are ideal for eliciting structural knowledge because they require little pre-requisite skills for subjects to complete. In comparison to concept mapping and drawing, card sorts are easy to perform and provide effective representations of cognitive structure (Jonassen et al., 1993; Enger, 1998; Novak, 1990).
