Level Up: Capturing Upper-Division Student Understanding of Natural Selection

The Concept Inventory of Natural Selection (CINS) and the Bishop & Anderson Open-Response Instrument (ORI) are frequently used to determine natural selection content knowledge in courses for introductory students or non-science majors.  The CINS (multiple choice) and the ORI (open-ended) both assess five key concepts of natural selection, including population variation, origin of variation, inheritance, fitness, and population change. While the CINS and ORI are considered valid and reliable when used in an introductory class, their utility for measuring student learning in advanced evolution courses is unknown. Here, we investigate the utility of the CINS and ORI for capturing student learning in a 400-level evolution course. We test the hypothesis that the ORI is more useful to assess upper division students’ knowledge related to natural selection than the CINS. Students (n=76) in two semesters completed both the CINS and the ORI at the start and end of the semester. Students significantly improved on the CINS (t=4.296, df=75, p=0.000) and had high scores on the pre and post-test (mean scores of 68% pre-test, 78% post-test); the normalized change score was 0.34. Students also significantly improved on the ORI (t=3.527, df=75, p=0.001), but scores on the ORI were lower (mean scores of 22.63% on the pre-test, 30.72% on the post-test); the normalized change score was 0.00. The normalized change scores for the CINS and ORI were significantly different (t=5.46, df=65, p=0.00). These results indicate that the CINS does not adequately capture student understanding of natural selection; students may be able to ‘game’ this assessment. The forced response nature of the CINS may allow students to use context clues rather than content knowledge about natural selection.  In contrast, students must construct their responses on the ORI.  Because student scores were so low on the ORI, we further investigated student responses on two items related to trait gain and trait loss. We used a modified Bishop and Anderson coding rubric (IRR=87%) to determine student performance on each question and to understand which of the five concepts of natural selection the students included in their responses. Students performed differentially based on the prompt structure (X²=14.46, df=4, p=0.006). For both prompts students were unlikely to include origin of variation in their response.  On the trait gain prompt students were more likely to include population variation and fitness concepts in their response; the trait loss question only consistently evoked population variation. Together, these results provide evidence for using the ORI over the CINS in upper division evolution courses. In addition, results from the ORI uncover persistent conceptual challenges for students related to natural selection. Further research is warranted to investigate assessment instruments for measuring advanced biology students' understanding of natural selection.