Evaluating Reform Efficacy by Longitudinal Analyses of Student Performance

Despite unified enthusiasm among biology educators for reforming introductory STEM courses, convincing stakeholders about reform efficacy can prove a substantive challenge.  The literature abounds with studies citing improved science affect and learning gains for students in the context of reformed courses using metrics such as existing course exams and/or concept inventories.  Yet, there is a paucity of data about longer term impacts, including whether reformed instruction influences student performance elsewhere in the curriculum. We designed a reformed, large-enrollment, introductory biology course for life science majors.  Our reform model is grounded in existing theory about how people learn and is consistent with objectives articulated in the 2011 report, Vision and Change in Undergraduate Biology Education.  We implemented the reformed course in 3 of 4 sections (approx. 180 students each) in Spring 2009.  In situ student achievement was measured as performance on a comprehensive, case-based final exam that included items across a range of cognitive skills (aka, Bloom levels).  To assess long-term impacts, students from the 2009 cohort are tracked as they matriculate into each of 3 upper-division target courses:  Genetics, Ecology, and Evolution.  Achievement is measured as performance on exam items scored by course instructors. A group of trained raters with IRR > 80% rated all exam items for level of cognitive complexity using Bloom's Taxonomy for Educational Objectives.  For the reformed introductory biology final exam, items ranged from Bloom level 2 - 5 (comprehension, application, analysis, and evaluation/synthesis) with a 3.54 mean rating.  Student achievement on the final exam ranged from 82 - 88% across the 3 sections, with an overall mean of 85% + 9.4%. At present, 10 upper-division exams have been analyzed with items (n=318) ranging from Bloom level 1 – 5 (mean = 1.7).  Overall, student performance on two Genetics midterms indicates a trend toward higher performance across all Bloom levels for reformed students compared to non-reform, with some item responses achieving statistical significance (p<.05). Our preliminary data suggest that a learner-centered, data-driven approach to course reform can promote student achievement on assessments of higher-order cognitive skills.  Further, low alignment between the assessment strategies used in introductory vs. upper-division courses illuminates the need for clarifying expected outcomes for biology majors at all points in the curriculum.