ANALYZING DATA QUALITY OF CITIZEN SCIENCE PROJECTS THAT DON'T HAVE A SINGLE CORRECT ANSWER

Background: Historically, data quality and validation have not received as much attention in the field of citizen science as other topics such as basic project development. The result is that many citizen science project developers ignore the problem and simply assume the results will be valid. This can be a dangerous assumption if the goal is to gain scientifically valid data that can be published or used as the basis for implementing new policies. We will use Arizona BatWatch, a new, online citizen science project focused on bat behaviors, as a case study to illustrate one way of addressing data quality questions for a project where identifying a single correct answer is often not possible. Methods: Arizona BatWatch uses archived videos that often show large numbers of bats potentially doing different things simultaneously. Our beta test indicated that with so much happening at once, it is easy for anyone to miss a behavior, regardless of expertise. This led us to focus our validation analysis on whether the citizen scientists’ results were as accurate as the experts’ results, not on whether the citizen scientists identified a single correct answer. We also wanted to know how many citizen scientists needed to watch each video to produce results statistically equal to our team of experts. We used chi-square analysis and Wilcoxon Signed-Rank Test to determine the answers to these questions. Results: Our initial results indicated that, yes, our citizen scientists could classify the videos as well as our experts (p < 0.05). To determine the answer to our second question, we performed multiple random samples with increasingly smaller sample sizes until we obtained a significant p-value for each behavior. The required sample sizes differed for each behavior; however, the largest sample size needed was 30 citizen scientists. Conclusion: A common goal among citizen science projects is to produce publishable results and/or results that can be used as the basis for new policies. To accomplish this goal, project developers are faced with the difficult task of proving their results are scientifically valid. The task becomes even more complicated in studies where there may not be a single correct answer such as behavioral studies or surveys focusing on bird or frog calls. These are often the studies where observer bias can play a significant role in data collection even among experts. We offer one way of statistically analyzing such data for QAQC purposes.