software development communities, I have observed that the latter often lives
up better than the former to our ideals of scientific openness and
reproducibility. I will explore the reasons behind this, and I will argue that
these problems are particularly acute in computational domains where they
should be in fact less prevalent.
Open source software development uses public fora for most discussion and
systems for sharing code and data that are, in practice, powerful provenance
tracking systems. There is a strong culture of public disclosure, tracking and
fixing of bugs, and development often includes exhaustive automatic validation
systems, that are executed automatically whenever changes are made to the
software and whose output is publicly available on the internet. This helps
with early detection of problems, mitigates their reoccurrence, and ensures
that the state and quality of the software is a known qunatity under a wide
variety of situations (operating systems, inputs, parameter ranges, etc).
Additionally, the very systems that are used for sharing the code track the
authorship of contributions. All of this ensures that open collaboration does
not dilute the merit or recognition of any individual developer, and allows for
a meritocracy of contributors to develop while enabling highly effective
collaboration.
In sharp contrast we have incentives in computational research, strongly biased
towards rapid publication of papers without any realistic requirement of
validation, that lead to a completely different outcome. Publications in
computationally-based research (applied to any specific discipline) often lack
any realistic hope of being reproduced, as the code behind them is not
available, or if it is it rarely has any automated validation, history
tracking, bug database, etc.
I will discuss how we can draw specific lessons from the open source community
both in terms of technical approaches and of changing the structure of
incentives, to make progress towards a more solid base for reproducible
computational research.
See more of: The Science Endeavor
See more of: Symposia