Advancing the Metagenomics Revolution

The vast majority of life on earth is microbial. Virtually all ecologies rely on the intricate biochemistry of microbial life to sustain themselves. Historically most research on microbes depended on laboratory cultures, but since 99% of microbes cannot be cultured, it is only recently that modern genetic sequencing techniques have allowed determination of the hundreds to thousands of microbial species present at a specific environmental location. The amount of data specifying the “metagenomics” of these microbial ecologies is explosively growing as researchers everywhere are acquiring next generation sequencing devices. Since many genes are related across microbial species, the community needs repositories in which diverse environmental metagenomics samples can be quickly compared, both by comparing genomic data or environmental metadata. I will give a quantitative example of the computing, storage, software, and networking architecture needed to handle this exponentially growing data flood by describing the Gordon and Betty Moore Foundation funded Community Cyberinfrastructure for Advanced Marine Microbial Ecology Research and Analysis (CAMERA) which is hosted by Calit2@UCSD. The CAMERA repository currently contains over 500 microbial metagenomics datasets (including Craig Venter’s Global Ocean Survey), as well as the full genomes of ~166 marine microbes. Registered end users, over 3000 from 70 countries, can access existing and contribute new metagenomics data either via the web or over novel dedicated 10 Gb/s light paths.  The user’s BLAST requests transparently activate programs on dedicated and shared parallel computing resources at UCSD.  To better support the CAMERA user community, we developed a new component-based cyberinfrastructure, CAMERA Version 2.0.  This new cyberinfrastructure will support future needs for data acquisition, data access through diverse modalities, the addition of externally developed tools, and the orchestration of these tools into reproducible analytical pipelines. The management of remote applications and analyses is accomplished via the Kepler workflow engine which supports the natural interaction of automated computational tools that can then be re-utilized and openly shared. Finally, CAMERA 2.0 includes an effective, flexible, and intuitive user interface that facilitates and enhances the process of collaborative scientific discovery for biosciences.  I will conclude by examining future trends in metagenomics data generation, data standardization, and the possible use of cloud computing and storage.