HBP-SciUnit
SciUnit is a Pythonic validation framework for assessing the agreement between computational models and experimental data. SciUnit is underlies several domain-specific libraries for model validation including NeuronUnit, MorphoUnit, and eFELunit.
Project Description & Objectives
Computational models serve as a powerful tool for furthering our understanding of both the normal function of the nervous system and the pathology associated with aging, neural trauma, or disease. Such models provide a framework for integrating data across spatial scales and for exploring hypotheses about the biological mechanisms underlying neuronal and network dynamics. However, as models increase in complexity, additional barriers emerge to the creation and validation of models. Many publications do not include a specific, rigorous statement of the criteria used to evaluate models during development; it is similarly rare for multiple models to be compared for concordance with the same suite of experimental data. Our group has developed a software framework for validating computational models called SciUnit, where data-driven model validation tests can be defined to recapitulate experimental protocols in simulations, and simulation results can be compared to corresponding experimental results. NeuronUnit provides a module within SciUnit that provides tests that can be used to automatically assess the scope and quality of computational models in neuroscience. This framework allows model developers to decouple the validation test definition from the model implementation, so that one can test different models for the same neuron type or change the implementation of a model, without having to modify or re-write the validation tests.
Currently, the HBP Co-Design Project 2 Product 2 Single Model Benchmark and Validation Suite provides components that are used to validate models developed at HBP by testing them against experimental data. The existing components include tests for models from specific brain regions: hippocampus, cerebellum, and basal ganglia. There are also components that are more general for testing specific model features: neuronal morphologies, resting state activity, electrophysiology feature extraction, and sub-cellular features. These can be used to test models across different brain regions. In addition, a web portal is available for sharing test results for any Human Brain Project models. As described on the HBP Collaboratory website, currently these validation tests are written in Python, using the SciUnit framework described above.
Collaboration with HBP
Every computational project benefits from a robust unit-testing framework. When the code represents a scientific model, these unit tests include model validation tests. We bring a robust model validation framework to HBP, one already in use in the broader neuroscience community, and will help to address some HBP-specific needs including interactive visualization of the model validation process, better representation of intermediate data analysis objects, and improved support for standard file formats.
Increasing the functionality and usability of the HBP Single Model Validation Suite will impact all HBP modelling projects since this is a platform that cuts across platforms, improving efficiency of model development, providing transparency about model performance, and promoting model reproducibility. Clear and transparent model evaluation also will contribute to model re-use by improving community confidence of these models. This collaboration between the SciUnit developers and the developers of the HBP Model Validation Suite also will provide critical feedback to SciUnit and NeuronUnit through these proposed HBP activities, impacting many additional modelling studies being performed by researchers outside of the HBP.
Partnering Organisations
Biographies
Richard C. Gerkin (Project Coordinator)
I work on developing data-driven validation tests for neuroscience models, especially those at the level of neurons, synapses, and ion channels. I also contribute to several large open source computational neuroscience software projects. I also build statistical models of human and rodent olfaction. Google Scholar Profile
My work focuses on developing and using mathematical and computational approaches to study the dynamics of neurons and neuronal networks. I am particularly interested in the mechanisms underlying structural plasticity in the nervous system and how sensory systems work and are integrated. I also contribute to an international effort to promote reproducibility, model sharing, and community-based collaborative model development in computational neuroscience research. Google Scholar Profile
Publications
R.C. Gerkin, J. Birgiolas, R.J. Jarvis, C. Omar, S.M. Crook, “NeuronUnit: A package for data-driven validation of neuron models using SciUnit”, bioRxiv 665331, reviewed at Frontiers in Neuroinformatics, 2019.
R.C. Gerkin, R.J. Jarvis, S.M. Crook, “Toward systematic, data-driven validation of a collaborative, multi-scale model of C. elegans”, Philosophical Transactions of the Royal Society B, 373 20170381, 10.1098/rstb.2017.0381, 2018.
G.P. Sarma, T.W. Jacobs, M.D. Watts, V. Ghayoomi, S.D. Larson, R.C. Gerkin, “Unit Testing, Model Validation, and Biological Simulation”, F1000Research, 2016, 5:1946.
