Continuous Research

Setting Integration as the Default


Science today is built almost entirely on a culture of competition. Formal integration of research from different groups working on the same topic happens very rarely. When it does happen, for example when results/algorithms are explicitly compared, these results are often highly biased1 or incomplete.

Continuous Research is the concept that all scientific outputs, every new dataset, technique, and hypothesis, can be can be integrated such a way that they are re-evaluated against all related research. In developer-speak we could say that we're "bringing Continuous Delivery to research".

What is Continuous Delivery?

The requirement to maintain large, distributed, and complex projects, along with the high availability of low-cost computing has driven the widespread adoption of "Continuous Delivery" in software development. The mechanics can be complex, but the principle is simple: automated tests are frequently run against the software so that even when a brand new feature is added, you can be sure that the software is still performing correctly and (at least close to) functional enough to be released immediately.

How is research done now?

You collect a dataset, you do some analysis to test a hypothesis, and then try to integrate that hypothesis with current theories. Great, there's your paper, submit it to Journal of Sciency Things and move onto the next project. But what happens when someone else collects a dataset that could also be used to test that hypothesis? Or someone working in the mathematics department writes a better version of your analysis. Best case scenario they might use your code/data in their papers, otherwise they might just mention or cite you. But the work will never be truly integrated, and each paper might be related to tens/hundreds of other pieces of research, meaning that manually finding and using all of the code/data associated with their work will not always be possible, even if it is open.

How does that change under Continuous Research?

Continuous Research is a framework we re-evaluate all hypotheses when any new data, analysis, or pipeline becomes available. In such a way, we obtain a live view of how much support there is for a particular theory, and under what conditions such a theory really holds. The big picture being"that we want to try and integrate related research as tightly as possible, rather than the loose collection of papers that is the de facto standard for a body of evidence.


Prototypical Workflow

Use Cases

Let's imagine someone has performed the following simplified study.

Dataset (Neural activity & the position of the animal)

Analysis (Divide the environment into a grid then count the number of "spikes" each neuron fires at each place in the grid)

Hypothesis (Some neurons will show some specificity at a particular location in this grid)

Theory (Place cells provide animals with a map that enables them to navigate the world around them)

Now we consider what could happen when new research is done in the area, and see how the principles of CR might our understanding to be rapidly updated

New Dataset

Someone does a similar set recordings but taken from a different region of the brain. All the analyses are automatically repeated. We don't find place cells here. We refine the hypothesis to "Neurons in the hippocampus will show specificity to a particular location" and update our theory.

New Analysis

Someone else thinks that counting the number of spikes might not be a great way of determining locational specificity of neuronal firing. Perhaps we should instead determine the information content that a neuron has about the position of the animal. This analysis is then automatically applied to all compatible datasets. It turns out only a subset of the neurons we thought were place cells actually contain information about where the animal could be.

New Pre-processing

Someone thinks that your results were an artifact of the way you identified which spikes came from which neuron (known as "spike-sorting"). 1) They provide a better spike-sorting method. All analyses are re-run using this method. We can now see whether this makes a difference. Or 2) They can already see that this study has been re-run with every available spike-sorting algorithm and the results seem robust to this choice.

New Visualisation

A colormap was used to plot the rate of a neuron depending on the location of the animal. Someone chose "jet" for the original figure but it's more clear using "hot". When this change is made, both versions of record will be available for people to see and decide what works for them

New Hypothesis

Someone might disagree that neurons showing spacial specificity at a particular location is evidence for the theory that place cells help animals to navigate. They can then dispute the link between the hypothesis and this theory. Or they can propose a new hypothesis which would test the theory, perhaps "disruption of these cells should prevent the animal from navigating effectively".


While technology continues to improve, the methods underlying much of research haven't quite entered the 21st Century. The pace of change is now increasing which could be a great thing, but I am concerned that we are still baking in many out of date values into our new practices. "Open Science" is not really open unless it is truly participatory. Small steps are great, and we encourage everyone to get started in whatever way is most accessible to them. Nonetheless simply putting the code you used for a piece of research isn't fulfilling the potential of what Open could be. But if your code is running live somewhere and anyone can come along and play with the parameters to see what effect that would have had on the outcome of your experiments - that's getting a bit closer. If there is a low-barrier to someone recreating your visualisations in a way that's accessible to them even better. In a technical sense open for me additionally nesseciates accessibility, extensibility and interoperability. And this project is about showing how powerful research can be when it meets these criteria.


What Problems Exactly can we Solve?

Independent & Explicit comparisons of Model/Algorithm/Software performance

In every field there seems to be at least one problem for which the current paradigm is "look here how our solution to this problem which we have been tuning for a year compares against all these other ones we just downloaded". With this way of thinking it becomes very difficult to really determine what works best. Within Continuous Research each new algorithm would automatically be benchmarked against all the previous algorithms, with all the previous datasets.

Exact Provenance tracking for the effects of changes to models / datasets

Not only are all changes to all parts transparent & version controlled, but we also have the ability to see what the impact of each change was to both the final output, and each intermediate stage.

Truly Participatory Science

The more modularisation we can achieve the more we can lower the barrier for people to contribute to one aspect of a project without knowing the ins and outs of the other parts. And they can see in real time what the impact of there contributions are.

"Incremental Progress is Still Progress"2

How many people have had their research put down because it was "incremental"? There is very little that annoys me more than this attitude because no research exists in a vacuum and ultimately if we keep making incremental progress then we will move forward just fast enough! The difficulty lies in that currently incremental improvements are lost because publishers, funders, and even other scientists undervalue these advances. With Continuous Research all relevant research is explicitly linked improving efficiency by preventing incremental improvements getting lost and/or repeated unnecessarily.

Fail Fast3

We are reducing the "knowledge cycle" - the time it takes to get feedback and release a project so that you and others can start building on top of it from months to < 1 day.

Why Continuous Research is Decentralized Research

Let's take a look at the features that define the current centralised system, and see how they would be inverted under this framework.

Single/Few Points of Control

Currently a handful of journals, funders, and senior scientists have the majority say in what is good science and what isn't. They make all of the decisions that matter, for better or worse.

With CR, the validity of a piece of work isn't decided at one point in time (on publication) or by a few people but is continuously re-assessed, both by the automatic re-running of pipelines and data as well as via the facility for comments and discussions of how strong evidence for the original hypotheses were.

High Costs / Barrier to Entry

The cost of completing a "formally recognised" scientific project at the moment is astronomical. There are fee's for reading papers, obtaining data, software for analysis, compute for analysis and of course the infamous and misleadingly named "Article Processing Charge". Moreover, there is increasingly the expectation that a single piece of scientific work invoke a wide range of skills or techniques. All of these costs can be minimised, but only to a certain degree, and often have tradeoffs in terms of the impact your work can have.

Continuous Research is totally modular, so you only need to submit one piece of the puzzle in order for your work to have impact. As we mentioned before, this lowers both the skill and the cost barrier to contributing to research. If you can't collect data yourself you can either search for existing datasets OR you can simply write the pipeline which will be immediately ready for when the data is available. All analytical pipelines are available and compatible which might reduce the need for proprietry software and compute resources can be distributed across seeders to facilitate demanding processes.

Technical Details

Implementation & MountainLab Study

A large subset of the implementation details for such a project overlap with MLStudy, a "low-friction data-analysis platform in the web-browser" being developed by Jeremy Magland at the Flatiron Institute.

Key Features:


With MLStudy we already have most of the ingredients we need for decentralisation.

The main missing part is distributed/de-centralised storage of data. At the moment data has to be stored locally or pulled from a cloud server. However luckily we don't have to solve this problem ourselves thanks to the Dat Project. Support for dat data sources will be the key next step towards decentralisation. From there it will also be possible to experiment with how well the whole project could run as a peer-to-peer network.


  1. Ref?
  2. Kirstie Whitaker - CollabW18 Talk
  3. I apologize for this horrible tech buzzword