Even more than sharing code, I think the key is to share raw data, postprocessed data and describe methods as exceptionally clear equations and pseudocode.
Some well intentioned papers share code which is hard to run years afterwards. It's sometimes much simpler to reimplement things that to get the code to run if they are described accurately.
Besides, some articles hide ugly things, nasty tricks and lies in the code, which make their results a lot less believable and valid. Being super upfront about models in terms of equations and pseudocode is important.
Of course, we should also have standards to make code reproducible. Perhaps depositing a VM with everything ready to run.
> Perhaps depositing a VM with everything ready to run.
The danger then is that the VM has so much undocumented complexity that if anything goes wrong, or goes "well" when it shouldn't, no one can explain why. Which also reintroduces a vector to hide nasty tricks.
this. the point of sharing reproducible steps and not the experiment itself is that it can be fully reproduced independently. not just independently verify that the result show what the paper claims.
Results that haven't been independently replicated are suspect. There are just too many factors that can lead an experiment to give some results that are not transferrable or not relevant.
The worst aspect of this is the lack of will or funding to replicate, replicate, and replicate again all significant results that get published. Post-processed data can be altered, but a TB of raw data is meaningless as well if it hasn't been produced properly, has been obfuscated, or is weirdly formatted.
Data availability is a red herring for the vast majority of the science being made right now (almost everything that does not depend on a multi-millions dollars experiment). If data availability is an end in itself, we would just have moved the goalposts and have a data quality problem instead of a reproducibility problem.
> Perhaps depositing a VM with everything ready to run.
Yes, this is hugely important. We need clearer requirements for what constitutes properly 'published' data, code and methods. It should include all raw data (both used and discarded) as well as complete snapshots of each preliminary and intermediate processing step in a time-stamped chronology.
This is an area where the expertise and best practices of software development, documentation and source control could help inform standards and tooling which would dramatically improve the quality of scientific publishing and replication.
> Some well intentioned papers share code which is hard to run years afterwards.
Umm, not really? How many years are we talking about here? Because even Cobol is runnable. Sure some OS related quirks may need changing but getting the raw source is much more likely to expose a subtle flaw in the researcher's method than just the equations. For example, an incorrect negative check (to string vs < 0) or an off by one error.
Don't think COBOL; think Python 2.1 (not 2.2) with this specific version of PIL, and some of the code implemented as a CPython module (for speed) in C that only compiles properly with one specific EGCS fork that adds support for IA-64. Parts of the code are written in slightly-buggy inline IA-64 assembly, but it's okay because after a system call, that particular operating system makes sure those registers are zeroed each loop iteration, if Python's line-buffering its `print`s so that the system call gets consistently run.
Also, the Python script crashes on loading the data files unless there are two (but not more than two) spaces in its full file path. This is not documented anywhere.
Yeah. I can easily run FORTRAN code from my PhD supervisor's PhD supervisor, written way back in the 1980s, but I cannot run some of the Python scripts written by a post-doc 10 years ago. It's a mess of abandoned libraries and things that work only with some specific versions of some libraries, and compiled modules that rely on the dodgy behaviour of an old compiler. Perl seems to be better, if only because people used to rely much less on external libraries.
But properly running code is not the solution either. I can count on the fingers of one had the downloads of some of the codes we've published (easily compilable modern Fortran), and AFAIK nobody ever published anything using them. Having a multitude of codes available does not mean much if nobody runs them, assuming they can be compiled. And I would guarantee that none of the scientists who download these codes would be able to understand what they do in any detail.
Indeed, that's not proper science, too many moving parts. If it cannot be easily replicated by anyone at any time, it is just an experiment which would need more refinement to get published. No experiment should be accepted if it requires a Rube-Goldberg machine.
Some well intentioned papers share code which is hard to run years afterwards. It's sometimes much simpler to reimplement things that to get the code to run if they are described accurately.
Besides, some articles hide ugly things, nasty tricks and lies in the code, which make their results a lot less believable and valid. Being super upfront about models in terms of equations and pseudocode is important.
Of course, we should also have standards to make code reproducible. Perhaps depositing a VM with everything ready to run.