What Is a Replication Package? Shipping the Code and Data With the Paper

A complete package holds everything needed to regenerate the reported results from the original data. That means the data itself where it can be shared, or precise instructions for obtaining it where it cannot. It means the code performing every step from raw data to final numbers, including the cleaning and selection decisions that papers describe in a sentence and that frequently change the results. It means the computing environment, since code depends on particular versions of particular libraries and breaks silently when those change, which is why packages increasingly ship a container specifying the whole environment. It means documentation stating what to run and in what order. And it means a licence, since material with no stated terms is legally awkward to use whatever the author intended.

Several different things get called replication and confusing them causes trouble:

• •Reproducing, meaning obtaining the same numbers from the same data and code, which tests whether the reported analysis was performed correctly
• •Replicating, meaning obtaining a comparable result from new data, which tests whether the finding holds
• •Robustness, meaning whether the result survives reasonable alternative analytical choices
• •Reanalysis, where someone applies a different method to the same data and reaches a different conclusion
• •Verification by a journal before publication, which some now perform and most do not
• •Each answers a different question, and a study can pass one and fail another

Systematic attempts to reproduce published computational results have been sobering. Projects working through economics, political science and psychology papers have found that a substantial proportion cannot be reproduced from the supplied materials, with reasons including missing files, code that does not run, undocumented manual steps and results that simply differ. Journals that introduced mandatory verification before publication report finding errors in a large share of submissions, most of them consequential enough to require correction before acceptance. Notable individual cases include an influential economics result that changed substantially when a spreadsheet error was found by a graduate student who had requested the data. None of that indicates widespread dishonesty, since the errors found are overwhelmingly ordinary mistakes, and it does indicate that unverified computational results carry a meaningful error rate.

Even without running anything, a reader can assess how checkable a paper is and the signals are straightforward. A statement that data and code are available at a named repository with a permanent identifier is the strongest signal, and available on request is the weakest, since studies following up such statements find that the majority of requests go unanswered or are refused. Specific version numbers for software indicate the authors thought about reproducibility. A preregistration link indicates the analysis was fixed in advance. Supplementary material containing the full output rather than selected tables allows checking that the reported numbers match. None of this establishes that a result is correct, and all of it distinguishes work that could be checked from work that cannot, which is a meaningful difference when deciding how much weight to place on a single paper.

The obstacles are real and mostly not about willingness. Data frequently cannot be shared, because it is personal, commercially confidential or licensed from a provider who prohibits redistribution, which affects entire fields. Preparing a package that runs on someone else's machine takes substantial effort that is not rewarded by anything in the career structure. Code written for personal use is embarrassing to publish, which is a genuine deterrent that people rarely admit. Long-term maintenance is nobody's job, so packages rot as dependencies change. And archiving requires infrastructure, which some institutions and journals provide and others do not. The measures that have worked are structural, namely journals requiring and verifying packages, funders mandating data management plans, and the establishment of repositories that assign permanent identifiers.

A package holds the data, the code for every step including cleaning, the computing environment and documentation, without which a computational result cannot be checked. Reproducing from the same data and replicating with new data answer different questions. Journals that verify before publication find errors in a large share of submissions, nearly all of them ordinary mistakes rather than misconduct.