Which problems Scientist has that Guix solves

Let draw a general picture separating what it concretely means a scientific result. Despite it is an obvious approximation of the reality, it helps to catch which parts are actionable when speaking about doing a scientific result.

The table reads: to do the result, it requires to have at hand all the three, the paper describing it, and, the associated data, and, using this analysis. Another instance, to do the analysis, it requires to have at hand all the three, the script describing it, and, the associated data, and, using this environment.

For each column, the situation is at best:

• audit is the “easy” part – people describe with more or less accuracy what they are doing,
• opaque is the hard part – some events happens every once in a while (astronomy or genetic mutation), some events are highly noisy (cells in Petri dish), some instruments are protected by patents (microscope, DNA sequencer), or even some instruments are too expensive to have two (LHC, supercomputer), etc.
• depend? is the column where we can act on! This column only depends on the practise we collectively agree on. Therefore, I am advocating to evacuate that as most as possible by turning it as audit.

The boundary between data (experiment) and analysis (numerical processing) is not clearly delimited and instead it is often a grey area. How to consider trained machine learning models? Especially if they run on GPU². Is it data or analysis? How to consider meteorological computations run after several days (week?) on supercomputer? Is it data or analysis? It is easy to find many more examples.

The true question, when speaking about producing a scientific result, is the control of the variability³. Is the variability intrinsic to the object under study? Or is it extrinsic? In this former case, what is the frame to control it? The aim in setting such frame is twofold:

• a) verify and thus convince other peers that the arguments hold and,
• b) do not build on sand, i.e., deeply understand, e.g., from where the resulting effects come.

These two items strongly require the capacity to scrutinize all the components: paper, protocol, script and more, data, analysis and… environment. That’s what Open Article, Open Data, Open Data cover for the former. For the latter, how to deal with computational environment in this scientific context?

A consequence of these two items is the capacity to redo previous results. Being able to redo what had been done is at the heart of Science and, somehow, this redo and its introspection is the core principle which turns a belief into a scientific knowledge. Let define weakly reproducibility by being able to redo what had been done. Alice runs thing at time \(t\) and Bob tries to run again thing at any another time \(t'\) possibly elsewhere. The correctness of thing and then considering if redo-thing is identical or not is the very aim of any scientific debates⁴. Science makes progress by arguing, thus how to argue if it is not possible to complete redo? Is it a sane discussion if computational environment is not controlled?

Still not convinced that computational environment matters for Science? Read on.

Any scientific practitioner knows that producing scientific knowledge is a collective activity. Many people or even teams are involved over the time. Concretely, these situations are what happens daily in the labs all around the world:

• Alice uses numerical tools R@4.1.1, FlowCore@2.4 and CATALYST@1.16.2;
• Carole collaborates on the project with Alice… but in the same time, Carole works on another project requiring the numerical tools R@4.0.4, FlowCore@2.0;
• Bob runs the same versions as Alice… but does not get the same outputs;
• Charlie upgrades their system then all is broken, reinstall everything… but is never able to restore the environment which worked;
• Dan tries to redo the Alice’s analysis months (year?) later… but hits the dependency hell⁵.

(Replace the tools R, FlowCore, CATALYST by any other tools from any scientific stack. Here, R is picked as example because it is very (too much?) popular in data processing from biological context.)

Any person who has seriously invested effort in numerical processing has hit at least one similar situation as Carole or Bob or Charlie or Dan. If not, please drop me an email exposing your workflow and habits.

All the situations have pragmatic solutions,

• package manager as apt or yum or etc.
• virtual environment as conda or modulefiles or etc.
• container as Docker or Singularity or etc.

and, for what my experience is worth, I have daily dealt with many of these solutions then they fail at one corner or the other – when speaking about computational reproducibility over the time. Their main drawback is that they do not tackle by design both the core issues of capturing the computational environment and being able to provide the same binaries over the time. Guix does both by design⁶.

Let dive into forensic exercise. First thing first, run a software means that somewhere a binary program must be involved. Is this R a binary program? The command which allows to know which software is used exactly – where the real software is located in the system. The command file determines the file type, i.e., if the file is a binary or text.

alice@laptop$ which R
/usr/bin/R
alice@laptop$ file /usr/bin/R
/usr/bin/R: Bourne-Again shell script, ASCII text executable

Interesting, typing R runs a script (text executable). After studying it a bit this script, we know two things:

1. at the top, it reads #!/bin/bash,
2. this /usr/lib/R/bin/exec/R is indeed run.

Typing R implies two binaries: the one (/bin/bash) to launch the script named /usr/bin/R and this script calls the other binary (/usr/lib/R/bin/exec/R). For the sake of the presentation, we let aside the former. However, keep in mind that all what we will say to the latter applies to both. Therefore, perhaps a first clue why it is complicated⁸.

alice@laptop$ file /bin/bash
/bin/bash: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=12f73d7a8e226c663034529c8dd20efec22dde54, stripped
alice@laptop$ file /usr/lib/R/bin/exec/R
/usr/lib/R/bin/exec/R: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=d875b0693e51ba0b8ac28ea332eb69b5ca6071af, stripped

Wow, what does ELF and dynamically linked mean?

…a mythological creature – ELF is for Executable and Linkable Format. In short, it is the binary we are looking for. Wait, we told that R is from a C program. How do we go from this human readable C program to that binary.

Recipe to make Yogurt: take some Milk and mix to some Skyr⁹ (other kind of Yogurt). This analogy applies to make binary and reads,

It is easy to imagine that the C compiler is an executable too. It comes from an human-readable source code and another executable. This other executable is also from an human-readable source code and another-bis executable. Recursion, oh recursion! At the end, a chicken or the egg problem arises (named bootstrapping compiler). Curious reader might be interested by this entry point.

It is a bit oversimplified since the file main.c alone is obviously not enough to produce all the features that R provides. Instead, all the source files are first transformed as binary object and then second linked altogether to produce the resulting unique binary.

In summary, the key component to keep in mind is that building a binary program requires one or more other binary program. It is often named build-time dependencies.

Let examine /usr/lib/R/bin/exec/R by using the command-line tool ldd which prints the shared object dependencies.

alice@laptop$ ldd /usr/lib/R/bin/exec/R
   libblas.so.3 => /usr/lib/x86_64-linux-gnu/libblas.so.3
   libR.so => /usr/lib/libR.so
   libgfortran.so.5 => /usr/lib/x86_64-linux-gnu/libgfortran.so.5
   libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0
   libblas.so.3 => /usr/lib/x86_64-linux-gnu/libblas.so.3
   libR.so => /usr/lib/libR.so
   libgfortran.so.5 => /usr/lib/x86_64-linux-gnu/libgfortran.so.5
   libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0
   ... many more ...

Euh, what does it mean? It means that the binary /usr/lib/R/bin/exec/R is dynamically linked to all these software. And wait for it, there are binaries too. All we said in the previous subsection apply for each item. We are explaining why it is complicated, so let slash at that!

Let pick one: BLAS provides all the basic linear algebra subprograms. The command ls -l allows to list all the properties of a file. On Ubuntu 18.04, it reads,

alice@laptop$ ls -l /usr/lib/x86_64-linux-gnu/libblas.so.3
lrwxrwxrwx 1 root root 47 avril 21 11:44 /usr/lib/x86_64-linux-gnu/libblas.so.3 -> /etc/alternatives/libblas.so.3-x86_64-linux-gnu

alice@laptop$ ls -l  /etc/alternatives/libblas.so.3-x86_64-linux-gnu
lrwxrwxrwx 1 root root 43 avril 21 11:44 /etc/alternatives/libblas.so.3-x86_64-linux-gnu -> /usr/lib/x86_64-linux-gnu/blas/libblas.so.3

alice@laptop$ ls -l /usr/lib/x86_64-linux-gnu/blas/libblas.so.3
lrwxrwxrwx 1 root root 16 nov.  28  2017 /usr/lib/x86_64-linux-gnu/blas/libblas.so.3 -> libblas.so.3.7.1

alice@laptop$ file /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
/usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=ca15f6c1fdcbd9438bb94298f38bf7fa0e99bc5f, stripped

The file /usr/lib/x86_64-linux-gnu/libblas.so.3 is a symbolic file which points to the file /etc/alternatives/libblas.so.3-x86_64-linux-gnu which is another symbolic link /usr/lib/x86_64-linux-gnu/blas/libblas.so.3 which points to the binary /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1.

In summary, the key component to keep in mind is that running a binary program requires one or more other binary programs. It is often named run-time dependencies.

We got it, it is complicated. The solution is to capture the graph of dependencies and package managers (apt, yum, conda, brew, etc.) does already that job. Indeed, but is it that simple?