Guix: Quasiquote and G-expression

Preparing some materials for the First Workshop on Reproducible Software Environments for Research and High-Performance Computing, I am re-reading the G-Expressions section from the Guix manual. A must to read! Some times ago, I have read a blog post by Marius Bakke, it inspired me this blog post, thanks Marius.

Well, the best explanation is show me the code, isn't it? So let start guix repl and explore…

Before introducing G-expressions we need to have a clear idea about classic Scheme concepts quasiquote ( ` ) and unquote ( , ); as well as quote ( ' ). In Lisp-family language, “quoted” data remains unevaluated.

scheme@(guix-user)> (define x 42)
scheme@(guix-user)> x
$1 = 42
scheme@(guix-user)> (quote x)
$2 = x

This quote is so frequent that it has syntactic sugar ( ' ). Somehow, '(x y z) is short¹ for (list 'x 'y 'z).

scheme@(guix-user)> '(x y z)
$3 = (x y z)
scheme@(guix-user)> (list? '(x y z))
$4 = #t
scheme@(guix-user)> (list 'x 'y 'z)
$5 = (x y z)
scheme@(guix-user)> (equal? '(x y z) (list 'x 'y 'z))
$6 = #t

Everything on the list is a value – e.g., 'x or 'y or 'z; namely there are symbols. In short, a symbol looks like a variable name except that it starts with quote ( ' ) and it plays a role similar as strings; somehow symbols are a great way to represent “symbolic” information as data.

That’s said, we would like to be able to escape back inside of a quoted list and evaluate something. Thanks to quasiquote ( ` ) and unquote ( , ), it is possible.

scheme@(guix-user)> (define y 24)
scheme@(guix-user)> `(,x y z)
$7 = (42 y z)

Here, the unquoted expression is evaluated during the construction of the list, while the other remaining unevaluated. Another example:

scheme@(guix-user)> (define something 'cool)
scheme@(guix-user)> (define (tell-me) `(Guix is ,something))
scheme@(guix-user)> (tell-me)
$8 = (Guix is cool)
scheme@(guix-user)> (define something 'awesome)
scheme@(guix-user)> (tell-me)
$9 = (Guix is awesome)

And unquote evaluates everything, including procedures if any.

scheme@(guix-user)> `(Again ,(tell-me))
$10 = (Again (Guix is awesome))
scheme@(guix-user)> `(1 ,(+ 2 3) 4)
$11 = (1 5 4)

For instance, let build² the Fibonacci sequence.

scheme@(guix-user)> (define (fibo n)
                      (if (or (= 0 n) (= 1 n))
                          `(begin ,n)
                          (let ((f_1 (fibo (- n 1)))
                                (f_2 (fibo (- n 2))))
                            `(begin (+ ,f_1 ,f_2)))))
scheme@(guix-user)> ,pp (fibo 7)
$12 = (begin
  (+ (begin
       (+ (begin
            (+ (begin
                 (+ (begin
                      (+ (begin (+ (begin 1) (begin 0))) (begin 1)))
                    (begin (+ (begin 1) (begin 0)))))
               (begin
                 (+ (begin (+ (begin 1) (begin 0))) (begin 1)))))
          (begin
            (+ (begin
                 (+ (begin (+ (begin 1) (begin 0))) (begin 1)))
               (begin (+ (begin 1) (begin 0)))))))
     (begin
       (+ (begin
            (+ (begin
                 (+ (begin (+ (begin 1) (begin 0))) (begin 1)))
               (begin (+ (begin 1) (begin 0)))))
          (begin
            (+ (begin (+ (begin 1) (begin 0))) (begin 1)))))))

Here, nothing is evaluated. All is data and symbolically manipulated. The evaluation (computation) is then done with eval,

scheme@(guix-user)> (eval (fibo 7) (interaction-environment))
$13 = 13

So far, so good. What about G-expressions? Quoting dedicated section of Guix manual:

To describe a derivation and its build actions, one typically needs to embed build code inside host code. It boils down to manipulating build code as data, and the homoiconicity of Scheme — code has a direct representation as data — comes in handy for that. But we need more than the normal quasiquote mechanism in Scheme to construct build expressions.

Somehow, G-expressions simplifies the machinery for staging code. Compared to classic expression, it introduces both: context – e.g., the set of inputs associated with the expression – and the ability to serialize high-level objects – i.e., to replace a reference to a package object with its /gnu/store/ file name.

G-expressions consist of syntactic forms: gexp, ungexp – or simply: #~ and #$ – which are comparable to quasiquote and unquote, respectively. Other said, it allows to control the context of the evaluation.

Let make it concrete with the Fibonacci example. We need the modules (guix gexp), (guix derivations) and (guix store).

scheme@(guix-user)> (use-modules (guix gexp)
                                 (guix derivations)
                                 (guix store))

Now, instead of manipulating quasiquote ( ` ) and unquote ( , ), we are going to manipulate gexp ( #~ ) and ungexp ( #$ ). Let start with three helpers:

(define (number->name n)
  (string-append "Fibonacci-of-"
                 (number->string n)))

(define (number->gexp n)
  #~(begin
      (use-modules (ice-9 format))
      (call-with-output-file #$output
        (lambda (port)
          (format port "~d" #$n)))))

(define (store-item->number path)
  #~(begin
      (use-modules ((ice-9 textual-ports) #:select (get-string-all)))
      (string->number
       (call-with-input-file #$path
         get-string-all))))

Nothing special to say about number->name. What do the others do? number->gexp takes an interger number and returns a G-expression, such that, after evaluation, it will write this number to some file. What makes the machinery convenient is that #$output will be replaced – evaluated with adequate context – by a string containing the output reference to its /gnu/store/ file name. Can you guess what store-item->number does?

The core: computing the Fibonacci sequence using G-expressions,

(define (fibonacci n)
  (if (or (= 0 n) (= 1 n))
      (gexp->derivation
       (number->name n)
       (number->gexp n))
      (let* ((store (open-connection))
             (drv-1 (run-with-store store
                      (fibonacci (- n 1))))
             (drv-2 (run-with-store store
                      (fibonacci (- n 2))))
             (f_1 (store-item->number drv-1))
             (f_2 (store-item->number drv-2)))
        (gexp->derivation
         (number->name n)
         (number->gexp #~(+ #$f_1 #$f_2))))))

First, gexp->derivation returns a derivation with the name ( number->name ) that runs the gexp ( number->gexp ); somehow this gexp stores something to compute. Second, run-with-store runs a gexp in the context of the store – remember monad.

The procedure fibonacci takes an integer number and constructs some G-expressions controlling the context of evaluation. Let run it!

scheme@(guix-user)> ,run-in-store (fibonacci 7)
$14 = #<derivation /gnu/store/db97xy9d5icaa64n2n9l7q2v66npmm6c-Fibonacci-of-7.drv => /gnu/store/8b5g05g4z5r9f3ash53ppb5m1r7kksfj-Fibonacci-of-7 7f7ca8ed3640>

Awesome! We get back a derivation. Note the handy ,run-in-store command – Guix specific REPL command – which hides the plumbing of run-with-store. From the REPL, it is possible to explore this derivation, although the pretty-printer is not handy. Well, this derivation reads:

Derive
([("out","/gnu/store/8b5g05g4z5r9f3ash53ppb5m1r7kksfj-Fibonacci-of-7","","")]
 ,[("/gnu/store/0wkxvd2ll0gff37wghamb12dz4x50n14-Fibonacci-of-6.drv",["out"])
   ,("/gnu/store/9r95y1j1rg4q7vb528lh51w0cz3c5hvi-Fibonacci-of-5.drv",["out"])
   ,("/gnu/store/zraigp7miin3vzr5dcbr4i9rvds0i07r-guile-3.0.9.drv",["out"])]
 ,["/gnu/store/z0jspla9advx77ihbc7nfjvnky2gfvjz-Fibonacci-of-7-builder"]
 ,"x86_64-linux","/gnu/store/g8p09w6r78hhkl2rv1747pcp9zbk6fxv-guile-3.0.9/bin/guile",["--no-auto-compile","/gnu/store/z0jspla9advx77ihbc7nfjvnky2gfvjz-Fibonacci-of-7-builder"]
 ,[("out","/gnu/store/8b5g05g4z5r9f3ash53ppb5m1r7kksfj-Fibonacci-of-7")])

And the 7th term depends on the 6th and 5th (\(F_{7} = F_{6} + F_{5}\)); the expected recursive sequence. The interesting part is the builder,

(begin
  (use-modules
   (ice-9 format))
  (call-with-output-file
      ((@
        (guile)
        getenv)
       "out")
    (lambda
        (port)
      (format port "~d"
              (+
               (begin
                 (use-modules
                  ((ice-9 textual-ports)
                   #:select
                   (get-string-all)))
                 (string->number
                  (call-with-input-file "/gnu/store/rhjmlgaz4f1niwhrnm2nsfdj2g6dya6h-Fibonacci-of-6" get-string-all)))
               (begin
                 (use-modules
                  ((ice-9 textual-ports)
                   #:select
                   (get-string-all)))
                 (string->number
                  (call-with-input-file "/gnu/store/xcp9b4j0nskk6lk5jxlpv3926j19vpw0-Fibonacci-of-5" get-string-all))))))))

All had been correctly replaced. Somehow, that builder script is similar as the output of the previous fibo procedure, and instead of eval, now the computation will be done by Guix daemon evaluating this builder script.

Let make that computation!

scheme@(guix-user)> ,build $14
$15 = "/gnu/store/8b5g05g4z5r9f3ash53ppb5m1r7kksfj-Fibonacci-of-7"

And guess what? This file contains the value 13. 👍

As you can see, all the previous values of the Fibonacci sequence are also computed via derivations and the result stored as files. Guix daemon starts to construct the derivation Fibonacci-of-0.drv, then Fibonacci-of-1.drv, and compute them by writing 0 then 1 inside the output store item files Fibonacci-of-0 and Fibonacci-of-1. Then Guix daemon evaluates the builder script of the derivation Fibonacci-of-2.drv, i.e., it reads the values from two previous store item files, adds them and writes the result inside the store item file Fibonacci-of-2. The Guix daemon repeats until 7. Other said, if we want to compute the value for the 8th Fibonacci number, all the previous computations are cached in the Guix store; the Guix store acts as a good memoization mean. Check it with:

$ guix gc --list-dead | grep Fibonacci-of

Willing to go further, give a look to the Code Staging in GNU Guix paper.

Join the fun, join Guix!