nixpkgs/doc/languages-frameworks/lisp.section.md

lisp-modules

This document describes the Nixpkgs infrastructure for building Common Lisp libraries that use ASDF (Another System Definition Facility). It lives in pkgs/development/lisp-modules.

Overview

The main entry point of the API are the Common Lisp implementation packages (e.g. abcl, ccl, clasp-common-lisp, clisp ecl, sbcl) themselves. They have the pkgs and withPackages attributes, which can be used to discover available packages and to build wrappers, respectively.

The pkgs attribute set contains packages that were automatically imported from Quicklisp, and any other manually defined ones. Not every package works for all the CL implementations (e.g. nyxt only makes sense for sbcl).

The withPackages function is of primary utility. It is used to build runnable wrappers, with a pinned and pre-built ASDF FASL available in the ASDF environment variable, and CL_SOURCE_REGISTRY/ASDF_OUTPUT_TRANSLATIONS configured to find the desired systems on runtime.

With a few exceptions, the primary thing that the infrastructure does is to run asdf:load-system for each system specified in the systems argument to build-asdf-system, and save the FASLs to the Nix store. Then, it makes these FASLs available to wrappers. Any other use-cases, such as producing SBCL executables with sb-ext:save-lisp-and-die, are achieved via overriding the buildPhase etc.

In addition, Lisps have the packageOverrides argument, which can be used (through override) to substitute any package in the scope of their pkgs. This will be useful together with overrideLispAttrs when dealing with slashy ASDF systems, because they should stay in the main package and be build by specifying the systems argument to build-asdf-system.

The 90% use case example

The most common way to use the library is to run ad-hoc wrappers like this:

nix-shell -p 'sbcl.withPackages (ps: with ps; [ alexandria ])'

Then, in a shell:

$ result/bin/sbcl
* (load (sb-ext:posix-getenv "ASDF"))
* (asdf:load-system 'alexandria)

Also one can create a pkgs.mkShell environment in shell.nix/flake.nix:

let
  sbcl' = sbcl.withPackages (ps: [ ps.alexandria ]);
in mkShell {
  buildInputs = [ sbcl' ];
}

Such a Lisp can be now used e.g. to compile your sources:

buildPhase = ''
  ${sbcl'}/bin/sbcl --load my-build-file.lisp
''

Importing packages from Quicklisp

The library is able to very quickly import all the packages distributed by Quicklisp by parsing its releases.txt and systems.txt files. These files are available from [http://beta.quicklisp.org/dist/quicklisp.txt].

The import process is implemented in the import directory as Common Lisp functions in the org.lispbuilds.nix ASDF system. To run the script, one can execute ql-import.lisp:

nix-shell --run 'sbcl --script ql-import.lisp'

The script will:

1. Download the latest Quicklisp systems.txt and releases.txt files
2. Generate an SQLite database of all QL systems in packages.sqlite
3. Generate an imported.nix file from the database

The maintainer's job there is to:

1. Re-run the ql-import.lisp script
2. Add missing native dependencies in ql.nix
3. For packages that still don't build, package them manually in packages.nix

Also, the imported.nix file must not be edited manually! It should only be generated as described in this section.

Adding native dependencies

The Quicklisp files contain ASDF dependency data, but don't include native library (CFFI) dependencies, and, in the case of ABCL, Java dependencies.

The ql.nix file contains a long list of overrides, where these dependencies can be added.

Packages defined in packages.nix contain these dependencies naturally.

Trusting systems.txt and releases.txt

The previous implementation of lisp-modules didn't fully trust the Quicklisp data, because there were times where the dependencies specified were not complete, and caused broken builds. It instead used a nix-shell environment to discover real dependencies by using the ASDF APIs.

The current implementation has chosen to trust this data, because it's faster to parse a text file than to build each system to generate its Nix file, and because that way packages can be mass-imported. Because of that, there may come a day where some packages will break, due to bugs in Quicklisp. In that case, the fix could be a manual override in packages.nix and ql.nix.

A known fact is that Quicklisp doesn't include dependencies on slashy systems in its data. This is an example of a situation where such fixes were used, e.g. to replace the systems attribute of the affected packages. (See the definition of iolib).

Quirks

During Quicklisp import:

• + in names are converted to _plus{_,}: cl+ssl->cl_plus_ssl, alexandria+->alexandria_plus
• . to _dot_: iolib.base->iolib_dot_base
• names starting with a number have a _ prepended (3d-vectors->_3d-vectors)
• _ in names is converted to __ for reversibility

Defining packages manually inside Nixpkgs

New packages, that for some reason are not in Quicklisp, and so cannot be auto-imported, can be written in the packages.nix file.

In that file, use the build-asdf-system function, which is a wrapper around mkDerivation for building ASDF systems. Various other hacks are present, such as build-with-compile-into-pwd for systems which create files during compilation.

The build-asdf-system function is documented with comments in nix-cl.nix. Also, packages.nix is full of examples of how to use it.

Defining packages manually outside Nixpkgs

Lisp derivations (abcl, sbcl etc.) also export the buildASDFSystem function, which is the same as build-asdf-system, except for the lisp argument which is set to the given CL implementation.

It can be used to define packages outside Nixpkgs, and, for example, add them into the package scope with override and packageOverrides, both of which will be discussed later on.

Including an external package in scope

A package defined outside Nixpkgs using buildASDFSystem can be woven into the Nixpkgs-provided scope like this:

let
  alexandria = sbcl.buildASDFSystem rec {
    pname = "alexandria";
    version = "1.4";
    src = fetchFromGitLab {
      domain = "gitlab.common-lisp.net";
      owner = "alexandria";
      repo = "alexandria";
      rev = "v${version}";
      hash = "sha256-1Hzxt65dZvgOFIljjjlSGgKYkj+YBLwJCACi5DZsKmQ=";
    };
  };
  sbcl' = sbcl.override {
    packageOverrides = self: super: {
      inherit alexandria;
    };
  };
in sbcl'.pkgs.alexandria

Overriding package attributes

Packages export the overrideLispAttrs function, which can be used to build a new package with different parameters.

Example of overriding alexandria:

sbcl.pkgs.alexandria.overrideLispAttrs (oldAttrs: rec {
  version = "1.4";
  src = fetchFromGitLab {
    domain = "gitlab.common-lisp.net";
    owner = "alexandria";
    repo = "alexandria";
    rev = "v${version}";
    hash = "sha256-1Hzxt65dZvgOFIljjjlSGgKYkj+YBLwJCACi5DZsKmQ=";
  };
})

Overriding packages in scope

Packages can be woven into a new scope by using override with packageOverrides:

let
  sbcl' = sbcl.override {
    packageOverrides = self: super: {
      alexandria = super.alexandria.overrideLispAttrs (oldAttrs: rec {
        pname = "alexandria";
        version = "1.4";
        src = fetchFromGitLab {
          domain = "gitlab.common-lisp.net";
          owner = "alexandria";
          repo = "alexandria";
          rev = "v${version}";
          hash = "sha256-1Hzxt65dZvgOFIljjjlSGgKYkj+YBLwJCACi5DZsKmQ=";
        };
      });
    };
  };
in builtins.elemAt sbcl'.pkgs.bordeaux-threads.lispLibs 0

Dealing with slashy systems

Slashy (secondary) systems should not exist in their own packages! Instead, they should be included in the parent package as an extra entry in the systems argument to the build-asdf-system/buildASDFSystem functions.

The reason is that ASDF searches for a secondary system in the .asd of the parent package. Thus, having them separate would cause either one of them not to load cleanly, because one will contains FASLs of itself but not the other, and vice versa.

To package slashy systems, use overrideLispAttrs, like so:

ecl.pkgs.alexandria.overrideLispAttrs (oldAttrs: {
  systems = oldAttrs.systems ++ [ "alexandria/tests" ];
  lispLibs = oldAttrs.lispLibs ++ [ ecl.pkgs.rt ];
})

See the respective section for how to weave it back into ecl.pkgs.

Note that sometimes the slashy systems might not only have more dependencies than the main one, but create a circular dependency between .asd files. Unfortunately, in this case an adhoc solution becomes necessary.

Building Wrappers

Wrappers can be built using the withPackages function of Common Lisp implementations (abcl, ecl, sbcl etc.):

sbcl.withPackages (ps: [ ps.alexandria ps.bordeaux-threads ])

Such a wrapper can then be executed like this:

result/bin/sbcl

Loading ASDF

For best results, avoid calling (require 'asdf) When using the library-generated wrappers.

Use (load (ext:getenv "ASDF")) instead, supplying your implementation's way of getting an environment variable for ext:getenv. This will load the (pre-compiled to FASL) Nixpkgs-provided version of ASDF.

Loading systems

There, you can simply use asdf:load-system. This works by setting the right values for the CL_SOURCE_REGISTRY/ASDF_OUTPUT_TRANSLATIONS environment variables, so that systems are found in the Nix store and pre-compiled FASLs are loaded.

Adding a new Lisp

Three additional functions are exposed, and are meant for wrapping Common Lisp derivations: commonLispPackagesFor, lispWithPackages andbuild-asdf-system.

commonLispPackagesFor returns a package set for the provided Lisp "spec". Such a spec is an attribute set of the following keys:

• pkg: the Lisp package derivation
• program: The name of executable file in ${pkg}/bin/
• flags: A list of flags to always pass to program
• faslExt: Implementation-specific extension for FASL files
• asdf: The ASDF version to use

The spec is an argument to every Lisp, and can be customized via override:

sbcl.override {
  spec = {
    pkg = pkgs.sbcl_2_1_1;
    flags = [ "--dynamic-space-size" "4096" ];
    faslExt = "fasl";
    asdf = pkgs.asdf_3_1;
  };
}

lispWithPackages returns a function to create wrappers.

build-asdf-system is the wrapper around stdenv.mkDerivation.

To wrap a new Lisp, include the following in its passthru:

passthru = let
    spec' = spec // { pkg = sbcl; };
    pkgs = (commonLispPackagesFor spec').overrideScope' packageOverrides;
in {
  inherit pkgs;
  withPackages = lispWithPackages pkgs;
  buildASDFSystem = args: build-asdf-system (args // spec');
}