Buildapp - Create executables with SBCL

Buildapp is an application for SBCL that configures and saves an executable Common Lisp image. It is similar to cl-launch and hu.dwim.build. Buildapp is available under a BSD-style license. The latest version is 1.1, released on February 12th, 2010.

Download shortcut: http://www.xach.com/lisp/buildapp.tgz

Installation
Example Use
Overview
Limitations
Implementation
Feedback
License

Installation

Buildapp does not require any libraries. To create and install the application, use make install. By default, it is installed in /usr/local/bin; to use another location, use make DESTDIR=/path install.

You can also create the buildapp binary by loading the buildapp system with asdf and running (buildapp:build-buildapp).

Example Use

Here's a small application:

$ buildapp \
    --eval '(defun main (argv) (declare (ignore argv)) (write-line "Hello, world"))' \
    --entry main \
    --output hello-world
[undoing binding stack and other enclosing state... done]
[saving current Lisp image into hello-world:
writing 6352 bytes from the read-only space at 0x20000000
writing 4064 bytes from the static space at 0x20100000
writing 44834816 bytes from the dynamic space at 0x1000000000
done]

$ ./hello-world
Hello, world

The following creates a toy curl-like application. (It's not quite practical, because any errors will land you in the interactive debugger.)

$ buildapp --output lisp-curl --asdf-path ~/src/clbuild/systems/ \
    --load-system drakma \
    --eval '(defun main (args) (write-string (drakma:http-request (second args))))' \
    --entry main
;; loading system sb-grovel (needed by drakma)
;;  from /usr/local/lib/sbcl/sb-grovel/
;; loading system sb-posix (needed by cl+ssl)
;;  from /usr/local/lib/sbcl/sb-posix/
;; loading system trivial-gray-streams (needed by chunga, cl+ssl, flexi-streams)
;;  from /home/xach/src/clbuild/source/trivial-gray-streams/
;; loading system flexi-streams (needed by drakma, cl+ssl)
;;  from /home/xach/src/clbuild/source/flexi-streams/
;; loading system alexandria (needed by cffi, babel)
;;  from /home/xach/src/clbuild/source/alexandria/
;; loading system trivial-features (needed by cffi, babel)
;;  from /home/xach/src/clbuild/source/trivial-features/
;; loading system babel (needed by cffi)
;;  from /home/xach/src/clbuild/source/babel/
;; loading system cffi (needed by cl+ssl)
;;  from /home/xach/src/clbuild/source/cffi/
;; loading system cl+ssl (needed by drakma)
;;  from /home/xach/src/clbuild/source/cl+ssl/
;; loading system sb-bsd-sockets (needed by usocket)
;;  from /usr/local/lib/sbcl/sb-bsd-sockets/
;; loading system usocket (needed by drakma)
;;  from /home/xach/src/clbuild/source/usocket/
;; loading system chunga (needed by drakma)
;;  from /home/xach/src/clbuild/source/chunga/
;; loading system cl-base64 (needed by drakma)
;;  from /home/xach/src/clbuild/source/cl-base64/
;; loading system puri (needed by drakma)
;;  from /home/xach/src/clbuild/source/puri/
;; loading system drakma 
;;  from /home/xach/src/clbuild/source/drakma/
[undoing binding stack and other enclosing state... done]
[saving current Lisp image into lisp-curl:
writing 6352 bytes from the read-only space at 0x20000000
writing 5472 bytes from the static space at 0x20100000
writing 61722624 bytes from the dynamic space at 0x1000000000
done]

$ ./lisp-curl http://xach.com/
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"
       "http://www.w3.org/TR/REC-html40/loose.dtd">
<HTML>
<HEAD>
<TITLE>www.xach.com</TITLE>
</HEAD>
...

Here's how the l1sp.org redirection service application is built:

$ make
buildapp --output l1sp --entry redirector:main \
                 --asdf-path /opt/l1sp/systems \
                 --require sb-aclrepl \
                 --eval '(pushnew :hunchentoot-no-ssl *features*)' \
                 --load-system swank \
                 --eval '(setf swank:*log-output* nil)' \
                 --load-system redirector
;; loading system sb-grovel
;;  from /usr/local/lib/sbcl/sb-grovel/
;; loading system sb-bsd-sockets
;;  from /usr/local/lib/sbcl/sb-bsd-sockets/
;; loading system sb-introspect
;;  from /usr/local/lib/sbcl/sb-introspect/
;; loading system sb-posix
;;  from /usr/local/lib/sbcl/sb-posix/
;; loading system sb-cltl2
;;  from /usr/local/lib/sbcl/sb-cltl2/
;; loading system swank
;;  from /opt/l1sp/src/slime/
;; loading system html-template (needed by redirector)
;;  from /opt/l1sp/src/html-template-0.9.1/
;; loading system sb-rotate-byte (needed by sb-md5)
;;  from /usr/local/lib/sbcl/sb-rotate-byte/
;; loading system sb-md5 (needed by redirector)
;;  from /usr/local/lib/sbcl/sb-md5/
;; loading system cl-who (needed by redirector)
;;  from /opt/l1sp/src/cl-who-0.11.1/
;; loading system cl-ppcre (needed by hunchentoot, redirector)
;;  from /opt/l1sp/src/cl-ppcre-2.0.0/
;; loading system url-rewrite (needed by hunchentoot)
;;  from /opt/l1sp/src/url-rewrite-0.1.1/
;; loading system rfc2388 (needed by hunchentoot)
;;  from /opt/l1sp/src/rfc2388/
;; loading system md5 (needed by hunchentoot)
;;  from /opt/l1sp/src/md5-1.8.5/
;; loading system cl-fad (needed by hunchentoot)
;;  from /opt/l1sp/src/cl-fad-0.6.2/
;; loading system cl-base64 (needed by hunchentoot)
;;  from /opt/l1sp/src/cl-base64-3.3.2/
;; loading system trivial-gray-streams (needed by flexi-streams)
;;  from /opt/l1sp/src/trivial-gray-streams-2006-09-16/
;; loading system flexi-streams (needed by chunga)
;;  from /opt/l1sp/src/flexi-streams-1.0.7/
;; loading system chunga (needed by hunchentoot)
;;  from /opt/l1sp/src/chunga-0.4.3/
;; loading system hunchentoot (needed by redirector)
;;  from /opt/l1sp/src/hunchentoot-0.15.7/
;; loading system redirector
;;  from /opt/l1sp/src/redirector/
[undoing binding stack and other enclosing state... done]
[saving current Lisp image into l1sp:
writing 6176 bytes from the read-only space at 0x20000000
writing 4064 bytes from the static space at 0x20100000
writing 61042688 bytes from the dynamic space at 0x1000000000
done]

$ ./l1sp
;; Swank started at port: 7717.
CL-USER(1):

The (setf swank:*log-output* nil) eval is needed to avoid problems when the image restarts.

redirector::main looks like this:

(defun main (argv)
  (declare (ignore argv))
  (load "/opt/l1sp/etc/init.lisp")
  (sb-impl::toplevel-repl nil))

Here's an example of the --dispatched-entry option, which was inspired by the desire to have a dozen different small utilities embedded in one big executable and called based on the binary name. First, the support files:

;;;; utils.lisp

(defpackage #:utils
  (:use #:cl))

(in-package #:utils)

(defun main (argv)
  (let ((name (pathname-name (first argv))))
    (format *error-output*
            "Unknown binary name ~S, try using cl-echo, cl-ls, or cl-true~%"
            name)
    (sb-ext:quit :unix-status 1)))

;;;; ls.lisp

(defpackage #:ls
  (:use #:cl))

(in-package #:ls)

(defun main (argv)
  (declare (ignore argv))
  (dolist (file (directory "*.*"))
    (write-line (namestring file))))

;;;; echo.lisp

(defpackage #:echo
  (:use #:cl))

(in-package #:echo)

(defun main (argv)
  (format t "~{~A~^ ~}~%" (rest argv)))

;;;; true.lisp

(defpackage #:true
  (:use #:cl))

(in-package #:true)

(defun main (argv)
  (declare (ignore argv))
  (sb-ext:quit :unix-status 0))

Buliding it all together looks like this:

$ buildapp --output utils \
   --load utils.lisp --dispatched-entry /utils:main \
   --load ls.lisp --dispatched-entry cl-ls/ls:main \
   --load echo.lisp --dispatched-entry cl-echo/echo:main \
   --load true.lisp --dispatched-entry cl-true/true:main
;; loading file #P"/tmp/demo/utils.lisp"
;; loading file #P"/tmp/demo/ls.lisp"
;; loading file #P"/tmp/demo/echo.lisp"
;; loading file #P"/tmp/demo/true.lisp"
[undoing binding stack and other enclosing state... done]
[saving current Lisp image into utils:
writing 6352 bytes from the read-only space at 0x20000000
writing 4064 bytes from the static space at 0x20100000
writing 45223936 bytes from the dynamic space at 0x1000000000
done]

$ ln -sf utils cl-ls

$ ln -sf utils cl-echo

$ ln -sf utils cl-true

$ ./cl-ls
/tmp/demo/cl-true
/tmp/demo/echo.lisp
/tmp/demo/ls.lisp
/tmp/demo/true.lisp
/tmp/demo/utils
/tmp/demo/utils.lisp

$ ./cl-echo Hello world
Hello world

$ ./cl-true && echo $?
0

Overview

Here is the usage output of buildapp:

Usage: buildapp --output OUTPUT-FILE [--flag1 value1 ...]

Required flags:
  --output OUTPUT-FILE      Use OUTPUT-FILE as the name of the executable
                              to create

Entry-point flags:
  --entry NAME              Use the function identified by NAME as the
                              executable's toplevel function. Called
                              with SB-EXT:*POSIX-ARGV* as its only
                              argument. If NAME has a colon, it is
                              treated as a package separator,
                              otherwise CL-USER is the implied
                              package.
  --dispatched-entry DNAME  Specify one possible entry function, depending
                              on the name of the file that is used to
                              start the application. The syntax of
                              DNAME is APPLICATION-NAME/ENTRY-NAME. If the
                              name used to start the executable matches
                              APPLICATION-NAME, use ENTRY-NAME as the
                              entry point. This can be used to choose
                              one of many possible entry points by
                              e.g. symlinking names to the application
                              executable. If APPLICATION-NAME is empty, the
                              specified ENTRY-NAME is used as a default
                              if no other application names match. There
                              may be any number of dispatched entry points,
                              but only one default.

Action flags:
  --load FILE               Load FILE. CL:*PACKAGE* is bound to the CL-USER
                              package before loading
  --load-system NAME        Load an ASDF system identified by NAME
  --require NAME            Use CL:REQUIRE to load NAME
  --eval CODE               Use CL:EVAL to evaulate CODE. The code is read
                              with CL:READ-FROM-STRING in the CL-USER package

There may be any number of load/load-system/require/eval flags. Each
is executed in command-line order before creating an executable.

Load path flags:
  --load-path DIRECTORY     When handling a --load, search DIRECTORY for
                              files to load
  --asdf-path DIRECTORY     When handling a --load-system, search DIRECTORY
                              for ASDF system files to load
  --asdf-tree DIRECTORY     When handling a --load-system, search DIRECTORY
                              and all its subdirectories for ASDF system
                              files to load

There may be any number of load-path/asdf-path/asdf-tree
flags. asdf-path arguments take precedence over asdf-tree arguments.

Other flags:
  --help                    Show this usage message
  --logfile FILE            Log compilation and load output to FILE
  --sbcl PATH-TO-SBCL       Use PATH-TO-SBCL instead of the sbcl program
                              found in your PATH environment variable

For the latest documentation, see http://www.xach.com/lisp/buildapp/

Limitations

Buildapp is limited in scope. It aims to make the following steps easy:

Define an application environment by loading files, loading ASDF systems, evaluating code, and using CL:REQUIRE
Dump an executable image with an arbitrary startup function

By design, it does not handle the following tasks:

Initialization tasks at app startup (use SB-EXT:*INIT-HOOKS* for that)
Debugger management (use SB-EXT:*INVOKE-DEBUGGER-HOOK* for that)
Command-line processing (use a library such as COMMAND-LINE-ARGUMENTS for that)
Obtaining libraries (use clbuild, LibCL, cl-librarian, asdf-install, etc. for that)

Implementation

Buildapp works like this:

It processes the command-line and creates an object that captures the command-line requirements: the output file, any eval/load/load-system/require actions, the entry function, etc.
It creates a new Lisp file (the dumpfile) with all the commands needed to implement the command-line options.
It runs "sbcl" with run-program. sbcl is invoked with no init files (it doesn't read user or system sbcl rc files) and loads the dumpfile:
1. The first few commands of the dumpfile establish a specialized loading environment:
  - The debugger is changed with *invoke-debugger-hook* to simply quit with a special exit code instead of entering the normal debugger
  - Most output output (ASDF system compilation output, low-level error messages, etc) is redirected to a log stream; that stream can be directed to a file with the --logfile argument
  - Stale FASLs are automatically recompiled with an :around method on asdf:perform
2. There are some sanity checks: Is the output file writable? Does this version of sbcl support the required :save-runtime-options argument?
3. The dumpfile performs the eval/load/load-system/require actions.
4. The dumpfile clears itself out of the environment:
  - Remove extra ASDF methods with remove-method
  - Reset sb-ext:*invoke-debugger-hook* to NIL
  - Delete the dumpfile package with delete-package
5. The dumpfile then creates an executable with save-lisp-and-die. This ends the sbcl subprocess.
It deletes the dumpfile.

There are two things to keep in mind to avoid conflicts with buildapp's implementation strategy:

sb-ext:*invoke-debugger-hook* is set to NIL before saving the image, so any changes to the variable must be done at application startup time (via your --entry function or sb-ext:*init-hooks*) instead of application load time
The ASDF central registry is temporarily extended with the --asdf-path and --asdf-tree arguments at load time, and reverts back to the default central registry value after that; changes to the central registry should be done at startup time instead of application load time

Feedback

If you have any questions or comments about buildapp, please email me, Zach Beane.

License

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR 'AS IS' AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.