Common Lisp: building a standalone binary issues and solutions

lisp standalone binary issues.md

update: more details in this blog post: https://lisp-journey.gitlab.io/blog/lisp-for-the-web-build-standalone-binaries-foreign-libraries-templates-static-assets/

How to build a standalone CL binary and make it run on another machine.

• use Deploy
• restrict ASDF
• pin all your dependencies (Mito along with dbd-sqlite3 for example).

but also

• compile your Djula templates in memory at build time (see Djula doc / future blog post)

Failed to find the WRITE-DATE of /builds/myopenbookstore/openbookstore/src/web/templates/login.html: No such file... {100870800F}> #<DJULA::COMPILED-TEMPLATE /builds/myopenbookstore/openbookstore/src/web/templates/login.html {100121854B}> NIL)

• same for the static files
• same for the translation, i18n files

libssl and other foreign libraries - use Deploy

ASDF tries to update itself on the target OS

Avoid any call to ASDF

No more asdf:system-relative-pathname and friends: this makes no sense in a binary release.

Use deploy:deployed-p

ql-dist:installedp is called and crashes

There is no applicable method for the generic function
     #<STANDARD-GENERIC-FUNCTION QL-DIST:INSTALLEDP (3)>
   when called with arguments
     (NIL).

We must get more clues. Let's print a backtrace around our main function.

(defun run ()
  "Call main, print a backtrace if an error occurs."
  (handler-bind ((error (lambda (c)
                          (format *error-output* "~&An error occured: ~a~&" c)
                          (format *error-output* "~&Backtrace:~&")
                          (trivial-backtrace:print-backtrace c))))
    (main)))

Now:

QLITE3 not found)
8: (SB-KERNEL::%SIGNAL Component :DBD-SQLITE3 not found)
9: (ERROR ASDF/FIND-COMPONENT:MISSING-COMPONENT :REQUIRES :DBD-SQLITE3)
10: ((:METHOD ASDF/OPERATE:OPERATE (SYMBOL T)) ASDF/LISP-ACTION:LOAD-OP :DBD-SQLITE3 :VERBOSE NIL) [fast-method]
11: ((SB-PCL::EMF ASDF/OPERATE:OPERATE) #<unused argument> #<unused argument> ASDF/LISP-ACTION:LOAD-OP :DBD-SQLITE3 :VERBOSE NIL)
12: ((LAMBDA NIL :IN ASDF/OPERATE:OPERATE))
13: (ASDF/SESSION:CALL-WITH-ASDF-SESSION #<FUNCTION (LAMBDA NIL :IN ASDF/OPERATE:OPERATE) {1005B93C2B}> :OVERRIDE NIL :KEY NIL :OVERRIDE-CACHE NIL :OVERRIDE-FORCING NIL)
14: ((:METHOD ASDF/OPERATE:OPERATE :AROUND (T T)) ASDF/LISP-ACTION:LOAD-OP :DBD-SQLITE3 :VERBOSE NIL) [fast-method]
15: (ASDF/OPERATE:LOAD-SYSTEM :DBD-SQLITE3 :VERBOSE NIL)
16: (DBI::LOAD-DRIVER :SQLITE3)
17: (DBI:CONNECT :SQLITE3 :DATABASE-NAME #P"/media/vince/93de6b92-7390-4253-bab1-425052ba7a10/home/vince/Téléchargements/bin/db.db")
18: (MITO.CONNECTION:CONNECT-TOPLEVEL :SQLITE3 :DATABASE-NAME #P"/media/vince/93de6b92-7390-4253-bab1-425052ba7a10/home/vince/Téléchargements/bin/db.db")
19: (BOOKSHOPS.MODELS:CONNECT #P"/home/vince/Téléchargements/bin/db.db")

Adding :dbd-sqlite3 in our .asd system dependencies fixes it.

Compile the static files at built-time, don't serve them from the file system

The need is to read the JS and CSS file contents at built time. The trick was: the regex dispatcher wants to return a string, not a function call that returns a string. So I used the #. reader macro, which lead me to define a few things in a separate file.

With Hunchentoot we don't use the usual "folder dispatcher", but we create new dispatchers and we return the content of the required file (as a string). The gist of it:

(push
  (ht:create-regex-dispatcher "/static/js/foo\.js" (lambda () "the-big-blob-of-js"))
  *dispatch-table*)

This is parameterized with deploy:deployed-p.

example: https://gitlab.com/myopenbookstore/openbookstore/-/commit/263fb20c6f512818bb564cb24666391193047efa

Bonus: embed in Neutralinojs (light Electron alternative)

Create a new Neutralino app. In the startup JavaScript file, start your binary with a SpawnProcess. Change the toml config to point the URL to yours, or create a new window.

https://neutralino.js.org/docs/