This is a draft – please send feedback and corrections to daniel@turtleware.eu.
It is important to know the difference between the language standard, implementation-specific extensions and the portability libraries. The language standard is something you can depend on any conforming implementation.
Sometimes it’s just not enough. You may want to do threading , or to serialize stuff, which simply can’t be or is very hard to express in the language provided by the standard. That’s where the implementation-specific extensions kick in. Why are they called “implementation-specific”? Because the API may be different between implementations – reaching consensus is a hard thing[fn:6].
The most straightforward approach I can imagine is to reach the
documentation of the Common Lisp implementation you are currently
using and to use the API provided by this implementation. I dare you
not to do that! It’s definitely the easiest thing to do at first, but
mind the consequences. You lock yourself and your users in the
implementation you prefer. What if you want to run it on the JVM
or
to make it as a shared library? Nope, you’re locked-in.
“What can I do then?” – you may ask. Before I’ll answer this question, I’ll tell you how many people do it (or did it in the past) – they used read-time conditionals directly in the code. Something like the following:
(defun my-baz ()
#+sbcl (sb-foo:do-baz-thing 'quux)
#+ccl (ccl:baz-thing 'quux)
#+(and ecl :baz-thing) (ext:baz 'quux)
#+abcl (ext:baz 'quux)
#+(and clisp :built-with-baz) (ext:baz-thingie 'quux)
#-(or sbcl ccl ecl abcl clisp)
(error "Your implementation isn't supported. Fix me!"))
If the creator felt more fancy and had an extra time, they put it in
the package my-app-compat
. It’s all great, now your application
works on all supported implementations. If somebody wanted theirs
implementation to work, they just send him a patch, the creator
would incorporate it and voila, everything works as desired.
There is one problem however. Libraries tend to depend on one another. There is also a lot of software which uses features beyond ANSI specification (it’s all good, programmers need these!). Do you see code duplication everywhere? How many times a snippet above has to be copy-pasted, or rewritten from scratch (it’s not black magic after all). API between ad-hoc implementations doesn’t exactly match, covered CL implementations differ.
So you quickload
your favourite library which depends on 10 other
libraries which implement BAZ
functionality in it’s own unique way,
with a slightly different API (that’s why we have my-baz
abstraction
after all, right?) on the unsupported implementation. Now, to make it
work, a user has to:
- Find which 10 libraries don’t work (not trivial!),
- find and clone the repositories (we want to use git for patches),
- fix each one of them (grep helps!), commit the changes,
- push the changes to your own forked repository and create a pull request (or send a diff to the mailing list) – ten times,
- voila, you’re done, profit, get rich, grab a beer.
It’s a lot of work which user probably won’t be bothered to do. They
will just drop the task, choose another implementation or hack their
own code creating the Yet Another Baz Library
for the
implementations he cares for, reinventing the wheel once more. It’s a
hacker’s mortal sin.
I’m going to tell you now what is the Right Thing™ here. Of course you are free to disagree. When you feel that there is a functionality you need which isn’t covered by the standard you should
- Look if there is a library that provides it.
You may ask on IRC, mailing list, check out the CLiki, do some research on the web. Names sometimes start with
trivial-*
, but it’s not a rule. In other words: do your homework. - If you can’t find such library, create one.
And by creating such library I mean comparing the API proposed by at least two CL implementations (three would be optimal IMHO), carefully designing your own API which covers the functionality (if it’s trivial, this should be easy) and implementing it in your library.
Preferably add a fallback implementation for implementations not covered (with the appropriate warning, that it may be inefficient or not complete in one way or another).
It may be worth reading the Maintaining Portable Lisp Programs paper written by Christophe Rhodes.
- Write beautiful documentation.
A CL implementation docs may be very rough. It takes time to write them and programmers tend to prioritize code over the documentation. It’s really bad, but it’s very common for the documentation to be incomplete or outdated.
Document your library, describe what it does, how to use it. Don’t be afraid of the greatness! People will praise you, women will come, world will be a better place. And most importantly, your library will be useful to others.
- Publish the library.
- Pull the library as your project’s dependency.
I know it’s not easy, but in the long term it’s beneficial. I guarantee you that. That’s how an ecosystem comes to its fruition. Less duplication, more cooperation – pure benefit.
Some people don’t follow this path. They didn’t think it through, or they did and decided that keeping the dependency list minimal is essential for their project, or were simply lazy and hacked their own solution. There are also some old projects which exported a number of features being a very big portability library and an application at the same time (ACL-compat, McCLIM and others). What to do then?
If it’s a conscious decision of the developer (who doesn’t want to depend on anything), you can do nothing provide a patch adding your own implementation to the supported list. It’s their project, their choice, we have to respect that.
But before doing that just may simply ask if they mind plugging these hacks with the proper portability library. If they don’t then do it, everybody will benefit.
There are a few additional benefits of this portability library approach for the implementations itself. Having these internal details in one place makes it more probable that your implementation is already supported. If the library has a bug it’s easier to fix it in one place. Also, if the CL implementation changes it’s API, it’s easy to fix it. New CL implementations have simplified the task of making their work usable with existing libraries.
It is worth noting, that creating such library paves the way to the
new quasi-standard functionalities. For instance Bordeaux Threads has
added recently CONDITION-WAIT
function, which isn’t implemented on
all implementations. It is a very good stimulus to add it there. This
is how library creators may have real impact on the implementation
creators decisions about what to implement next.
Here are some great projects helping make many CL implementations member of a usable ecosystem. Many of these are considered being part of the de-facto standard:
- bordeaux-threads
-
Provides thread primitives, locks and conditionals.
- cl-store
-
Serializing and deserializing CL objects from streams.
- cffi
-
Foreign function interface (accessing foreign libraries).
- closer-mop
-
Meta-object protocol. Provides it’s own
closer-common-lisp-user
package (redefines for instancedefmethod
). - usocket
-
TCP/IP and UDP/IP socket interface.
- osicat
-
Osicat is a lightweight operating system interface for Common Lisp on POSIX-like systems, including Windows.
- cl-fad
-
Portable pathname library.
- trivial-garbage
-
trivial-garbage
provides a portable API to finalizers, weak hash-tables and weak pointers. - trivial-features
-
trivial-features
ensures consistent*FEATURES*
across multiple Common Lisp implementations. - trivial-gray-streams
-
trivial-gray-streams
system provides an extremely thin compatibility layer for gray streams. - external-program
-
external-program
enables running programs outside the Lisp process.
There are many other very good libraries which span multiple implementations. Some of them have theirs drawbacks.
For instance IOlib is a great library, but piggy-backs on UN*X.
UIOP is also a nice set of utilities, but isn’t documented well, does too many things at once and tries to deprecate other actively maintained projects – that is counterproductive and socially wrong. I’d discourage using it.
There are a few arguments supporting UIOP
’s state – it is a direct
dependency of ASDF
, so it can’t (or doesn’t want to) depend on other
libraries, but many utilities are needed by this commonly used system
definition library. My reasoning here is as follows: UIOP
goes
beyond ASDF
’s requirements and tries to make actively maintained
projects obsolete.
Dear Daniel,
while there is a variety of valid opinions based on different interests and preferences, I believe your judgment of UIOP is based on incorrect premises.
First, I object to calling UIOP "not well documented". While UIOP isn't the best documented project around, all its exported functions and variables have pretty decent DOCSTRINGs, and there is at least one automatic document extractor, HEΛP, that can deal with the fact that UIOP is made of many packages, and extract the docstrings into a set of web pages, with a public heλp site listed in the UIOP README.md. The fact that some popular docstring extractors such as quickdocs can't deal with the many packages that UIOP creates with its own uiop:define-package doesn't mean that UIOP is less documented than other projects on which these extractors work well, it's a bug in these extractors.
Second, regarding the deprecation of other projects: yes, UIOP does try to deprecate other projects, but (a) it's a good thing, and (b) I don't know that any of the projects being deprecated is "actively maintained". It's a good thing to try to deprecate other lesser libraries, as I've argued in my article Consolidating Common Lisp libraries: whoever writes any library should work hard so it will deprecate all its rivals, or so that a better library will deprecate his and all rivals (such as optima deprecating my fare-matcher). That's what being serious about a library is all about. As for the quality of the libraries I'm deprecating, one widely-used project the functionality of which is completely covered by UIOP is cl-fad. cl-fad was a great improvement in its day, but some of its API is plain broken (e.g. the
:directories
argument to its walk-directory function has values with bogus names, while its many pathname manipulation functions get things subtly wrong in corner cases), and its implementation not quite as portable as UIOP (that works on all known actively used implementations). There is no reason whatsoever to ever choose cl-fad over UIOP for a new project. Another project is trivial-backtrace. I reproduced most of its functionality, except in a more stable, more portable way (to every single CL implementation). The only interface I didn't reproduce from it is map-backtrace, which is actually not portable in trivial-backtrace (only for SBCL and CCL), whereas serious portable backtrace users will want to use SLIME's or SLY's API, anyway. As for external-program, a good thing it has for it is some support for asynchronous execution of subprocesses; but it fails to abstract much over the discrepancies between implementations and operating systems, and is much less portable than uiop:run-program (as for trivial-shell, it just doesn't compete).UIOP is also ubiquitous in a way that other libraries aren't: all implementations will let you
(require "asdf")
out of the box at which point you have UIOP available (exception: mostly dead implementations like Corman Lisp, GCL, Genera, SCL, XCL, may require you to install ASDF 3 on top of their code; still they are all supported by UIOP, whereas most portability libraries don't even bother with any of them). This ubiquity is important when writing scripts. Indeed, all the functionality in UIOP is so basic that ASDF needed it at some point — there is nothing in UIOP that wasn't itself required by some of ASDF's functionality, contrary to your claim that "UIOP goes beyond ASDF's requirements" (exception: I added one function or two to match the functionality in cl-fad, such as delete-directory-tree which BTW has an important safeguard argument:validate
; but even those functions are used if not by ASDF itself, at least by the scripts used to release ASDF itself). I never decided "hey, let's make a better portability library, for the heck of it". Instead, I started making ASDF portable and robust, and at some point the portability code became a large chunk of ASDF and I made it into its own library, and because ASDF is targetting 16 different implementations and has to actually work on them, this library soon became much more portable, much more complete and much more robust than any other portability library, and I worked hard to achieve feature parity with all the libraries I was thereby deprecating.Finally, a lot of the functionality that UIOP offers is just not offered by any other library, much less with any pretense of universal portability.