portability-libs.org

This is a draft – please send feedback and corrections to daniel@turtleware.eu.

It is important to know the difference between the language standard, implementation-specific extensions and the portability libraries. The language standard is something you can depend on any conforming implementation.

Sometimes it’s just not enough. You may want to do threading , or to serialize stuff, which simply can’t be or is very hard to express in the language provided by the standard. That’s where the implementation-specific extensions kick in. Why are they called “implementation-specific”? Because the API may be different between implementations – reaching consensus is a hard thing[fn:6].

The most straightforward approach I can imagine is to reach the documentation of the Common Lisp implementation you are currently using and to use the API provided by this implementation. I dare you not to do that! It’s definitely the easiest thing to do at first, but mind the consequences. You lock yourself and your users in the implementation you prefer. What if you want to run it on the JVM or to make it as a shared library? Nope, you’re locked-in.

“What can I do then?” – you may ask. Before I’ll answer this question, I’ll tell you how many people do it (or did it in the past) – they used read-time conditionals directly in the code. Something like the following:

(defun my-baz ()
  #+sbcl                        (sb-foo:do-baz-thing 'quux)
  #+ccl                         (ccl:baz-thing       'quux)
  #+(and ecl :baz-thing)        (ext:baz             'quux)
  #+abcl                        (ext:baz             'quux)
  #+(and clisp :built-with-baz) (ext:baz-thingie     'quux)
  #-(or sbcl ccl ecl abcl clisp)
  (error "Your implementation isn't supported. Fix me!"))

If the creator felt more fancy and had an extra time, they put it in the package my-app-compat. It’s all great, now your application works on all supported implementations. If somebody wanted theirs implementation to work, they just send him a patch, the creator would incorporate it and voila, everything works as desired.

There is one problem however. Libraries tend to depend on one another. There is also a lot of software which uses features beyond ANSI specification (it’s all good, programmers need these!). Do you see code duplication everywhere? How many times a snippet above has to be copy-pasted, or rewritten from scratch (it’s not black magic after all). API between ad-hoc implementations doesn’t exactly match, covered CL implementations differ.

So you quickload your favourite library which depends on 10 other libraries which implement BAZ functionality in it’s own unique way, with a slightly different API (that’s why we have my-baz abstraction after all, right?) on the unsupported implementation. Now, to make it work, a user has to:

1. Find which 10 libraries don’t work (not trivial!),
2. find and clone the repositories (we want to use git for patches),
3. fix each one of them (grep helps!), commit the changes,
4. push the changes to your own forked repository and create a pull request (or send a diff to the mailing list) – ten times,
5. voila, you’re done, profit, get rich, grab a beer.

It’s a lot of work which user probably won’t be bothered to do. They will just drop the task, choose another implementation or hack their own code creating the Yet Another Baz Library for the implementations he cares for, reinventing the wheel once more. It’s a hacker’s mortal sin.

I’m going to tell you now what is the Right Thing™ here. Of course you are free to disagree. When you feel that there is a functionality you need which isn’t covered by the standard you should

1. Look if there is a library that provides it.

   You may ask on IRC, mailing list, check out the CLiki, do some research on the web. Names sometimes start with trivial-*, but it’s not a rule. In other words: do your homework.

2. If you can’t find such library, create one.

   And by creating such library I mean comparing the API proposed by at least two CL implementations (three would be optimal IMHO), carefully designing your own API which covers the functionality (if it’s trivial, this should be easy) and implementing it in your library.

   Preferably add a fallback implementation for implementations not covered (with the appropriate warning, that it may be inefficient or not complete in one way or another).

   It may be worth reading the Maintaining Portable Lisp Programs paper written by Christophe Rhodes.

3. Write beautiful documentation.

   A CL implementation docs may be very rough. It takes time to write them and programmers tend to prioritize code over the documentation. It’s really bad, but it’s very common for the documentation to be incomplete or outdated.

   Document your library, describe what it does, how to use it. Don’t be afraid of the greatness! People will praise you, women will come, world will be a better place. And most importantly, your library will be useful to others.

4. Publish the library.
5. Pull the library as your project’s dependency.

I know it’s not easy, but in the long term it’s beneficial. I guarantee you that. That’s how an ecosystem comes to its fruition. Less duplication, more cooperation – pure benefit.

Some people don’t follow this path. They didn’t think it through, or they did and decided that keeping the dependency list minimal is essential for their project, or were simply lazy and hacked their own solution. There are also some old projects which exported a number of features being a very big portability library and an application at the same time (ACL-compat, McCLIM and others). What to do then?

If it’s a conscious decision of the developer (who doesn’t want to depend on anything), you can do nothing provide a patch adding your own implementation to the supported list. It’s their project, their choice, we have to respect that.

But before doing that just may simply ask if they mind plugging these hacks with the proper portability library. If they don’t then do it, everybody will benefit.

There are a few additional benefits of this portability library approach for the implementations itself. Having these internal details in one place makes it more probable that your implementation is already supported. If the library has a bug it’s easier to fix it in one place. Also, if the CL implementation changes it’s API, it’s easy to fix it. New CL implementations have simplified the task of making their work usable with existing libraries.

It is worth noting, that creating such library paves the way to the new quasi-standard functionalities. For instance Bordeaux Threads has added recently CONDITION-WAIT function, which isn’t implemented on all implementations. It is a very good stimulus to add it there. This is how library creators may have real impact on the implementation creators decisions about what to implement next.

Here are some great projects helping make many CL implementations member of a usable ecosystem. Many of these are considered being part of the de-facto standard:

bordeaux-threads

Provides thread primitives, locks and conditionals.

cl-store

Serializing and deserializing CL objects from streams.

cffi

Foreign function interface (accessing foreign libraries).

closer-mop

Meta-object protocol. Provides it’s own closer-common-lisp-user package (redefines for instance defmethod).

usocket

TCP/IP and UDP/IP socket interface.

osicat

Osicat is a lightweight operating system interface for Common Lisp on POSIX-like systems, including Windows.

cl-fad

Portable pathname library.

trivial-garbage

trivial-garbage provides a portable API to finalizers, weak hash-tables and weak pointers.

trivial-features

trivial-features ensures consistent *FEATURES* across multiple Common Lisp implementations.

trivial-gray-streams

trivial-gray-streams system provides an extremely thin compatibility layer for gray streams.

external-program

external-program enables running programs outside the Lisp process.

There are many other very good libraries which span multiple implementations. Some of them have theirs drawbacks.

For instance IOlib is a great library, but piggy-backs on UN*X.

UIOP is also a nice set of utilities, but isn’t documented well, does too many things at once and tries to deprecate other actively maintained projects – that is counterproductive and socially wrong. I’d discourage using it.

There are a few arguments supporting UIOP’s state – it is a direct dependency of ASDF, so it can’t (or doesn’t want to) depend on other libraries, but many utilities are needed by this commonly used system definition library. My reasoning here is as follows: UIOP goes beyond ASDF’s requirements and tries to make actively maintained projects obsolete.

Dear Daniel,

while there is a variety of valid opinions based on different interests and preferences, I believe your judgment of UIOP is based on incorrect premises.

First, I object to calling UIOP "not well documented". While UIOP isn't the best documented project around, all its exported functions and variables have pretty decent DOCSTRINGs, and there is at least one automatic document extractor, HEΛP, that can deal with the fact that UIOP is made of many packages, and extract the docstrings into a set of web pages, with a public heλp site listed in the UIOP README.md. The fact that some popular docstring extractors such as quickdocs can't deal with the many packages that UIOP creates with its own uiop:define-package doesn't mean that UIOP is less documented than other projects on which these extractors work well, it's a bug in these extractors.

Second, regarding the deprecation of other projects: yes, UIOP does try to deprecate other projects, but (a) it's a good thing, and (b) I don't know that any of the projects being deprecated is "actively maintained". It's a good thing to try to deprecate other lesser libraries, as I've argued in my article Consolidating Common Lisp libraries: whoever writes any library should work hard so it will deprecate all its rivals, or so that a better library will deprecate his and all rivals (such as optima deprecating my fare-matcher). That's what being serious about a library is all about. As for the quality of the libraries I'm deprecating, one widely-used project the functionality of which is completely covered by UIOP is cl-fad. cl-fad was a great improvement in its day, but some of its API is plain broken (e.g. the :directories argument to its walk-directory function has values with bogus names, while its many pathname manipulation functions get things subtly wrong in corner cases), and its implementation not quite as portable as UIOP (that works on all known actively used implementations). There is no reason whatsoever to ever choose cl-fad over UIOP for a new project. Another project is trivial-backtrace. I reproduced most of its functionality, except in a more stable, more portable way (to every single CL implementation). The only interface I didn't reproduce from it is map-backtrace, which is actually not portable in trivial-backtrace (only for SBCL and CCL), whereas serious portable backtrace users will want to use SLIME's or SLY's API, anyway. As for external-program, a good thing it has for it is some support for asynchronous execution of subprocesses; but it fails to abstract much over the discrepancies between implementations and operating systems, and is much less portable than uiop:run-program (as for trivial-shell, it just doesn't compete).

UIOP is also ubiquitous in a way that other libraries aren't: all implementations will let you (require "asdf") out of the box at which point you have UIOP available (exception: mostly dead implementations like Corman Lisp, GCL, Genera, SCL, XCL, may require you to install ASDF 3 on top of their code; still they are all supported by UIOP, whereas most portability libraries don't even bother with any of them). This ubiquity is important when writing scripts. Indeed, all the functionality in UIOP is so basic that ASDF needed it at some point — there is nothing in UIOP that wasn't itself required by some of ASDF's functionality, contrary to your claim that "UIOP goes beyond ASDF's requirements" (exception: I added one function or two to match the functionality in cl-fad, such as delete-directory-tree which BTW has an important safeguard argument :validate; but even those functions are used if not by ASDF itself, at least by the scripts used to release ASDF itself). I never decided "hey, let's make a better portability library, for the heck of it". Instead, I started making ASDF portable and robust, and at some point the portability code became a large chunk of ASDF and I made it into its own library, and because ASDF is targetting 16 different implementations and has to actually work on them, this library soon became much more portable, much more complete and much more robust than any other portability library, and I worked hard to achieve feature parity with all the libraries I was thereby deprecating.

Finally, a lot of the functionality that UIOP offers is just not offered by any other library, much less with any pretense of universal portability.

For the documentation thing, I really think Quickdocs could do a better job. The bug #24 stated that problem, however, it's remain to be solved. I will check this out if I have free time recently.

I use UIOP a lot in my previous company, the reason is simple and maybe a little naive: my manager didn't want to involve too many add-ons in the software. UIOP is shipped together with ASDF, it's really "convenient", and its robustness is the final reason why I will stick to it. If people understand how UIOP came out in the history from ASDF2 to ASDF3, I think people will understand why it's acting like deprecating several other projects -- that's not the original idea of it.

But anyway, I really learned a lot from this post and also the comments. In my opinion, avoid reinventing the wheels is the right idea and directions for this community. So from that perspective, I support @fare's idea "It's a good thing to try to deprecate other lesser libraries". Including this article and along with Maintaining Portable Lisp Programs and @fare's Consolidating Common Lisp Libraries, we should let more people involved in this topic.