Uploading files with drakma

file-upload

Uploading files with drakma
===========================

## The project 

I'm writing an interface to the [Deluge](http://deluge-torrent.org)
torrent client in Common Lisp. I'm using drakma for the http stuff,
and yason for JSON parsing.

## The problem

Sending the vanilla POST requests to the deluge webui was easy; it
wasn't until I started trying to upload the .torrent files to the
server that I run into problems.

What I needed to do was this: use an `http-request` to POST the file
using `Content-Type: multipart/form-data`. Happily, this is well
supported in drakma, but it's not very well documented. Here is the
[parameter](http://www.weitz.de/drakma/#arg-parameters) to
`http-request` that I couldn't quite grok:

> _parameters_ is an alist of name/value pairs (the car and the cdr each
> being a string) which denotes the parameters which are added to the
> query part of the URL or (in the case of a POST request) comprise the
> body of the request. (But see _content_ below.) The values can also be
> `NIL` in which case only the name (without an equal sign) is used in the
> query string. The name/value pairs are URL-encoded using the
> FLEXI-STREAMS external format _external-format-out_ before they are sent
> to the server unless _form-data_ is true in which case the POST request
> body is sent as 'multipart/form-data' using _external-format-out_. The
> values of the _parameters_ alist can also be pathnames, open binary
> input streams, unary functions, or lists where the first element is of
> one of the former types. These values denote files which should be
> sent as part of the request body. If files are present in _parameters_,
> the content type of the request is always 'multipart/form-data'. If
> the value is a list, the part of the list behind the first element is
> treated as a plist which can be used to specify a content type and/or
> a filename for the file, i.e. such a value could look like, e.g.,
> `(#p"/tmp/my_file.doc" :content-type "application/msword" :filename
> "upload.doc")`. 

This is definitely what I need, since I need to specify both a
`content-type` of `application/x-bittorrent` and the appropriate file
name. But what I couldn't figure out is what the 'key' part of that
alist was supposed to be.

Now, the beauty of OSS is that I can just dig into the code and check!
In drakma's request.lisp, here is what we have:

    (dolist (name/value parameters)
      (destructuring-bind (name . value)
          name/value
        (when (or (pathnamep value)
                  (streamp value)
                  (functionp value))
          (setq value (list value)))
        (format stream "--~A" boundary)
        (crlf)
        (format stream "Content-Disposition: form-data; name=\"~A\"" name)
        ;; do more stuff
        ))

So whatever I specify for the key of this pair will go into `name=""`. A
quick look at Fiddler tells me that the deluge server is expecting
`name="file"`. Great! So here's how I'll build the request:

    (let ((params `((file . (,path
                             :content-type "application/x-bittorrent"
                             :filename ,path)))))
      (drakma:http-request (make-deluge-uri host port "/upload")
                           :proxy '("localhost" 8888) ;; fiddler2
                           :method :post
                           :form-data t
                           :parameters params
                           :cookie-jar cookie-jar))

When I send this to the server, I'm expecting a JSON response
like this:

    {
        "files": ["/tmp/delugeweb-7SCFdk/tmpSn6Nhy.torrent"], 
        "success": true
    }

But what I end up getting is this:

    {
        "files": [], 
        "success": true
    }

Weird... Again, let's dig into the deluge code. In
deluge/ui/web/server.py, we have this:

    if "file" not in request.args:
        request.setResponseCode(http.OK)
        return common.json.dumps({
                'success': True,
                'files': []
                })

So what's the problem? Turns out it's string case. Deluge is expecting
`name="file"`, but drakma's `(format stream "name=\"~A\"" name)` is
returning `name="FILE"`, since symbols are upcased by lisp's reader.

Fix that, and now the request looks like this:

    (let ((params '((|file| . (path
                               :content-type "application/x-bittorrent"
                               :filename path)))))
      (drakma:http-request (make-deluge-uri host port "/upload")
                           :proxy '("localhost" 8888) ;; fiddler2
                           :method :post
                           :form-data t
                           :parameters params
                           :cookie-jar cookie-jar))

And now everything is working!