acaporrini/compress_urls.md

## compress_urls.md

      
    Raw
  

              compress_urls.md
            
          
    How to compress url parameters (and any other string) in Ruby

Introduction

Working with web applications and microservices we often have the problem of passing long lists of settings through http requests.
While the canonical way of doing this would be to use a POST request and pass the values in the body of the request, this is not always an option for us.
In fact there can be cases where we need to generate URLs that can be passed to other users as a link, and reused by them later to trigger an action in the target Rails server.
The only way to achieve this is for us to be able to pass these values as URL parameters in a GET request, but unfortunately these URLs exceed often the maximum url length limit allowed by the browsers or web servers.
To workaround this problem we decided to compress these values using the zlib library, which turned out to be very effective for this kind of compression, especially in cases of list of settings which usually contain repetitive sequences of characters.
Example

Our client application generates a request URL to another app, appending a list of values as an array parameter. Normally the request url would look like:
http://serverapp.com?myvalues[]=value_1&myvalues[]=value_2&myvalues[]=value_3... # A very long list that exceed the maximum allowed number

This would usually trigger an error if the URL exceeds 2083 characters or any other browser/server specific URL limit.
In order to send the values inside the my_values array we need to create a compressed string.
Client logic and url generation

First we merge the array of values in a single comma separated string:
comma_separated_values = my_values.join(",")
=> "value_1,value_2,value_3"
We compress the values using Zlib:
compressed_values = Zlib::Deflate.deflate(comma_separated_values)
=> "x\x9C+K\xCC)M\x8D7\xD4)\x03\xD3FP\xDA\x18\x00g$\bc"
Then we encode the compressed values using base64:
encoded_values = Base64.strict_encode64(compressed_values)
 => "eJwrS8wpTY031CkD00ZQ2hgAZyQIYw=="
Finally we encode the string to be safely passed as a string parameter:
url_safe_values = CGI.escape(encoded_values)
=>  "eJwrS8wpTY031CkD00ZQ2hgAZyQIYw%3D%3D"
Now we can append the compressed string to our URL as a parameter:
request_url =  "http://serverapp.com?myvalues=#{url_safe_values}"
=> "http://serverapp.com?myvalues=eJwrS8wpTY031CkD00ZQ2hgAZyQIYw%3D%3D"
This URL can be used to issue requests to our server, both programmatically and by a user clicking on the link. In order  to process this, the server must implement the decompression logic, which will basically do the reverse process of what we have done in the client:
Server logic and URL parameter decompression

Unescape the values:
unescaped_values = CGI.unescape(params[:my_values])
=> "eJwrS8wpTY031CkD00ZQ2hgAZyQIYw=="
Decode them:
encoded_values = Base64.strict_decode64(unescaped_values)
=> "x\x9C+K\xCC)M\x8D7\xD4)\x03\xD3FP\xDA\x18\x00g$\bc"
Decompress them:
comma_separated_values = Zlib::Inflate.inflate(encoded_values)
=> "value_1,value_2,value_3"
And finally expand the comma separated values in to the original array:
array_of_values = comma_separated_values.split(",")
=> ["value_1", "value_2", "value_3"]
That's how we implemented an effective technique to pass a long list of values through a GET request without exceeding the maximum allowed length.