Skip to content

Instantly share code, notes, and snippets.

@hobu

hobu/sorting.md

Last active August 7, 2017 13:22
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save hobu/28a5412f10efb429649874b84eb7ce4a to your computer and use it in GitHub Desktop.
Save hobu/28a5412f10efb429649874b84eb7ce4a to your computer and use it in GitHub Desktop.
Download ZIP
Raw
sorting.md

On Aug 4, 2017, at 11:01 AM, Martin Isenburg martin.isenburg@gmail.com wrote:

Hello,

I watched this nice video on how to download LiDAR from the USGS Earth Explorer.

http://youtu.be/kdzC5_XPc0k

But then I saw the moment that the files were downloaded and I was initially shocked to see that the LAZ file was hardly smaller than the winzipped LAS file. So I decided to investigate by following the steps in the video to download the exact same tile (see attached picture series).

Turns our the points are in a horrible order for compression. They are ordered by elevation from top to bottom. Does anyone know who is behind these LiDAR deliveries? Or who is in charge for the LiDAR holdings on the USGS Earth Explorer This is from quite a while ago but it would still make sense to improve the files as the download sizes shrinks by over 60 percent. Maybe it helps to note that the files name "Aero-Metric LASLib" as their generating software.

A simple sort by GPS time and return number reduces the compressed LAZ file size from 56 MB to 22 MB. The reason for this incredible gain is that a sort in z order pretty much destroys all predictive rules used by LASzip:

http://rapidlasso.com/lassort

lassort -i CO_ArkansasValley_2010_000536.laz^
            -gps_time -return_number ^
            -odix _sorted -olaz

Here the file sizes for comparison:

 85,144,477 CO_ArkansasValley_2010_000536.ZIP
162,272,731 CO_ArkansasValley_2010_000536.las
 56,629,291 CO_ArkansasValley_2010_000536.laz
 22,292,134 CO_ArkansasValley_2010_000536_sorted.laz

Martin,

When I use PDAL to sort it with only GPSTime before compressing, I get a barely smaller file than your attempt (98 bytes!). Does including ReturnNumber in the sorting typically make for a big compression prediction win?

Presumably it was "Aero-Metric" or a vendor using that who delivered these files to USGS in 2010 or whenever. I doubt USGS did anything to the files before getting them besides just pushing them through las2las.

Howard

pdal CO_ArkansasValley_2010_000536.laz CO_ArkansasValley_2010_000536-time-only.laz sort 
    --filters.sort.dimension=GPSTime 
    --writers.las.forward=all

-rw-r--r--+  1 hobu  staff  22292036 Aug  4 16:46 CO_ArkansasValley_2010_000536-time-only.laz
-rw-r--r--@  1 hobu  staff  56629291 Aug  4 16:37 CO_ArkansasValley_2010_000536.laz
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment