Skip to content

Instantly share code, notes, and snippets.

@adulau

adulau/Hashlookup service - how to improve your digital forensic investigations.md

Last active May 20, 2022 06:21
Show Gist options
  • Star 3 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save adulau/e9e95fead4f32ac0fe725cb2a32fdb51 to your computer and use it in GitHub Desktop.
Save adulau/e9e95fead4f32ac0fe725cb2a32fdb51 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
Hashlookup service - how to improve your digital forensic investigations.md
title author institute theme colortheme fonttheme mainfont fontsize urlcolor linkstyle aspectratio date lang section-titles
Hashlookup service - how to improve your digital forensic investigations
Alexandre Dulaunoy
CIRCL
Frankfurt
beaver
professionalfonts
Hack Nerd Font
10pt
red
bold
169
30/09/2021
en-US
false

Introduction

  • State of current NIST NSRL databases and other known file filters (KFF)
    • too few Operating Systems / Software available (e.g. OSX?, Linux distributions),
    • nsrllookup.com / nsrlsrv use their own protocol, next version will be gRPC,
    • nsrlsrv only support MD5,
    • many sources are difficult to use (e.g. NSRL ISOs), ill-maintained or outdated.

Requirements

  • The need of a public and easy to use API for all sources (NSRL is not alone)
    • A global public instance of all known sources,
    • a common ReST API normalizes the access to several datasources,
    • available for MD5, and SHA1 (and more),
    • that includes fuzzy hashes,
    • additional datapoints available through the intersection of datasources,

CIRCL hashlookup API

  • https://hashlookup.circl.lu/ - OpenAPI Swagger
  • Current databases:
    • NIST NSRL - RDS hash sets including current, modern, android, iOS and legacy,
    • Ubuntu packages distribution,
    • CentOS core OS distribution,
    • Fedora project EPEL repository,
    • Kali linux packages distribution and OpenSUSE distribution.

How to use the service

Checking a single SHA-1 or MD5 hash

Where is this errno.h (found in a specific package) include available?

https://hashlookup.circl.lu/lookup/sha1/40f68888766b102b94f4c414eb0574ac6467be93

{
  "CRC32": "D6614BD0",
  "FileName": "./usr/s390x-linux-gnu/include/errno.h",
  "FileSize": "1832",
  "MD5": "5CD0FF97AAC5391C34757DCA05E2B6BA",
  "OpSystemCode": {
    "MfgCode": "1006",
    "OpSystemCode": "362",
    "OpSystemName": "TBD",
    "OpSystemVersion": "none"
  },
  "ProductCode": {
    "ApplicationType": "Operating System",
    "Language": "Multilanguage",
    "MfgCode": "2529",
    "OpSystemCode": "910",
    "ProductCode": "217853",
    "ProductName": "Linux Mint - Tricia",
    "ProductVersion": "19.3"
  },
  "SHA-1": "40F68888766B102B94F4C414EB0574AC6467BE93",
  "SHA-256": "B6DEBCA969A2C6AA756232E2DD69EBFC5EE8FEDBDC0229B869917518D15689A5",
  "SSDEEP": "48:826ROuNkUynHJO+S8RGWv8J1AkJ8IZ9th0MGJ0:3uNA4+Ynj05J0",
  "SpecialCode": "",
  "TLSH": "T1D431FE192A15437395D20763A9CF20DAC41EAAEB32A6A4D0064D965E8C4BA7B13F36D8",
  "db": "nsrl_modern_rds",
  "insert-timestamp": "1630959315.4398303",
  "source": "NSRL",
  "parents": [
    {
      "FileSize": "2080372",
      "MD5": "059D45A0ABF3D857B267C17260BDE9FD",
      "PackageDescription": "GNU C Library: Development Libraries and Header Files (for cross-compiling)\n This package was generated by dpkg-cross for cross compiling.\n .\n Contains the symlinks, headers, and object files needed to compile\n and link programs which use the standard C library.",
      "PackageMaintainer": "Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>",
      "PackageName": "libc6-dev-mipsel-cross",
      "PackageSection": "libdevel",
      "PackageVersion": "2.27-3ubuntu1cross1",
      "SHA-1": "03F71DB19EA877E81894B4B6CF0C05C92F82E0A8",
      "SHA-256": "A6B5664F40EFC472FF167052F9B82F5268C1797CBB8EF433D3130D09E2FE6FAC"
    },
    {
      "FileSize": "2548052",
      "MD5": "E3E77A9F2DD94EF20D4BA4ED2F32BD3A",
      "PackageDescription": "GNU C Library: Development Libraries and Header Files (for cross-compiling)\n This package was generated by dpkg-cross for cross compiling.\n .\n Contains the symlinks, headers, and object files needed to compile\n and link programs which use the standard C library.",
      "PackageMaintainer": "Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>",
      "PackageName": "libc6-dev-ppc64el-cross",
      "PackageSection": "libdevel",
      "PackageVersion": "2.27-3ubuntu1cross1.1",
      "SHA-1": "05B73A3E3CCE0A194218FE86EA769B402E0FB8F8",
      "SHA-256": "ED647442B2C6F8C5169C130CADDA3D0B664E8DF2CDADF7B3EE6D2C66D4AB98A3"
    },
    {
      "FileSize": "2469956",
      "MD5": "0ECD877AF006D8D6F37BAC9619F9D1FB",
      "PackageDescription": "GNU C Library: Development Libraries and Header Files (for cross-compiling)\n This package was generated by dpkg-cross for cross compiling.\n .\n Contains the symlinks, headers, and object files needed to compile\n and link programs which use the standard C library.",
      "PackageMaintainer": "GNU Libc Maintainers <debian-glibc@lists.debian.org>",
      "PackageName": "libc6-dev-mips64el-cross",
      "PackageSection": "libdevel",
      "PackageVersion": "2.27-3cross1",
      "SHA-1": "06B1989344676FFF1CD2D554E8EB7A864940B5E1",
      "SHA-256": "3547033EADABF0C1F72E88C21B00CE68250861B8D3A38F6BD274B0307AA30F85"
    },
    {
      "FileSize": "2347880",
      "MD5": "09A161D0E52F411E3EF53A0F1C3522A3",
      "PackageDescription": "GNU C Library: Development Libraries and Header Files (for cross-compiling)\n This package was generated by dpkg-cross for cross compiling.\n .\n Contains the symlinks, headers, and object files needed to compile\n and link programs which use the standard C library.",
      "PackageMaintainer": "GNU Libc Maintainers <debian-glibc@lists.debian.org>",
      "PackageName": "libc6-dev-hppa-cross",
      "PackageSection": "libdevel",
      "PackageVersion": "2.27-3cross3",
      "SHA-1": "082386F6B58058E1149842666EB370F03FC4E0C4",
      "SHA-256": "112381BACAA49D2F533E78144964E72DB70743332E9DED993EB439DB0568ED62"
    },
    {
      "FileSize": "2280492",
      "MD5": "6CC56E1C26C33DC88A1F9E777B4DD376",
      "PackageDescription": "GNU C Library: Development Libraries and Header Files (for cross-compiling)\n This package was generated by dpkg-cross for cross compiling.\n .\n Contains the symlinks, headers, and object files needed to compile\n and link programs which use the standard C library.",
      "PackageMaintainer": "GNU Libc Maintainers <debian-glibc@lists.debian.org>",
      "PackageName": "libc6-dev-mipsel-cross",
      "PackageSection": "libdevel",
      "PackageVersion": "2.27-6cross1",
      "SHA-1": "099DCFA33665BABFE039CCDFA7BBB5B40FCA7842",
      "SHA-256": "5E5A4CF231AFE98B8ABB4E48A082D1A19BBD32D013A9D235632C0CD8D143E595"
    },
    {
      "FileSize": "2424220",
      "MD5": "C134893A8E7F8889128DB8C4470B0C85",
      "PackageDescription": "GNU C Library: Development Libraries and Header Files (for cross-compiling)\n This package was generated by dpkg-cross for cross compiling.\n .\n Contains the symlinks, headers, and object files needed to compile\n and link programs which use the standard C library.",
      "PackageMaintainer": "Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>",
      "PackageName": "libc6-dev-x32-cross",
      "PackageSection": "libdevel",
      "PackageVersion": "2.27-3ubuntu1cross1",
      "SHA-1": "0AB737A2240CDF265CF2F2CD4DD20194DD3B951F",
      "SHA-256": "C532514305E00BB0F3017689C60EDF3ECAB025FB49C50DE53AC529486D218BB9"
    },
    {
      "FileSize": "2282248",
      "MD5": "8E1B993D406D3A63236FDA01319B9415",
      "PackageDescription": "GNU C Library: Development Libraries and Header Files (for cross-compiling)\n This package was generated by dpkg-cross for cross compiling.\n .\n Contains the symlinks, headers, and object files needed to compile\n and link programs which use the standard C library.",
      "PackageMaintainer": "Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>",
      "PackageName": "libc6-dev-mipsr6el-cross",
      "PackageSection": "libdevel",
      "PackageVersion": "2.27-3ubuntu1cross1.2",
      "SHA-1": "0B2610137605678ABD4FB95B1691133D87534144",
      "SHA-256": "0B8100CCAA5C723A40EB7661D326885AAC865A70FA6798BBAD1B1F9C5D9F5C25"
    },
    {
      "FileSize": "2219952",
      "MD5": "6CAAC685999DD5C1484C16AECB670757",
      "PackageDescription": "GNU C Library: Development Libraries and Header Files (for cross-compiling)\n This package was generated by dpkg-cross for cross compiling.\n .\n Contains the symlinks, headers, and object files needed to compile\n and link programs which use the standard C library.",
      "PackageMaintainer": "GNU Libc Maintainers <debian-glibc@lists.debian.org>",
      "PackageName": "libc6-dev-sh4-cross",
      "PackageSection": "libdevel",
      "PackageVersion": "2.27-2cross1",
      "SHA-1": "0CBFF2F15C06E5772900E3DC00923BC66148237F",
      "SHA-256": "91607A291D6A2DF448C382A00A333E41723B0A92D056F332084C4C3C5C38ACBC"
    },
    {
      "FileSize": "2450276",
      "MD5": "CDD89E81A7D366E6C4D60A0A8045389C",
      "PackageDescription": "GNU C Library: Development Libraries and Header Files (for cross-compiling)\n This package was generated by dpkg-cross for cross compiling.\n .\n Contains the symlinks, headers, and object files needed to compile\n and link programs which use the standard C library.",
      "PackageMaintainer": "GNU Libc Maintainers <debian-glibc@lists.debian.org>",
      "PackageName": "libc6-dev-i386-cross",
      "PackageSection": "libdevel",
      "PackageVersion": "2.27-3cross1",
      "SHA-1": "0D0FB569EE34282584DD4B2E62A9B52DDF34EAD3",
      "SHA-256": "7A906141BC778069510FF11B6157EF25D3A79D1C03947C36432C5EE5A5883EC0"
    },
    {
      "FileSize": "2480072",
      "MD5": "46D2AA0A88BC8C0DF2152CD98B2A4083",
      "PackageDescription": "GNU C Library: Development Libraries and Header Files (for cross-compiling)\n This package was generated by dpkg-cross for cross compiling.\n .\n Contains the symlinks, headers, and object files needed to compile\n and link programs which use the standard C library.",
      "PackageMaintainer": "GNU Libc Maintainers <debian-glibc@lists.debian.org>",
      "PackageName": "libc6-dev-mipsr6el-cross",
      "PackageSection": "libdevel",
      "PackageVersion": "2.27-3cross3",
      "SHA-1": "0D8A3CB782F20B1203688AFEF84D3BAC807CA3D8",
      "SHA-256": "8C42CE2D680AA1C03158907F54908CA7FFDB240785DC5682435C4CDE768369EA"
    }
  ]
}

A tar file found on a Ubuntu distribution

https://hashlookup.circl.lu/lookup/sha1/b0b105d1c3cb3da684a1d487b46156238579ffb4

{
  "FileName": "./bin/tar",
  "FileSize": "423312",
  "MD5": "749AD0837C043E1AE63198678E0F5161",
  "SHA-1": "B0B105D1C3CB3DA684A1D487B46156238579FFB4",
  "SHA-256": "D45FE4E12DBE475EF2C283FA4A5A1658749F3FBD8EB8056EAE3C1EF9228876D0",
  "SSDEEP": "6144:686SGkTRyemT4nKgLDlSYMKHa8lwlT2lRZtFaNsGohPMQ/bEFPoPw8wTBu:6864TRtm83QJawloBMNsRPMfN",
  "TLSH": "T1E5943A16FDA214BCC1A6C9B0467B9376797274D843222A7F359CD5303E42F642F1EBA2",
  "parents": [
    {
      "FileSize": "233760",
      "MD5": "E3F8D82F755B246685409B087D312037",
      "PackageDescription": "GNU version of the tar archiving utility\n Tar is a program for packaging a set of files as a single archive in tar\n format.  The function it performs is conceptually similar to cpio, and to\n things like PKZIP in the DOS world.  It is heavily used by the Debian package\n management system, and is useful for performing system backups and exchanging\n sets of files with others.",
      "PackageMaintainer": "Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>",
      "PackageName": "tar",
      "PackageSection": "utils",
      "PackageVersion": "1.29b-2ubuntu0.2",
      "SHA-1": "C449F470704294CC40DA488C32A4ADB191E2C5DA",
      "SHA-256": "6BDBB90C9C073F8E8F92DA231A2E553410CE397489F2F1F77D1AE8DDBD0C7BC4"
    }
  ]
}

How to use the service

Forensic use-case

A compromised Linux server needs to be analyzed. How can I quickly find the binaries from the original distribution from others?

sha1sum * | cut -f1 -d" " | parallel 'curl -s https://hashlookup.circl.lu/lookup/sha1/{}' | jq .

hashlookup-forensic-analyser

hashlookup-forensic-analyser is a simple Python script to analyse a forensic target (such as a directory) to find and report files found and not found from CIRCL hashlookup public service. This tool can help a digital forensic investigator to know the context, origin of specific files during a digital forensic investigation.

hashlookup-forensic-analyser

How to use hashlookup offline?

If you don't want to share your lookups online, hashlookup provides a bloom filter to download.

A bloom filter (a compact representation of the dataset) is available at https://cra.circl.lu/hashlookup/hashlookup-full.bloom (~700MB) with all the SHA-1 value known in hashlookup. The format of the bloom filter is the DCSO bloom library and cli.

How to use locally the bloomfilter (just don't forget to install the DCSO bloom cli):

find /usr/bin/ -type f -print0 | xargs -0 sha1sum | awk '{ print $1 }'  | tr a-f A-F | bloom c /home/adulau/hashlookup-full.bloom

The bloom filter doesn't contain any metadata. It's just the SHA-1 hash value. You check the statistics and info with the following command

adulau@kolmogorov ~/hashlookup $ bloom s hashlookup-full.bloom 
File:			/home/adulau/hashlookup/hashlookup-full.bloom
Capacity:		296893697
Elements present:	296890922
FP probability:		1.00e-04
Bits:			5691486835
Hash functions:		14

How can you help us?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment