Skip to content

Instantly share code, notes, and snippets.

@alanshaw

alanshaw/linkdex.md

Created August 24, 2022 07:50
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save alanshaw/876aafba676b13890398500b96f82d8e to your computer and use it in GitHub Desktop.
Save alanshaw/876aafba676b13890398500b96f82d8e to your computer and use it in GitHub Desktop.
Download ZIP
Linkdex idea
Raw
linkdex.md

Linkdex

An index mapping block CID to linked block CID.

Characteristics

  • Small - just the link blocks - no data for raw leaves. The majority of files are small files, so the DAGs are not super deep.
  • Root CID independent - it doesn’t matter what the root CID is, we can just iteratively decode each block in a CAR and index the links it has

What is it useful for?

  • Do we have a complete DAG?
    • Communicate upload completeness for DAGs split between multiple CARs
    • PSA - Do not try to fetch the DAG if we have the DAG
    • Trigger for other DAG insights that should only be operated on complete DAGs
  • Elastic IPFS bitswap oversharing.
    • If we store the linking block data in the index (because it is very smol or maybe we just “reinflate” because we know it’s unixfs) we can easily overshare the entire tree for a given root CID using only the index.
    • No going to the block positions table in DynamoDB and no going to S3 to get a block range.
  • Gateway unixfs bucket exporter.
    • A gateway that can easily export file data without IPFS nodes/bitswap.

How to build? 💡

  • Hook into CAR events as trigger
  • Cloudflare KV stores the index
  • Dagula get retrieves and indexes the data
  • Submit CARs for indexing via API
  • Ask completeness via API
  • Cache for repeat asks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment