Skip to content

Instantly share code, notes, and snippets.

@dskvr
Last active March 28, 2024 14:56
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save dskvr/abccd012f90f9771fc53fc173e005f33 to your computer and use it in GitHub Desktop.
Save dskvr/abccd012f90f9771fc53fc173e005f33 to your computer and use it in GitHub Desktop.
maybe some useful bits in here for anyone who feels like entering the battle arena.

NIP-AND-FILTERS

Enable AND within a single tag filter by using an & modifier in filters for indexable tags.

filters: {
  "kinds": [1],
  "&t": ["meme", "cat"],
  "#t": ["black", "white"]
}
// returns kind `1` events with `t` tags that have both "meme" and "cat" that have the tag "black" or "white"

Rationale

  • Reduce bandwidth for all, with an emphasis on mobile users: meme AND cat objectively consumes less bandwidth than meme OR cat
  • Reduce clock-time for relays, indexing with AND is fast for all common index formats, and faster compared to OR for some index formats. (See section below)
  • Reduce client-side caching requirements
  • Reduce centralization vectors by reducing or even eliminating the need for centralized REST, GraphQL APIs or specialized relay "feed" endpoints.
  • Give relays the option to be more useful at the protocol level while improving efficiency for all parties.

Rules

  • AND MUST take precedence over OR
  • Tags used in AND SHOULD NOT be used in standard OR tags [#]
  • Any tag used in AND SHOULD be ignored in OR

Considerations

  • New field for NIP-11.limitations: max_tags_per_and and max_tags_and
  • Benchmarking should be conducted to validate that bandwidth and protocol usability as benefits supercede implementation and clock-time cost.

Index Efficiency

Index Type AND Operation Efficiency OR Operation Efficiency Notes
B-Tree High Moderate B-Tree indexes are very efficient for AND operations, especially with compound indexes. For OR operations, they are less efficient than for AND, as the database engine might need to traverse multiple paths.
Bitmap High High Bitmap indexes excel in both AND and OR operations, particularly for columns with low cardinality. They utilize fast bitwise operations, making them ideal for read-heavy environments.
Hash Not Applicable Not Applicable Hash indexes are designed for equality checks and do not directly support range-based queries or optimize for AND/OR operations efficiently.
Full-Text High High Optimized for text search, full-text indexes efficiently handle both AND and OR conditions, making them suitable for complex text queries.
@dskvr
Copy link
Author

dskvr commented Mar 28, 2024

Screen Shot 2024-03-28 at 3 23 13 AM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment