Skip to content

Instantly share code, notes, and snippets.

@dbreunig
Created June 26, 2025 20:21
Show Gist options
  • Select an option

  • Save dbreunig/b3bdd9eb34bc264574954b2b954ebe83 to your computer and use it in GitHub Desktop.

Select an option

Save dbreunig/b3bdd9eb34bc264574954b2b954ebe83 to your computer and use it in GitHub Desktop.

Alameda and Bay Farm Islands are connected via the Bay Farm Island Bridge, and the Bay Farm Island Bicycle Bridge (the only pedestrian/bicycle-only drawbridge in the United States[^70]). [^71] California State Route 61 runs down city streets from the Posey and Webster Street Tubes, across the Bay Farm Island Bridge, and south to the Oakland Airport. The island is just minutes off Interstate 880 in Oakland.

Transportation options include:

@joshyim
Copy link
Copy Markdown

joshyim commented Jul 24, 2025

Loved the article.
Instead of Provenance "nailing it", this is an example of difficulty in extracting the salient information, as the Provenance missed 3 bridges and 1 tube connections. Immediately preceding "Alameda and Bay Farm Islands..." text that it extracted, the wiki article contains, "Vehicle access to Alameda Island is via three bridges from Oakland (Park Street, Fruitvale Avenue, and High Street Bridges), as well as the two one-way Posey and Webster Street Tubes leading into Oakland's Chinatown."

https://en.wikipedia.org/wiki/Alameda,_California#Transportation

@dbreunig
Copy link
Copy Markdown
Author

It’s hard!

Good catch. I only count 2 bridges missed; opening graph covers the tubes and two bridges.

@joshyim
Copy link
Copy Markdown

joshyim commented Jul 24, 2025

Sorry for being pedantic. However, those are actually another example of failure modes, aren't they?
The extracted information is overemphasizing connection between Aladema island and Bay Farm Island. The bigger connection points are omitted, thereby creating incorrect information.

  • 3 bridges omission
    There are 5 connections that connects Alameda. You can see this visually on the wiki article. (see screenshot from the wiki article)
Screenshot 2025-07-24 at 16 11 59

Alameda Island and connections:

  1. Posey and Webster Street tubes
  2. Park Street Bridge
  3. Fruitvale Bridge
  4. High Street Bridge
  5. Bay Farm Island Bridge

The extracted information missed bridges 2, 3, and 4. The three I mentioned in the message above. On a side note, the extraction emphasized the fact that #5 is in fact two - 1. car and 2. pedestrian/bike only - bridges, which is technically not what the original intent of the code.

  • Incorrect Extracted Info about the Posey and Webster Street Tube
    As for the Street Tubes, the extracted info does not explicitly mention if the Tube connects Alameda off-island. It just says Tube is connected to CA61 which in turn connected to to Bay Farm island. If I use this extracted snippet as the only input context to LLM to figure out a way off the island, I am not very confident that LLM will generate a valid output that Street Tubes is a valid way off of Alameda, since the extracted text is so roundabout as opposed to the original information which is very conclusive. If this info is a part of RAG pipeline, I'm pretty sure it will yield lower relevance score and likely be rejected even before getting into the context window.

Thoughts?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment