Skip to content

Instantly share code, notes, and snippets.

@nadineisabel
Created August 19, 2025 08:38
Show Gist options
  • Select an option

  • Save nadineisabel/4b624edb55763883c28149c795b58030 to your computer and use it in GitHub Desktop.

Select an option

Save nadineisabel/4b624edb55763883c28149c795b58030 to your computer and use it in GitHub Desktop.
Includes pagination, filters, and token.
let
// CONFIG
BaseUrl = "https://blue.demo.data-humdata-org.ahconu.org/api/3/action/datastore_search",
ResourceID = "ADD_RESOURCE_ID",
PageSize = 32000,
ApiToken = "ADD_PERSONAL_TOKEN",
OrgFilter = "IOM",
// filters={"org":"IOM"} URL-encoded
FiltersRecord = [ org = OrgFilter ],
FiltersParam = "&filters=" & Uri.EscapeDataString( Text.FromBinary( Json.FromValue( FiltersRecord ) ) ),
// get total rows after filtering
FirstUrl = BaseUrl & "?" & "resource_id=" & ResourceID & "&limit=1" & FiltersParam,
FirstCall = Json.Document( Web.Contents( FirstUrl, [ Headers = [ Authorization = ApiToken ] ] ) ),
TotalRows = try FirstCall[result][total] otherwise 0,
// fetch one page with the same filter
GetPage = (Offset as number) as list =>
let
Url = BaseUrl
& "?"
& "resource_id=" & ResourceID
& "&limit=" & Number.ToText(PageSize)
& "&offset=" & Number.ToText(Offset)
& FiltersParam,
Source = Json.Document( Web.Contents( Url, [ Headers = [ Authorization = ApiToken ] ] ) ),
Records = try Source[result][records] otherwise {}
in
Records,
// paginate and flatten
Offsets = List.Numbers(0, Number.RoundUp(TotalRows / PageSize), PageSize),
Pages = List.Transform(Offsets, each GetPage(_)),
Flattened = List.Combine(Pages),
// to table
TableOut = if List.IsEmpty(Flattened) then #table({}, {}) else Table.FromRecords(Flattened)
in
TableOut
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment