Skip to content

Instantly share code, notes, and snippets.

@davidbuniat

davidbuniat/Eliminate refcount atomic overhead in zero.md

Created January 28, 2026 08:12
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save davidbuniat/6a73aa7c9b94a0569163dd69a19c582c to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save davidbuniat/6a73aa7c9b94a0569163dd69a19c582c to your computer and use it in GitHub Desktop.
Download ZIP
Raw
Eliminate refcount atomic overhead in zero.md

Eliminate refcount/atomic overhead in zero-copy string reads to accelerate TPCH-style workloads

Plan ID: plan-243af5b1

Status: reviewing

Created: 2026-01-25

Updated: 2026-01-25

Goal

Use profiling insights from a TPCH-like analytical workload to reduce end-to-end query execution time by targeting a dominant hotspot in the string read path.

Primary objective: remove atomic/refcount churn on the hot path of per-row string access by switching to a zero-copy, pointer-cached access strategy, validated with unit tests + sanitizers, and verified via perf/flamegraph deltas.

Relevant Files

  • cpp/nd/string_array_holder.hpp
  • cpp/nd/string_array_holder.cpp
  • pg_deeplake/build/deeplake/cpp/deeplake_pg/duckdb_deeplake_scan.cpp
  • pg_deeplake/build/deeplake/cpp/deeplake_pg/table_data.hpp
  • pg_deeplake/build/deeplake/cpp/deeplake_pg/table_data_impl.hpp
  • cpp/chunk_format/impl/chunk_array.hpp
  • cpp/CMakeLists.txt
  • cpp/CMakeLists.tests.cmake
  • factory/docker/benchmark.py

Problem Summary

Profiling indicates a large portion of runtime is spent in atomic operations related to reference counting / shared ownership during repeated string extraction. In the baseline run, the top hotspot was an atomic refcount increment on ARM (acquire/release) and a secondary hotspot was a reference-count base class atomic path.

Interpretation: a per-access helper function returns a string view by passing through a shared-ownership buffer wrapper. Even if the returned view is “zero-copy”, the wrapper copy triggers atomic refcount operations each time, which dominates runtime when executed per element.

Key insight: we can preserve zero-copy semantics while eliminating atomic refcount overhead by extracting and caching raw backing pointers once per batch/segment initialization, then using those cached pointers in subsequent string reads.

Scope / Non-Goals

In scope

  • Removing refcount-related atomics for the project’s string array “holder/streamer” component.
  • Adding minimal unit tests to prevent regressions.
  • Sanitizer validation for lifetime and bounds safety.
  • Perf validation that the refcount atomic hotspot is eliminated.

Not in scope (for this iteration)

  • Optimizing downstream query engine internals (e.g., engine-owned buffers, vector caches, destructors).
  • Achieving a guaranteed 2× reduction in end-to-end TPCH time (may be bounded by other engine overhead).

Architecture Context (sanitized)

We have a String Holder abstraction that serves string values to a scan/operator layer. It supports two underlying layouts:

  1. Range/offset-backed strings: strings stored in a contiguous buffer with an offset table.
  2. Dynamic strings: strings stored as variable chunks/arrays, accessed through an indirection layer.

Baseline hot path characteristics:

  • Per-element get_string(i) calls cause a shared-ownership wrapper to be copied.
  • That copy triggers atomic refcount operations at very high frequency.

Optimized hot path strategy:

  • During holder initialization (per batch/segment), extract:
    • pointer to raw data buffer
    • pointer to offsets buffer (if applicable)
    • base offset/stride info (if applicable)
  • Store these as a read-only cache inside the holder.
  • For dynamic strings, maintain a cache of (char*, length) pairs (or equivalent) for zero-copy views.
  • Ensure a safe fallback path exists for unsupported adapters/layouts.

Proposed Implementation Approaches

A) Verify and complete existing pointer-caching optimization (RECOMMENDED)

Assumption (based on internal prototype): a pointer-caching implementation exists and needs verification, tests, comments, and safety validation.

Work

  1. Add a minimal unit test suite for the String Holder API (range + dynamic).
  2. Run a profiling sanity check to confirm refcount/atomic samples collapse to near-zero.
  3. Validate initialization validity logic (no false-invalid states).
  4. Document that the “fast” sequential access path also covers dynamic strings (even if not inlined).
  5. Run ASAN/UBSAN to validate lifetime/bounds safety.
  6. Commit with success criteria focused on eliminating the holder-level atomic hotspot.

Complexity: low

Risk: low-to-medium (lifetime safety must be proven)

B) Switch scan/operator integration to the sequential fast-path

If the scan layer accesses strings sequentially within a batch, route reads through a get_string_fast(i) API that avoids per-access searching (e.g., binary search on ranges).

Complexity: low

Risk: low

Expected impact: marginal unless search overhead is visible after atomic removal

C) SIMD accelerate offset processing (only if it becomes the next bottleneck)

After atomics are removed, check whether offset computation (not memory access) dominates. If yes, implement vectorized offset decoding (NEON/AVX).

Complexity: medium

Risk: low

Expected impact: only meaningful if offset compute shows up prominently in profiling

Selected Approach and Rationale

Select A (Verify + test + validate pointer-caching optimization)

Rationale:

  • The baseline profile shows refcount/atomic operations dominate the string holder’s hot path.
  • Caching raw pointers once during initialization avoids repeated shared-ownership wrapper copies.
  • This targets the dominant overhead with minimal code change and is testable and measurable.

Implementation Steps

Step 1: Add minimal unit tests for the String Holder

Create a focused test file covering:

  • Range/offset-backed strings:
    • correctness of get_string(i)
    • correctness of get_string_fast(i) for sequential access
    • correctness of get_batch(start, count) style API
  • Dynamic strings:
    • correctness of zero-copy views (pointer/length stable within owner lifetime)
    • fallback path correctness if caching is not applicable

Test cases

  • Small array with known strings (including empty string and long string).
  • A batch boundary case (transition between batches/segments).
  • Dynamic strings with varying lengths.

Pass criteria

  • Results match expected values.
  • Fast and slow APIs return identical content for sequential access patterns.
  • Batch API populates output buffers correctly.

Step 2: Profiling sanity check on optimized code

Run the benchmark workload with perf sampling and generate a collapsed stack trace / flamegraph.

Check

  • Search for the baseline atomic/refcount symbol(s) dominating the hot path.
  • Expect them to drop from “dominant” to “near-zero.”

Pass criteria

  • Atomic/refcount operations no longer appear as a significant fraction of samples in the string holder read path.
  • The string holder read stack is now dominated by memory access / offset decode rather than refcount churn.

(Public note: exact symbol names vary by platform and toolchain; we validate by comparing before/after flamegraphs.)

Step 3: Validate holder initialization “validity” logic

Review the initialization flow to ensure:

  • successful initialization sets is_valid = true
  • fallback initialization also sets is_valid = true when it is intended to be usable
  • failure paths clear caches and do not leave partially initialized pointers

Pass criteria

  • No successful initialization leaves is_valid false.
  • Fallback is selected only when caching is not applicable.

Step 4: Add documentation comment clarifying fast-path coverage

If get_string_fast(i) uses a generic helper for dynamic strings (not inlined), add a comment explaining:

  • range strings: served directly from cached buffer + offsets
  • dynamic strings: served via cached (ptr, len) entries (or equivalent) even if the code path looks indirect

Pass criteria

  • Comment accurately describes behavior.
  • No logic change required.

Step 5: Run sanitizers to validate memory safety

Because we cache raw pointers, lifetime must be correct:

  • backing storage must outlive the holder
  • no out-of-bounds reads
  • no use-after-free

Build

  • Debug + ASAN (and optionally UBSAN)

Run

  • new unit tests (string holder tests)
  • existing unit tests (at least a smoke run)

Pass criteria

  • No sanitizer reports.

Step 6: Commit with a performance-focused message (public-safe)

Commit message should include:

  • what was slow (refcount atomics in per-element string reads)
  • why (shared-ownership wrapper copied per access)
  • what changed (cached raw pointers / cached dynamic string views during initialization)
  • what it affects (holder-level overhead)
  • what it does not affect (downstream engine overhead)

Step 7: Full benchmark profiling run + report

Run the benchmark at the same scale/config as baseline (warmup + multiple measurements), generate flamegraph, and compare:

Primary metric

  • Holder-level atomic/refcount hotspot: “dominant” → “near-zero”

Secondary metrics

  • Query time improvement (expect improvement, but not necessarily 2×)
  • If overall speedup is limited, identify next hotspots (often downstream engine internals)

Step 8: Integration lifetime review

Review scan/operator integration to ensure holder lifetime is safe:

  • holder does not outlive the backing array/buffer owner
  • batch/segment transitions correctly refresh caches (if applicable)

Pass criteria

  • No cached pointer used after backing owner is destroyed.
  • Batch switching calls cache reset/refresh where required.

Step 9 (Optional): Switch integration to sequential fast-path

Only do this if profiling shows meaningful time in search/dispatch overhead after atomics are removed.

Change

  • Replace per-element calls with get_string_fast(i) within a sequential batch loop.
  • Ensure cache reset occurs at batch boundaries.

Pass criteria

  • Correctness maintained.
  • Profiling shows reduced search overhead (if it was visible).

Risks and Mitigations

Risk: Cached raw pointers outlive backing storage (use-after-free)

Mitigation

  • Sanitizers (ASAN/UBSAN)
  • Integration lifetime review
  • Unit tests covering batch boundary transitions

Risk: Not all string layouts are cacheable

Mitigation

  • Keep a correct fallback path
  • Add tests for fallback behavior

Risk: End-to-end speedup limited by downstream engine overhead

Mitigation

  • Define success as elimination of holder-level atomic hotspot
  • Use post-opt profiling to decide next targets

Success Criteria

REQUIRED

  • The baseline refcount/atomic hotspot in the string holder read path is reduced to near-zero in perf/flamegraphs.
  • New unit tests for range + dynamic strings pass.
  • Sanitizers report no errors.
  • No regressions in existing test suite.

EXPECTED

  • Some measurable improvement in benchmark runtime, but magnitude may be bounded by downstream components.

NICE-TO-HAVE

  • If a new dominant hotspot emerges (e.g., engine buffer lifecycle), document it as the next optimization candidate.

Verification Commands (generic, public-friendly)

Replace these with your project’s exact build and benchmark commands.

  1. Build tests
cmake -S . -B build -DENABLE_TESTS=ON
cmake --build build -j
ctest --test-dir build
  1. Sanitizer build
cmake -S . -B build-asan -DENABLE_TESTS=ON -DENABLE_ASAN=ON -DCMAKE_BUILD_TYPE=Debug
cmake --build build-asan -j
ctest --test-dir build-asan
  1. Perf profile baseline vs optimized
# Baseline
./run_benchmark --profile --warmup 2 --runs 30 --out baseline_profile

# Optimized
./run_benchmark --profile --warmup 2 --runs 30 --out optimized_profile

# Compare flamegraphs / collapsed stacks
# (Exact tools vary; the goal is “atomic/refcount hotspot disappears”.)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment