RobinL/splink_5_changes_so_far.md Secret

## splink_5_changes_so_far.md

      
    Raw
  

              splink_5_changes_so_far.md
            
          
    What's New in Splink 5: upgrading from Splink 4

Valid through commit
2d57cd67e174a075c63db9904646c80c6613d727
(10 September 2026), checked against master on 15 September 2026.
This guide covers the changes and migration steps for upgrading from Splink 4
to Splink 5, including new features, API changes, and development workflows.
Headline changes


Inputs must be registered as SplinkDataFrames before they are passed to
the Linker. Pandas/Arrow tables are no longer accepted as direct inputs.
pandas and numpy are no longer required dependencies. Splink's
default install is now much smaller and works in pandas-free environments.
Internal calculations now use match weights, not Bayes factors. Output
column prefixes change from bf_ to mw_.
predict() now supports chunking for very large datasets via
num_chunks_left / num_chunks_right, with per-chunk progress logging and
runtime estimates. DuckDB now also prunes the source records used to score
each chunk, reducing work in the post-blocking joins.
Large prediction jobs can be distributed across machines. predict_chunk()
scores one self-contained slice of the grid, and there is an explicit
compute / register / predict workflow for persisting blocked pairs.
estimate_u_using_random_sampling() is faster thanks to chunked
processing with early stopping via min_count_per_level.
Blocking analysis is faster and has a new API. Comparison counts are now
estimated from a record sample by default using
record_sample_proportion, and the public functions have been renamed or
consolidated.
EM training and prior estimation are faster.
estimate_parameters_using_expectation_maximisation() gains max_pairs,
and estimate_probability_two_random_records_match() gains
record_sample_proportion.
The ad-hoc scoring API has been reworked.
compare_two_records() is replaced by score_pair() and score_pairs(),
find_matches_to_new_records() is removed, and new
predict_within() / predict_between() methods score blocked pairs over
arbitrary tables.
as_record_dict() is renamed to as_record_list(). It still returns
list[dict], but the Splink 4 method name is removed.
DuckDB can materialise results directly to local Parquet files. The
new opt-in backend setting applies to intermediate and final SQL results
and works with chunking and profiling.
Charts and profiling have improved. profile_columns() now returns a
SplinkChart with individually accessible subcharts, DuckDB comparison
viewer generation is faster, and DuckDB query profiling defaults to JSON.
Spark supports cosine similarity comparisons over numeric arrays.
The calculation uses native Spark SQL array operations.


1. The new input contract: db_api.register() everywhere

What changed

In Splink 4, you could pass a pandas.DataFrame, a string table name, or other
raw inputs directly to the Linker. In Splink 5, you must first register your
input data with the database API to obtain a SplinkDataFrame. The Linker
only accepts SplinkDataFrames.
The Linker.__init__ signature now reads:
def __init__(
    self,
    splink_dataframe_or_dataframes: SplinkDataFrame | Sequence[SplinkDataFrame],
    settings: SettingsCreator | dict[str, Any] | Path | str,
    log_level: int | str | None = logging.INFO,
    validate_settings: bool = True,
): ...
The db_api argument has been removed from the constructor. The database API is
now derived from the registered SplinkDataFrame or SplinkDataFrames passed
to the Linker. (The set_up_basic_logging argument has also been replaced by
log_level - see the logging section below.)
If you pass a raw pandas.DataFrame, pyarrow.Table, dict, list, or other
unregistered object where a SplinkDataFrame is expected (the Splink 4 style),
Splink now raises a clear, actionable TypeError that tells you to register the
data first:
Expected a SplinkDataFrame but received a pandas.core.frame.DataFrame.
From Splink 5 onwards, input data must be registered with the database API
before it is passed to Splink. ...
    df = db_api.register(my_data, dataset_display_name="my_dataset")
    linker = Linker(df, settings)

db_api.register() now distinguishes between:

dataset_display_name - the human-readable dataset label that populates the
source_dataset column in outputs
table_name - the internal templated table name used inside Splink

In many cases you will only provide dataset_display_name. If you need precise
control over the registered table name, you can also pass table_name.
Why it matters


The lifecycle of input tables is explicit.
Source-dataset names for link_only / link_and_dedupe are passed at
registration time, not derived from positional ordering.
The same registration pattern works across pandas, Arrow, DuckDB, Spark,
dict/list inputs, and existing backend tables.
This change enables Splink 5's pandas-free workflows.

Example

import duckdb
import pandas as pd
from splink import DuckDBAPI, Linker, SettingsCreator, block_on
from splink.comparison_library import ExactMatch

con = duckdb.connect()
db_api = DuckDBAPI(con)

df_a = db_api.register(
    pd.read_parquet("people_2020.parquet"),
    dataset_display_name="people_2020",
)

df_b = db_api.register(
    pd.read_parquet("people_2024.parquet"),
    dataset_display_name="people_2024",
)

settings = SettingsCreator(
    link_type="link_only",
    blocking_rules_to_generate_predictions=[block_on("first_name", "surname")],
    comparisons=[ExactMatch("first_name"), ExactMatch("surname")],
)

linker = Linker([df_a, df_b], settings)
db_api.register() accepts:

a string, interpreted as an existing table already in the backend
a pandas.DataFrame, if pandas is installed
a pyarrow.Table
a dict[str, list] or list[dict] (the conversion uses PyArrow)
a backend-native object, such as duckdb.DuckDBPyRelation or
pyspark.sql.DataFrame

String registration refers to an existing backend table. For a Parquet file,
register a relation created by the same DuckDB connection, for example
db_api.register(db_api.duckdb_con.read_parquet("people.parquet")).
Related: PR #2863, #2865, #2866, #3109.

2. Pandas and NumPy are now optional

What changed

pandas and numpy are removed from Splink's required dependencies. You can
now install Splink without pandas:
pip install 'splink>=5,<6'            # pandas not required
pip install 'splink>=5,<6' pandas     # add pandas when your code needs it
pip install 'splink[pyarrow]>=5,<6'   # Arrow inputs/outputs and dict/list registration
To make this possible, internal code paths that previously materialised
intermediate results into pandas have been rewritten to use:

as_record_list() — pure-Python list of dicts, no pandas required
as_dict() — columnar dict-of-lists, always available
as_pyarrow_table() — Arrow table, requires PyArrow
as_pandas_dataframe() — still available, but requires pandas

At the commit covered here, pandas is a development dependency group, not
a published splink[pandas] extra. PyArrow is also optional, exposed through
splink[pyarrow]. Install the libraries needed by your chosen input/output
formats explicitly; a pandas-free workflow can still use Arrow.
This affects training, prediction, charts, dashboards, cluster studio, accuracy
charts, completeness charts, profile-columns charts, and debug-mode
pretty-printing.
Why it matters


Splink installs are smaller.
Splink can run in environments where pandas is not available or not desired.
Fewer dependency conflicts in shared environments such as Spark clusters or
JupyterHub deployments.
Default outputs avoid unnecessary data movement out of the database.

Example

from splink import DuckDBAPI, Linker, SettingsCreator, block_on
from splink.comparison_library import ExactMatch
import pyarrow.parquet as pq

db_api = DuckDBAPI()
arrow_tbl = pq.read_table("people.parquet")

df = db_api.register(arrow_tbl, dataset_display_name="people")

linker = Linker(
    df,
    SettingsCreator(
        link_type="dedupe_only",
        blocking_rules_to_generate_predictions=[block_on("first_name")],
        comparisons=[ExactMatch("first_name"), ExactMatch("surname")],
    ),
)

linker.training.estimate_u_using_random_sampling(max_pairs=1e6)
df_predict = linker.inference.predict(threshold_match_probability=0.9)

records = df_predict.as_record_list(limit=10)
arrow = df_predict.as_pyarrow_table()
ddbrel = df_predict.as_duckdbpyrelation()
Related: PR #2883, #2937, #2956, #2969, #2970, #2985, #2987, #2916, #3150.

3. New SplinkDataFrame outputs and the as_record_list() rename

What changed

SplinkDataFrame now exposes a richer set of materialisation methods:


Method
Returns
Requirements


as_record_list(limit=None)
list[dict]
no pandas or Arrow needed for this conversion


as_dict(limit=None)
dict[str, list]
no pandas or Arrow needed for this conversion


as_pyarrow_table(limit=None)
pyarrow.Table
PyArrow installed


as_pandas_dataframe(limit=None)
pandas.DataFrame
pandas installed


as_duckdbpyrelation(limit=None)
duckdb.DuckDBPyRelation
native relation on DuckDB


as_spark_dataframe()
pyspark.sql.DataFrame
Spark backend


Breaking rename: PR #3150 removes as_record_dict() and replaces it with
as_record_list() across all backends, internal code, tests, and examples.
There is no compatibility alias. The return shape and optional limit
argument are unchanged:
# Splink 4
records = df_predict.as_record_dict(limit=10)

# Splink 5
records = df_predict.as_record_list(limit=10)
Keep as_dict() when you want a columnar dictionary; it is a different format
and has not been renamed.
as_duckdbpyrelation() is especially useful because it lets you continue
working with large outputs without leaving the database.
SplinkDataFrame also gains a query_sql() method, so you can run further SQL
against an output table and get back another SplinkDataFrame without going via
linker.misc.query_sql(). Refer to the table itself as {this} (escape the
braces as {{this}} inside an f-string):
df_predict = linker.inference.predict(threshold_match_probability=0.9)
top = df_predict.query_sql(
    "select unique_id_l, unique_id_r, match_weight "
    "from {this} where match_weight > 5"
)
Example

df_predict = linker.inference.predict(threshold_match_probability=0.9)

rel = df_predict.as_duckdbpyrelation()
high_score = rel.filter("match_probability > 0.99").project(
    "unique_id_l, unique_id_r"
)
high_score.to_parquet("high_confidence_pairs.parquet")
With DuckDB's new Parquet materialisation mode, the SplinkDataFrame wraps a
view over backing files. These access methods and query_sql() still work;
see section 5d for configuration and the lifetime of those files.
Related: PR #2916, #3150, #3274.

4. linker.misc.query_sql() returns a SplinkDataFrame by default

What changed

Previously, query_sql() defaulted to returning a pandas.DataFrame. In
Splink 5, it returns a SplinkDataFrame by default. Pass
output_type="pandas" for the old behaviour.
# Splink 4 default
df = linker.misc.query_sql("select * from __splink__df_predict limit 100")
# -> pandas.DataFrame

# Splink 5 default
sdf = linker.misc.query_sql("select * from __splink__df_predict limit 100")
# -> SplinkDataFrame

# Old behaviour in Splink 5
df = linker.misc.query_sql("select ...", output_type="pandas")
Why it matters


Aligns with the pandas-optional architecture.
Avoids accidental materialisation of large query results into Python memory.
Allows further SQL operations without an extra round trip.

The backend-level db_api.query_sql(...) also returns a SplinkDataFrame.
With DuckDBAPI(materialisation="parquet", ...), both backend SQL queries
and sdf.query_sql(...) materialise their results using Parquet-backed views
(section 5d).
Related: PR #2970, #3274.

5. Chunking and Parquet materialisation for large jobs

This is one of the most important changes for users working with large data.

5a. Chunked prediction

What changed

predict() and the new predict_chunk() can partition both sides of the input
into deterministic, hash-based chunks. This allows prediction jobs to be split
into smaller pieces.
Two new arguments are available on predict():
def predict(
    self,
    threshold_match_probability: float | None = None,
    threshold_match_weight: float | None = None,
    num_chunks_left: int | None = None,
    num_chunks_right: int | None = None,
    warning_mode: PredictUntrainedWarningMode = "auto",
) -> SplinkDataFrame: ...
If num_chunks_left=L and num_chunks_right=R, Splink iterates through all
L * R chunk pairs, executes a complete predict pipeline for each, logs
progress, and finally UNION ALLs the results into a single output
SplinkDataFrame.
Chunking is implemented as a deterministic hash filter on the composite
unique ID:
chunk_expr = f"(ABS({hash_expr}) % {num_chunks}) + 1"
chunk_filter = f" AND {chunk_expr} = {chunk_num}"
The hash filter limits which source records enter each blocking join. Chunk
sizes can differ because the records and their blocking keys are not evenly
distributed.
New DuckDB optimisation (PR #3267): for effective chunks (either side has
more than one chunk), Splink also materialises separate left/right subsets of
the source records that actually appear in the blocked pairs. It then joins
TF values and computes comparison vectors from these smaller tables. This
avoids scoring joins over the full source for every chunk and can reduce peak
memory and runtime. The subsets are dropped after scoring.
This happens automatically on DuckDB for chunked predict() and
predict_chunk(); there is no new argument. Ordinary unchunked prediction
keeps the existing path. Source pruning also applies to manually registered
blocked pairs (section 5c). Other backends keep their existing execution path.
Example

# Lower peak memory, with per-chunk progress logging; the result is
# identical to an unchunked predict()
df_predict = linker.inference.predict(
    threshold_match_weight=0,
    num_chunks_left=4,
    num_chunks_right=4,
)
Chunks are processed in series and UNION ALLed together, so the returned
SplinkDataFrame is exactly the same as an unchunked predict(). Because the
chunks run one after another, Splink logs progress and a running runtime
estimate after each one (at the default INFO log level):
Processing chunk (1, 4) x (1, 4) [1/16]
Completed chunk 1/16 (6%) | Elapsed: 60.0s | Remaining: ~900.0s | Total: ~960.0s

Once blocked pairs have been manually registered (see 5c), the
num_chunks_left / num_chunks_right arguments are not available, because
Splink cannot re-chunk a table you have already materialised.

5b. Single-chunk prediction and distributing across machines

What changed

predict_chunk() computes and scores a single slice of the chunk grid. It is
the building block behind chunked predict(), but exposing it directly enables
two things:

Verifying a pipeline cheaply. Run one chunk end to end before committing
to a long full run.
Distributing a job across machines. Because each predict_chunk() call is
completely self-contained, different chunks can be scored on different
machines - even with DuckDB, which is otherwise single-node. Each worker loads
the same saved model and input data, scores its assigned
(left_chunk, right_chunk) slice, and writes the output to shared storage. A
coordinator then unions the per-chunk outputs.

predict_chunk() is for letting Splink compute the blocking for a chunk itself.
It is not used with manually registered blocked pairs - use predict() for that
(see 5c).
Example

# Verify a single slice runs end to end
single_chunk = linker.inference.predict_chunk(
    left_chunk=(1, 4),
    right_chunk=(1, 4),
    threshold_match_probability=0.9,
)

# Distributed: each worker rebuilds the linker from a shared model + data,
# scores its assigned chunk, and writes it out. Use the same backend/version,
# unique IDs, dataset labels, model, input snapshot, and grid on every worker.
db_api_worker = DuckDBAPI()
df_worker = db_api_worker.register(
    db_api_worker.duckdb_con.read_parquet("people.parquet"),
    dataset_display_name="people",
)
linker_worker = Linker(df_worker, "model.json")

chunk_preds = linker_worker.inference.predict_chunk(
    left_chunk=(1, 4),
    right_chunk=(1, 4),
    threshold_match_probability=0.9,
)
chunk_preds.as_duckdbpyrelation().to_parquet("predictions_chunk_1_1.parquet")
There is no built-in scheduler: you trigger the per-chunk jobs and union their
outputs yourself.

5c. Persisted blocked-pairs workflow

What changed

For the very largest jobs you can separate blocking from scoring, persisting
the blocked pairs (the candidate record-id pairs) between the two steps. This is
a niche technique - reach for it only when scoring even a single chunk in one
pass is too large.
Splink 5 provides three methods for this workflow (PR #3127, #3128):

compute_blocked_pairs_for_predict() materialises the full blocked-pairs
table; compute_blocked_pairs_for_predict_chunk(left_chunk, right_chunk)
materialises the pairs for a single chunk.
register_blocked_pairs_for_predict(blocked_pairs) registers a previously
computed blocked-pairs table back into a Linker. It takes a single
SplinkDataFrame argument. Register the raw data with db_api.register()
first to obtain the SplinkDataFrame.
predict() then scores exactly the registered table - no re-blocking.

Once blocked pairs are registered, the chunking arguments on predict() and the
predict_chunk() method are unavailable (Splink cannot own chunking of a table
you have already materialised); calling them raises a SplinkException.
Example

# Job 1: compute and persist blocked pairs
blocked_pairs = linker.inference.compute_blocked_pairs_for_predict()
blocked_pairs.as_duckdbpyrelation().to_parquet("blocked_pairs.parquet")

# Job 2: in a separate session / machine, register and score them
blocked_pairs = db_api.register(
    db_api.duckdb_con.read_parquet("blocked_pairs.parquet")
)
linker.table_management.register_blocked_pairs_for_predict(blocked_pairs)
predictions = linker.inference.predict()
To split blocking itself into chunks, use
compute_blocked_pairs_for_predict_chunk(left_chunk=..., right_chunk=...) and
persist each chunk separately.
Why it matters


Blocking and scoring can run as separate jobs in a DAG.
Blocked pairs can be reused across separate scoring jobs.
Registration takes a single SplinkDataFrame, with no chunk bookkeeping
to keep in sync.
On DuckDB, scoring registered pairs now automatically prunes the source
records separately on each side before comparison-vector generation,
whether the registered table contains one chunk or the full set of pairs.

Related: PR #2850, #2915, #2957, #3127, #3128, #3267.

5d. DuckDB can materialise SQL results directly to Parquet

What changed

DuckDBAPI gains three keyword-only constructor arguments:

materialisation="table" (the default) or "parquet".
materialisation_dir: a required local directory in Parquet mode.
parquet_materialisation_options: an optional ParquetWriteOptions object.

Parquet mode executes COPY (query) TO ... (FORMAT PARQUET) and exposes the
files through a DuckDB view. It applies to SQL results throughout the backend,
including training intermediates, blocked pairs, chunk outputs, predictions,
and query_sql() results. Registering an existing input still registers that
input; the mode controls how Splink materialises subsequent SQL results.
Why it matters

Writing large results directly to Parquet can be faster when an in-memory
DuckDB connection would otherwise hit memory limits while storing them. It
can be combined with chunking and source pruning. Query execution still needs
memory and sufficient disk space, so this is an option to measure on your
workload rather than a guarantee that any size of job will fit.
Example

from splink import DuckDBAPI, Linker
from splink.backends.duckdb import ParquetWriteOptions

db_api = DuckDBAPI(
    materialisation="parquet",
    materialisation_dir="splink_working_data",
    parquet_materialisation_options=ParquetWriteOptions(
        compression="zstd",
        per_thread_output=True,
        file_size_bytes="512MB",
    ),
)
df = db_api.register(
    db_api.duckdb_con.read_parquet("people.parquet"),
    dataset_display_name="people",
)
linker = Linker(df, "model.json")
preds = linker.inference.predict(num_chunks_left=4, num_chunks_right=4)
records = preds.as_record_list(limit=10)

# Export a result that should survive cleanup of the working data.
preds.as_duckdbpyrelation().to_parquet("predictions.parquet")
preds.drop_table_from_database_and_remove_from_cache()
ParquetWriteOptions supports compression, compression_level,
per_thread_output (default True), file_size_bytes, and row_group_size.
The remaining options default to None, leaving DuckDB's writer defaults in
place. Passing a Parquet directory or options with materialisation="table"
raises ValueError; remote URLs such as s3://... are not accepted for the
working directory.
Splink creates a separate workspace and per-result directories under the
chosen directory. Dropping a Splink-owned result also deletes its backing
files. db_api.delete_tables_created_by_splink_from_db() cleans up tracked
results, leaving unrelated user files alone. Export results you want to keep
before cleaning up; this working directory is not the persisted/distributed
output contract from section 5c.
DuckDBAPIWithProfiling accepts the same options and profiles the actual
COPY operation (section 12). The comparison viewer uses its general SQL path
in Parquet mode because views do not expose DuckDB table rowids (section 18).
Related: PR #3274.

6. estimate_u_using_random_sampling() is much faster

What changed

The u-estimation routine was rewritten to:

Process the right-hand-side sample in chunks.
Accumulate per-comparison-level u-counts into an in-Python
_MUCountsAccumulator.
Stop early as soon as every comparison level has reached
min_count_per_level observations.
Use a probe phase that runs a small chunk first to detect whether early
stopping is likely to occur.

The new signature is:
def estimate_u_using_random_sampling(
    self,
    max_pairs: float = 1e6,
    seed: int | None = None,
    min_count_per_level: int | None = 100,
    num_chunks: int = 10,
) -> None: ...
Why it matters

For many datasets, every comparison level has enough observations after a small
fraction of max_pairs has been processed. Early stopping means the computation
can finish once the least-observed level has crossed min_count_per_level,
rather than always processing the full sample.
The chunked execution also makes u-estimation memory-bounded in the same way as
chunked prediction.
Example

# Default: stops as soon as every level has 100+ observations
linker.training.estimate_u_using_random_sampling(max_pairs=5e7)

# Splink 4-style behaviour: process all of max_pairs
linker.training.estimate_u_using_random_sampling(
    max_pairs=5e7,
    min_count_per_level=None,
)

# More conservative threshold
linker.training.estimate_u_using_random_sampling(
    max_pairs=5e7,
    min_count_per_level=1000,
    num_chunks=20,
)
Deterministic sampling across supported backends

The random sample used by estimate_u_using_random_sampling() (and by the
sampling used in clustering) is now drawn with a deterministic hash filter on
the unique-ID columns rather than backend-specific TABLESAMPLE-style syntax.
The same approach is used across DuckDB, Spark, Postgres, and SQLite, but each
backend has its own hash implementation. For fixed IDs, data, seed, and backend
version, sampling is repeatable; the same seed does not imply identical
sample membership across different backends. The sampling modulus is large
(1e9) while remaining within the range of 32-bit hashes.
Disabling early stopping processes the full sampled workload; it does not
recreate the exact random sample or estimates from Splink 4.
Related: PR #2870, #3025, #3122.

7. max_pairs for EM training

What changed

estimate_parameters_using_expectation_maximisation() gains two new
keyword-only arguments that let you cap the size of an EM training run:
def estimate_parameters_using_expectation_maximisation(
    self,
    blocking_rule,
    estimate_without_term_frequencies: bool = False,
    fix_probability_two_random_records_match: bool = False,
    fix_m_probabilities: bool = False,
    fix_u_probabilities: bool = True,
    populate_probability_two_random_records_match_from_trained_values: bool = False,
    *,
    max_pairs: float | None = None,
    record_sample_proportion: float = 0.01,
) -> EMTrainingSession: ...
If max_pairs is set, Splink first runs a cheap, record-sampled blocking pass
using record_sample_proportion of records on each side. It uses this to
estimate the full blocked-pair count for the training rule. Splink then applies
a deterministic hash-based filter to the input records so that the number of
blocked pairs scored during EM is approximately max_pairs.
Why it matters

Some training blocking rules generate far more comparisons than EM needs to
converge. In Splink 4, the main way to reduce this was to make the blocking rule
tighter, which also changed which records were compared. With max_pairs, the
same blocking rule can be kept while bounding the amount of work.
Example

br_training = block_on("first_name", "dob")

linker.training.estimate_parameters_using_expectation_maximisation(
    br_training,
    max_pairs=1e6,
)

# Splink 4-style behaviour: no cap
linker.training.estimate_parameters_using_expectation_maximisation(br_training)
Related: PR #3050, #3090.

8. Blocking analysis: faster, sampled, and a reworked API

What changed

Blocking analysis has been refactored. The main change is a much faster way to
estimate the number of pairs a blocking rule will generate by sampling the input
rows. The public API has also been consolidated.
New or renamed public functions in splink.blocking_analysis:
from splink.blocking_analysis import (
    count_comparisons_from_blocking_rules,
    chart_comparisons_from_blocking_rules,
    n_largest_blocks,
)
Removed public functions and their replacements:


Splink 4 function removed in Splink 5
Replacement in Splink 5


count_comparisons_from_blocking_rule
count_comparisons_from_blocking_rules


cumulative_comparisons_to_be_scored_from_blocking_rules_data
count_comparisons_from_blocking_rules


cumulative_comparisons_to_be_scored_from_blocking_rules_chart
chart_comparisons_from_blocking_rules


There is also a new linker.blocking_analysis component:
linker.blocking_analysis.count_comparisons_from_blocking_rules()
linker.blocking_analysis.chart_comparisons_from_blocking_rules()
linker.blocking_analysis.n_largest_blocks(...)
These methods default to the linker's input tables, link type, unique-ID column,
source-dataset column, and blocking_rules_to_generate_predictions.
When overriding rules on the linker component's count/chart methods, supply
an iterable, even for one rule: blocking_rules=[block_on("email")].
The standalone functions below accept a single rule or an iterable of rules.
count_comparisons_from_blocking_rules() now accepts either a single blocking
rule or an iterable, and always returns a list of records. Each record reports:

marginal_comparison_count
cumulative_comparison_count
equi-join conditions
filter conditions
link-type conditions

Sampling support

count_comparisons_from_blocking_rules() and
chart_comparisons_from_blocking_rules() now support
record_sample_proportion.
By default, record_sample_proportion=0.05. Splink samples records on both
sides of the blocking join, counts the observed pairs, and scales the result
back up. Because a p-fraction sample per side yields roughly a p²-fraction
of pairs, the scale-up factor is 1 / p².
Sampled records are marked with:
{"record_sample_proportion": 0.05, "is_estimate": True}
Pass record_sample_proportion=1.0 for exact counts.
estimate_probability_two_random_records_match() gains the same
record_sample_proportion argument. Setting it below 1.0 estimates the
deterministic-rule match count from a sample.
Why it matters

When designing blocking rules, exact comparison counts are often unnecessary.
A sampled estimate is usually enough to understand whether a rule is too broad
or too narrow. Sampling makes blocking-rule design and prior estimation much
faster on large datasets, while exact counts remain available when needed.
Example

from splink.blocking_analysis import count_comparisons_from_blocking_rules

# Fast default: estimate from a 5% per-side sample
records = count_comparisons_from_blocking_rules(
    df,
    blocking_rules=[block_on("first_name", "surname"), block_on("email")],
    link_type="dedupe_only",
)

# Exact count
records = count_comparisons_from_blocking_rules(
    df,
    blocking_rules=block_on("postcode"),
    link_type="dedupe_only",
    record_sample_proportion=1.0,
)

# From the linker
records = linker.blocking_analysis.count_comparisons_from_blocking_rules()
chart = linker.blocking_analysis.chart_comparisons_from_blocking_rules()
biggest = linker.blocking_analysis.n_largest_blocks(...)

# Faster prior estimation on large data
linker.training.estimate_probability_two_random_records_match(
    [block_on("email"), block_on("first_name", "dob")],
    recall=0.8,
    record_sample_proportion=0.1,
)
Related: PR #3088, #3163.

9. Bayes factors are replaced by match weights

What changed

The internal arithmetic of computing partial scores and combining evidence
across comparisons is now done in log-space match weights:
$$
mw = \log_2(m/u)
$$
Splink 4 used raw Bayes factors:
$$
m/u
$$
User-visible changes:


Splink 4
Splink 5


bayes_factor_column_prefix, default "bf_"
match_weight_column_prefix, default "mw_"


Predict output columns such as bf_first_name, bf_surname
Predict output columns such as mw_first_name, mw_surname


Final aggregate column match_weight
Unchanged


Dashboards reference bf_ columns
Dashboards now reference mw_ columns


The deprecated bayes_factor_column_prefix setting is accepted with a warning
and ignored.
Why it matters


Match weights are numerically more stable than raw Bayes factors.
Evidence can be combined with addition rather than multiplication.
Per-comparison output columns are now consistent with the existing aggregate
match_weight column.

The match probability is computed using a numerically safe form:
CASE
  WHEN mw >= 0 THEN 1.0 / (1.0 + POWER(2, -mw))
  ELSE          POWER(2, mw) / (1.0 + POWER(2, mw))
END
Migration

Downstream code that explicitly reads bf_* columns must be updated to read
mw_* columns instead. The values are now log-base-2 match weights. If a Bayes
factor is still needed, it can be recovered with 2 ** mw.
df = predict.as_pandas_dataframe()

# Splink 4
df["bf_first_name"]

# Splink 5 equivalent Bayes factor
2 ** df["mw_first_name"]
Related: PR #2851, #2952.

10. Term frequencies are now joined at predict time

What changed

In Splink 4, term-frequency tables were precomputed or implicitly cached into a
wide __splink__df_concat_with_tf table before pipelines that needed them.
In Splink 5, term-frequency tables for each TF-adjusted column are joined at the
start of the predict pipeline, into the blocked pairs, rather than being baked
into the input table.
Mechanically, enqueue_df_concat_with_tf() now produces a small CTE that
left-joins each per-column TF table to the input rows on demand.
Why it matters


Less wasted I/O when no TF columns are configured.
Cleaner cache semantics.
Predict pipeline SQL is easier to reason about and debug.
Explicit precomputation is still possible with
linker.table_management.compute_tf_table(...).

For DuckDB chunked prediction and manually registered blocked pairs, PR #3267
now applies these joins to the separately pruned left/right source tables.
TF lookup values still come from the linker's full input data or registered
lookups; they are not re-estimated from each chunk.
No code change is required for standard prediction or u-estimation workflows.
Related: PR #3010, #3267.

11. Reworked inference API: score_pair, score_pairs, predict_within, and predict_between

What changed

compare_two_records() has been removed and replaced by two methods:

score_pair() for scoring one known left/right pair
score_pairs() for scoring the cartesian product of two collections

There is no deprecated alias. Any code calling:
linker.inference.compare_two_records(...)
must be updated to:
linker.inference.score_pair(...)
for a single pair, or:
linker.inference.score_pairs(...)
for multiple records.
score_pair() accepts record dicts or SplinkDataFrames. Use it with one
record per side for a single-pair score. Supplying multi-row frames invokes
the same cartesian-product scoring as score_pairs(); it does not zip rows
into corresponding pairs.
score_pairs() scores the cartesian product of two record collections. No
blocking is applied. Every left record is compared to every right record. The
inputs may be list[dict] collections or SplinkDataFrames.
Both score_pair() and score_pairs() accept an
include_found_by_blocking_rules flag (default False). When True, the
output includes a found_by_blocking_rules column indicating whether each
scored pair would have been generated by the model's blocking rules.
find_matches_to_new_records() is removed. For an unblocked search, pass the
existing and new records to score_pairs() and filter on match_weight.
To retain blocking when searching a large existing dataset, use
predict_between() with the required blocking rules and threshold.
Two new experimental methods have also been added:

predict_within(dfs, ...)
predict_between(left, right, ...)

predict_within() generates blocked predictions within a supplied collection.
It mirrors the Linker constructor's input shape and respects the model's
link_type.
predict_between() generates blocked predictions between two collections. The
left record is always drawn from left, and the right record is always drawn
from right.
Both methods accept optional link_type and
blocking_rules_to_generate_predictions overrides, plus the usual
threshold_match_probability, threshold_match_weight, and warning_mode
arguments.
Unlike predict(), these methods do not derive term-frequency values from the
supplied data. TF tables must be registered, or hardcoded tf_* columns must be
supplied. Otherwise, a SplinkException is raised.
predict_between() supports exploding array blocking rules.
Why it matters


score_pair() makes the single-pair case explicit.
score_pairs() makes the cartesian-product case explicit.
Real-time and incremental linkage use the same scoring machinery as other
workflows.
predict_within() and predict_between() allow an existing trained model to
score new tables without constructing a new Linker around those tables.

Example

record_left = {
    "unique_id": 1,
    "first_name": "John",
    "surname": "Smith",
    "dob": "1971-05-24",
    "tf_first_name": 0.001,
}

record_right = {
    "unique_id": 1,
    "first_name": "Jon",
    "surname": "Smith",
    "dob": "1971-05-23",
    "tf_first_name": 0.0005,
}

df_score = linker.inference.score_pair(record_left, record_right)
Unblocked search for matches to new records:
new_record = db_api.register([{...}], dataset_display_name="incoming")
all_existing = ...

matches = linker.inference.score_pairs(all_existing, new_record)
good = matches.as_duckdbpyrelation().filter("match_weight > -4")
Experimental blocked prediction methods:
preds_within = linker.inference.predict_within(
    [df_a, df_b],
    threshold_match_probability=0.9,
)

preds_between = linker.inference.predict_between(
    left=df_existing,
    right=df_new,
    threshold_match_weight=0,
)
Related: PR #3104.

12. SQL pipeline profiling for DuckDB and Spark

What changed

Splink 5 includes two profiling database APIs:

DuckDBAPIWithProfiling
SparkAPIWithProfiling

They are drop-in replacements for DuckDBAPI and SparkAPI. They capture
detailed per-query profiling information to disk.
The DuckDB profiler captures native profiling information from the query
that materialises the result, with a single execution. Its defaults are:

enable_profiling="json": machine-readable .json profiles. Alternatives
are "query_tree" or "query_tree_optimizer" for text, and "no_output"
to collect profiling without writing a file.
profiling_mode="standard": execution metrics. "detailed" also includes
planner/optimiser timings; "all" requires a DuckDB version supporting it.
profiling_coverage="ALL": includes non-SELECT statements such as
CREATE TABLE AS and, in Parquet mode, COPY. "SELECT" is available for
narrower coverage. Older DuckDB versions without this setting use their
native coverage.

Profiling is disabled after execution, including on errors, to avoid
profiling unrelated later queries on the connection. Output filenames include
a UTC timestamp, counter, output name, and format-specific extension, with
collision checks to avoid overwriting existing profiles.
For Spark, each file contains:

the executed SQL
the final physical plan tree
the wall-clock duration in nanoseconds
each operator's runtime metrics

Why it matters

Profiling output makes it easier to identify which stage of a Splink job is
expensive. Files can be matched to pipeline outputs, and DuckDB's JSON output
can be inspected programmatically without running the materialisation again.
Example

from splink.backends.duckdb import DuckDBAPIWithProfiling
from splink import Linker, SettingsCreator, block_on
from splink.comparison_library import ExactMatch

db_api = DuckDBAPIWithProfiling(
    connection=":memory:",
    query_profiling_dir="my_profiles",
    enable_profiling="json",
    profiling_mode="detailed",
    profiling_coverage="ALL",
)

df = db_api.register(my_data, dataset_display_name="people")
linker = Linker(df, SettingsCreator(...))

linker.training.estimate_u_using_random_sampling(max_pairs=1e6)
linker.inference.predict(threshold_match_weight=0)
Spark equivalent:
from splink.internals.spark.database_api_with_profiling import SparkAPIWithProfiling

db_api = SparkAPIWithProfiling(
    spark_session=spark,
    query_profiling_dir="spark_profiles",
)
To profile direct Parquet writes, also pass materialisation="parquet" and
materialisation_dir=..., with optional parquet_materialisation_options
(section 5d). These constructor options belong to the DuckDB profiler;
Spark writes physical-plan and runtime-metrics output.
Related: PR #3021, #3265, #3274.

13. Removed features and simplifications

13a. Implicit caching removed

The use_cache argument has been removed from database execution methods,
along with the implicit caching mechanism behind it.
Caching is now driven explicitly via linker.table_management, including:

compute_tf_table()
register_term_frequency_lookup()
register_table_predict()
register_blocked_pairs_for_predict()
invalidate_cache()

The old system silently reused tables based on string matching of templated
names. Explicit registration makes cache usage visible and predictable.
These registration helpers now accept a SplinkDataFrame only, and the
overwrite argument has been dropped from them. Register raw data with
db_api.register() first, then pass the resulting SplinkDataFrame:
# Splink 4
linker.table_management.register_table_predict(pd.read_parquet("preds.parquet"))

# Splink 5
preds = db_api.register(pd.read_parquet("preds.parquet"))
linker.table_management.register_table_predict(preds)
This applies to register_table_predict(),
register_term_frequency_lookup(), register_blocked_pairs_for_predict(), and
register_labels_table(). Passing an unregistered object raises the same clear
"Expected a SplinkDataFrame" error described in section 1.
Related: PR #2847, #2848, #3128.

13b. Salting removed

DuckDB no longer needs the salting mechanism that was previously used to
parallelise blocking joins.
Public API change:
salting_partitions
has been removed from block_on() and blocking-rule constructors.
Settings dictionaries that still contain salting keys are accepted with a
warning and the keys are ignored.
The result is simpler blocking SQL, fewer generated unions, and less debug
noise.
Related: PR #2849.

13c. Athena backend removed

The Amazon Athena backend, AthenaAPI, is removed.
Users on Athena should migrate to DuckDB or Spark. DuckDB can read S3 Parquet
directly via httpfs.
# Splink 4
from splink.athena import AthenaAPI

db_api = AthenaAPI(
    database="my_db",
    s3_output_path="s3://...",
)

# Splink 5
import duckdb
from splink import DuckDBAPI

con = duckdb.connect()
con.execute("INSTALL httpfs; LOAD httpfs;")

db_api = DuckDBAPI(con)

df = db_api.register(
    con.read_parquet("s3://bucket/people/*.parquet"),
    dataset_display_name="people",
)
Related: PR #2858.

13d. materialise_blocked_pairs removed

Blocked pairs are always materialised in Splink 5. The
materialise_blocked_pairs=False shortcut from Splink 4 no longer exists.
This supports caching, sharing, chunking, and the persisted blocked-pairs
workflow.

14. predict() warnings are emitted once per call

What changed

Splink 4 could emit many repeated warnings during a single prediction job when a
comparison level had not been trained. In Splink 5, each
comparison-level/m-or-u warning is emitted at most once per predict() call.
There is also a new warning_mode argument:
linker.inference.predict(warning_mode="auto")
linker.inference.predict(warning_mode="always")
linker.inference.predict(warning_mode="never")
Internally, _m_warning_sent and _u_warning_sent flags on each
ComparisonLevel track whether a warning has been issued for the current call.
Related: PR #3003.

15. Reworked logging API

What changed

Logging has been overhauled so that Splink configures its own logger without
taking over logging for the whole Python process.


The Linker's set_up_basic_logging: bool = True argument is replaced by
log_level: int | str | None = logging.INFO. There is no back-compatible
alias - passing set_up_basic_logging= now raises a TypeError.


Splink no longer calls logging.basicConfig() or changes the root logger.
Instead it attaches its own handler to the splink logger, sets
propagate=False, and writes to stderr. Applications keep full control of
their own logging setup.


If your application has already configured handlers on the splink logger,
Splink detects this and leaves your configuration in place.


A new public splink.logging module exposes enable() and disable() so you
can configure Splink logging independently of constructing a Linker.


Three custom levels are registered alongside the standard ones, giving
progressively more detail:


Level
Value
Output


INFO
20
User-facing training/progress messages (default)


VERBOSE
15
Timing and parameter-estimation detail


DEBUG
10
Names of the SQL statements executed


PIPELINE
7
Names of the components of each SQL pipeline


SQL
5
The SQL statements themselves


log_level accepts either an integer or a level name, so
Linker(df, settings, log_level="SQL") is a quick way to view the (now
pretty-printed) SQL Splink generates.
Why it matters


Splink is better behaved inside larger applications: it no longer mutates the
root logger as a side effect of constructing a Linker.
You can turn Splink logging on or off at any time, not just at construction.
The extra levels make it easy to dial in exactly how much detail you want.

Example

import logging
import splink.logging

# Configure when creating the linker (most common)
linker = Linker(df, settings, log_level=logging.DEBUG)

# Do not let Splink configure logging at all
linker = Linker(df, settings, log_level=None)

# Configure independently of the linker
splink.logging.enable(logging.INFO)
linker = Linker(df, settings)

# See the generated SQL
linker = Linker(df, settings, log_level="SQL")

# Remove Splink's handler again
splink.logging.disable()
Related: PR #3110.

16. Blocking SQL is leaner

What changed

Several optimisations reduce the amount of SQL generated and the amount of data
moved through blocking joins:

Blocking now selects only the columns required by the blocking rule and
downstream comparison logic, rather than SELECT *.
The historical SELECT * FROM ... CTE used only to alias the input table as
__splink__df_concat has been removed.
The same column-pruning optimisation has been applied to exploding blocking
rules that use arrays_to_explode.
Blocking-count SQL now skips identity-only CTEs for standard rules and
produces shorter SQL.
DuckDB chunked and registered-pair predictions now materialise only the
source records required for scoring after blocking (PR #3267; section 5a).
Python-side SQL construction does less repeated parsing: comparison-level
column extraction is cached by SQL condition/dialect, and the reference
signatures used to recognise input columns are built once (PR #3162).

Why it matters

For wide tables, especially those with many columns, this can substantially
reduce data movement through blocking joins. Blocking-rule analysis is also
faster.
New guidance for expensive comparisons

PR #3141 expands the DuckDB performance guide with an opt-in technique for
avoiding repeated fuzzy-function evaluation across thresholds. A custom null
level can include a never-true sentinel test of the same function, allowing
DuckDB to recognise that it can compute the value once and reuse it.
Keep .configure(is_null_level=True) so null rows retain their training
semantics. This PR adds guidance, not a new configure() API or an automatic
rewrite of comparisons.
Only use the technique when the function is safe to evaluate on every input:
the earlier null/blank branch may otherwise be preventing an error. The guide
uses Jaro-Winkler as a worked example and explains why this is not safe for
every similarity function.
PR #3149 also expands the performance guidance on sampling, chunking, and
prediction thresholds. Raising a prediction threshold reduces the amount of
output retained/written; it does not replace blocking or avoid scoring all
candidate pairs.
Related: PR #2972, #2973, #2974, #3016, #3141, #3149, #3162, #3267.

17. Behavioural ports from Splink 4

Splink 5 includes the following behavioural changes that originally landed in
Splink 4.0.14 or 4.0.15:

Two-dataset link_only joins are faster when one side is much smaller than
the other.
Exploding blocking rules in two-dataset link_only mode are computed only on
the relevant side.
min_source_dataset behaviour for link_only has been restored to match
Splink 4.0.15 semantics, fixing edge cases where the wrong source-dataset
filtering was applied.

These are performance and correctness fixes. No API changes are required.
Related: PR #2974, #2975.

18. Charts are now first-class SplinkChart objects

What changed

Splink 4 returned charts as Altair Chart objects. Splink 5 introduces a new
SplinkChart abstraction:
class SplinkChart(ABC, Generic[T]):
    def __init__(self, records: Sequence[T]): ...

    @property
    def chart_dict(self) -> dict: ...
    @property
    def chart_data(self) -> Sequence[T]: ...
    def set_width_height(self, *, width=None, height=None): ...
    def save_offline_chart(self, path): ...
Chart data is now constructed from typed dataclasses, such as:

ModelParameterDetailedRecord
ModelParameterIterationDetailedRecord
ComparisonLevelDetailedRecord

Why it matters


Charts no longer require pandas.
Chart data can be inspected programmatically via chart.chart_data.
Width and height adjustment is consistent across charts.
Charts can still be displayed in notebooks and saved offline.

Example

chart = linker.visualisations.match_weights_chart()
chart.set_width_height(width=600, height=400)
chart.save_offline_chart("mw.html")

for r in chart.chart_data:
    print(
        r.comparison_name,
        r.label_for_charts,
        r.m_probability,
        r.u_probability,
    )
Column profiling now uses the same chart API

profile_columns() now returns a ProfileColumnsChart, a SplinkChart,
instead of an Altair chart/dict or None. Its .subcharts property exposes
the individual ProfileSingleColumnCharts in expression order for the
expressions that could be profiled. Each subchart exposes .col_name,
.chart_data (distribution records), .top_n_data, and .bottom_n_data.
You can display or save the combined chart or an individual subchart:
from splink.exploratory import profile_columns

profile = profile_columns(df, column_expressions=["first_name", "surname"])
profile.save_offline_chart("column_profiles.html")
profile.subcharts[0].save_offline_chart("first_name_profile.html")
print(profile.subcharts[0].top_n_data)
Use .chart_dict for the Vega-Lite specification or .altair_chart for
Altair-specific operations. .save(...) also delegates to Altair. If no
non-null column expressions can be profiled, the function now raises
SplinkException("No non-null column expressions found to profile.");
replace any code that previously checked for a None return.
Faster DuckDB comparison viewer

PR #3161 replaces the DuckDB comparison viewer's full-table random ranking
with a bounded selection of example row IDs per comparison-vector group,
then fetches the selected records. Group counts and score summaries retain
their purpose, but examples are now selected by table rowid, so they should
not be treated as a random sample. The optimisation applies in DuckDB table
materialisation mode. PR #3274 keeps the general path for Parquet-backed
views, which do not have table rowids; other backends also keep that path.
Related: PR #2940, #2941, #2956, #3161, #3175, #3274.

19. Spark-specific improvements

What changed

Splink 5 includes several Spark-specific improvements:

The __splink__filtered_neighbours table is now automatically persisted in
Spark alongside the other tables on Splink's persist list.
Collecting results from Spark is faster. Splink now materialises a Spark
output with a single collect() job rather than a separate
toLocalIterator() pass per column. This removes a Splink 4 -> 5 performance
regression that particularly affected EM training and estimate_u, where the
per-column collection caused many redundant Spark jobs.
Spark profiling is available via SparkAPIWithProfiling.
The Spark profiling output captures the executed physical plan after AQE,
along with operator metrics.
Spark 4 is supported, including an updated scala-udf-similarity jar for
Spark 4. At this commit, the published extras are splink[spark] and
splink[pyspark] (both require PySpark 3.5+); spark-3 and spark-4 are
repository development groups. Pin PySpark explicitly when choosing a
major version, for example pip install 'splink[spark]>=5,<6' 'pyspark>=4,<5'
or pip install 'splink[spark]>=5,<6' 'pyspark>=3.5,<4'.
CosineSimilarityLevel and CosineSimilarityAtThresholds now work on
Spark for numeric array/vector columns (PR #3176). Native higher-order
array operations compute the dot product and norms, without a Python UDF.
A zero norm produces SQL NULL through a guarded denominator rather than
a division-by-zero result.
Existing Spark optimisations from Splink 4, such as
repartition_after_blocking and partition controls, are unchanged.

Why it matters

Clustering jobs that re-read __splink__filtered_neighbours multiple times
during connected-components iteration should be faster on Spark. The faster
result collection closes a regression that made some Spark training steps slower
than in Splink 4. The profiling output also gives much better visibility into
slow Spark jobs. Vector/embedding comparisons can now use the same cosine
comparison definitions on DuckDB and Spark:
from splink.comparison_library import CosineSimilarityAtThresholds

vector_comparison = CosineSimilarityAtThresholds(
    "text_vector", [0.9, 0.8, 0.7]
)
PR #3147 switches Spark tests from Parquet to checkpoint-based lineage
breaking and disables constraint propagation in the test session to avoid a
Spark planner error. PR #3149 adds Spark chunking guidance. These are test
configuration and documentation changes; they do not introduce new
SparkAPI defaults.
Related: PR #2928, #3021, #3108, #3146, #3147, #3176.

20. Developer experience and packaging changes

These changes mainly affect contributors, maintainers, and users managing
Splink environments.
Packaging and development environments


Splink 5 requires Python 3.10+; Python 3.9 support is dropped.
Makefile and uv provide the development workflow. Dependency groups
include core, dev, testing-core, testing, linting, typechecking,
typechecking-dev, docs, pandas, postgres, sqlite, spark-3, and
spark-4. The docs group requires Python 3.13+.
Published extras and development groups are distinct. At this commit the
extras are spark, pyspark, sqlite, postgres, and pyarrow;
pandas, spark-3, and spark-4 are development groups, not pip extras.
Demo dependencies now live in a separate project under docs/demos/, with
its own pyproject.toml and uv.lock and an editable dependency on the local
Splink checkout. Run notebooks in that project, or use
make demos DEMO_PATH=... (PR #3165).
The demo environment explicitly includes pandas/NumPy; this does not make
them runtime requirements for Splink.
The library test suite uses pandas-free paths to guard against accidental
pandas dependencies. Demo dependencies are resolved separately.

Type checking, linting, and CI


Ruff and mypy are unpinned from their old versions and updated (PR #3153,
#3154). Minimum development versions are now Ruff 0.15.18 and mypy 2.1.0.
Type annotations throughout the library now explicitly include None where
accepted. Mypy's implicit_optional allowance is removed; overloads,
blocking-rule types, and callable protocols are clarified (PR #3163).
Normalising iterable inputs now handles tuples/generators while preserving
strings/dicts as single values. The linker blocking-analysis component
expects an iterable when rules are overridden (section 8).
mypy and ty run as separate CI jobs. ty no longer uses --exit-zero:
errors fail its job, although unresolved imports are explicitly ignored.
Deprecated logger.warn and abstractproperty usages are also cleaned up.
A lockfile check runs uv lock --check when the root manifest/lock changes
(PR #3152).
Spark CI uses shared sessions and parallel tests; further Spark configuration
changes improve test runtime and reliability (PR #3147).
Splink 5 removes codecov integration and the JSON-Schema settings
validator (splink/files/settings_jsonschema.json,
internals/validate_jsonschema.py, internals/default_from_jsonschema.py).
Settings validation is in Python code and emitted SQL is pretty-printed.

Documentation and contributor workflow


Notebooks are stored as .nb.py Jupytext sources. Tutorials and examples use
the registered-input API and now as_record_list(). Performance guidance
covers sampling, chunking, distributed prediction, and safe reuse of fuzzy
function results (PR #3141, #3149, #3150).
The documentation workflow now builds for master as well as
splink4_maintenance, rendering notebooks before MkDocs builds the site
(PR #3249). The demo renderer supports bounded parallelism, per-notebook
logs/timings, and failure summaries. PR #3256 retries a failed kernel startup
once and uses a 300-second timeout where the timeout command is available;
failures within the notebook remain failures.
Documentation also updates the Australian Institute of Health and Welfare
use case and Splink 3 documentation links (PR #3203), and corrects the
duplicated wording in the Fellegi–Sunter theory guide (PR #3280).
Contributor guidance now asks first-time code contributors to discuss scope
and wait for a maintainer before starting, with scope discussions also
expected for substantial changes. It allows AI assistance, asks external
contributors to declare significant use, and asks contributors to write
their own PR descriptions and interactions with the team (PR #3209).
API references, README diagrams, examples, link checks, and publish
permissions are also updated for Splink 5.

Dependency and workflow refreshes

Splink 5 updates both lockfiles and CI actions. The root lock
includes DuckDB 1.5.5, SQLGlot 30.18.0, Altair 6.2.2, and PyArrow 25.0.1;
other refreshes cover pytest, ty, SQLAlchemy, GitPython, Griffe, MkDocs tooling,
Mistune, msgpack, and Tornado. The separate demo lock includes updates to
PySpark, Jupytext, ipywidgets, and Tornado. These are resolved development/demo
versions, not new minimum runtime requirements for every Splink installation.
Workflow updates include Checkout, setup-uv, Deploy Pages, and CodeQL. The PR
index below groups the dependency/workflow changes, including the dedicated
CodeQL maintenance PR #3192.
Related: PR #2861, #2899, #2908, #2971, #3005, #3007, #3012, #3014, #3015,
#3018, #3027, #3053, #3079, #3092, #3093, #3108, #3112, #3114, #3115, #3133,
#3134, #3136, #3139, #3140, #3141, #3147, #3149, #3150, #3152, #3153, #3154,
#3163, #3165, #3192, #3203, #3209, #3249, #3256, #3280.

21. Before/after migration cheat-sheet

# === Splink 4 ===

import pandas as pd
from splink import Linker, DuckDBAPI

db_api = DuckDBAPI()
df = pd.read_parquet("people.parquet")

linker = Linker(df, settings, db_api=db_api)

linker.training.estimate_u_using_random_sampling(max_pairs=1e6)

preds = linker.inference.predict(materialise_blocked_pairs=True)
preds_df = preds.as_pandas_dataframe()

# Result columns include:
# bf_first_name, bf_surname, ...


# === Splink 5 ===

from splink import Linker, DuckDBAPI

db_api = DuckDBAPI()

df = db_api.register(
    db_api.duckdb_con.read_parquet("people.parquet"),
    dataset_display_name="people",
)

linker = Linker(df, settings)

linker.training.estimate_u_using_random_sampling(
    max_pairs=1e6,
    min_count_per_level=100,
)

preds = linker.inference.predict(
    threshold_match_weight=0,
    num_chunks_left=4,
    num_chunks_right=4,
)

preds_records = preds.as_record_list(limit=10)  # renamed from as_record_dict()
preds_arrow = preds.as_pyarrow_table()  # requires PyArrow

# Result columns include:
# mw_first_name, mw_surname, ...
Additional migrations and opt-in features:


Splink 4 code or workflow
Splink 5


sdf.as_record_dict(...)
sdf.as_record_list(...); same list[dict] result


Standalone blocking analysis
New linker count/chart methods default to the model settings; pass overrides as a list of rules


profile_columns(...) used as an Altair object/dict
Use .altair_chart / .chart_dict; access individual charts via .subcharts


profile_columns(...) is None check
Handle SplinkException when no non-null expressions can be profiled


Investigating slow SQL pipelines
Use the new DuckDB or Spark profiling API; DuckDB defaults to JSON profiles


Running example notebooks
Use the docs/demos project or make demos DEMO_PATH=...


Large DuckDB materialisations
Optionally configure materialisation="parquet" and a local materialisation_dir on the API


Spark vector comparisons
CosineSimilarityAtThresholds / CosineSimilarityLevel are now supported


For pip installation, use the extras actually declared at this commit
(spark/pyspark, sqlite, postgres, pyarrow) and install pandas
separately if needed. Repository pandas, spark-3, and spark-4 groups are
not published extras.

Appendix: PR index

The PRs below provide background for the changes from Splink 4 to Splink 5.
Dependency-only PRs are grouped separately; their resolved versions are not
additional runtime API changes. Feature details were checked against merged
code and tests.


Theme
PRs


Input contract
#2863, #2865, #2866, #3109


Pandas/numpy optional
#2883, #2937, #2956, #2969, #2985, #2987


New SDF outputs and record-list rename
#2916, #3150


query_sql defaults
#2970


Chunked predict and DuckDB source pruning
#2850, #2915, #2957, #3267


Blocked-pairs compute/register API and source pruning
#3127, #3128, #3267


Faster estimate_u
#2870, #3025


Deterministic hash-based sampling
#3122


max_pairs for EM training
#3050, #3090


Blocking analysis, sampled counts, and iterable overrides
#3088, #3163


Match weights
#2851, #2952


TF at predict and on pruned source inputs
#3010, #3267


Reworked inference API
#3104


Reworked logging API
#3110


SQL profiling and native DuckDB JSON profiles
#3021, #3265, #3274


Removals: cache, salting, Athena
#2847, #2848, #2849, #2858


Predict warnings
#3003


Blocking SQL and Python SQL-construction speedups
#2972, #2973, #2974, #3016, #3162, #3267


Splink 4 ports / merge-forward
#2974, #2975, #3108


Charts, column profiling, and comparison viewer
#2940, #2941, #2956, #3161, #3175, #3274


Spark execution, test configuration, and cosine similarity
#2928, #3021, #3146, #3147, #3176


Drop Python 3.9
#3053


Docs, examples, tutorials, and notebook rendering
#3112, #3114, #3133, #3134, #3136, #3139, #3149, #3150, #3165, #3203, #3249, #3256, #3280


Dev, packaging, type checking, and CI
#2861, #2899, #2908, #2971, #3005, #3007, #3012, #3014, #3015, #3018, #3027, #3079, #3092, #3093, #3115, #3140, #3152, #3153, #3154, #3163, #3165


DuckDB direct Parquet materialisation
#3274


DuckDB comparison-function performance guidance
#3141


Contributor scope and AI-use guidance
#3209


Dependency locks: DuckDB, SQLGlot, Altair, PyArrow
#3130, #3156, #3217, #3221, #3222, #3226, #3237, #3244, #3246, #3282


Dependency locks: testing and development tools
#3129, #3143, #3145, #3172, #3214, #3228, #3245, #3248, #3279


Dependency locks: documentation tooling
#3144, #3155, #3183, #3196, #3218, #3219, #3247, #3251, #3260, #3261, #3264, #3275


Dependency locks: separate demo project
#3262, #3271, #3272, #3273


CI actions and CodeQL maintenance
#3131, #3188, #3192, #3204, #3208, #3212, #3220, #3231, #3233, #3234, #3241, #3242, #3243, #3259, #3276


Appendix: LLM agent prompt for upgrading Splink 4 code to Splink 5

The block below is a ready-to-use prompt. Paste it into a coding agent that has
access to the user's Splink 4 codebase. It encodes the breaking changes in this
document so the agent can perform the upgrade.
You are upgrading a Python codebase from Splink 4 to Splink 5, using the API
at commit 2d57cd67e174a075c63db9904646c80c6613d727 as the reference.
Splink 5 contains breaking API changes. Find and rewrite all Splink usage so it runs
on Splink 5, preserving the original behaviour wherever possible. Work file by
file, explain each change, and do not alter non-Splink logic.

Apply ALL of the following transformations:

1. INPUT CONTRACT — register inputs before constructing the Linker.
   - Splink 4: `Linker(df, settings, db_api=db_api)` where `df` is a pandas
     DataFrame / table name / list of tables.
   - Splink 5: first register each input via
     `sdf = db_api.register(df, dataset_display_name="...")`, then call
     `Linker(sdf_or_list_of_sdfs, settings)`. The `db_api=` kwarg is REMOVED —
     the Linker derives the db_api from the registered SplinkDataFrame(s).
   - `db_api.register(...)` accepts a string (existing table), pandas DataFrame,
     pyarrow.Table, dict[str, list], list[dict], or a backend-native object
     (duckdb relation / spark DataFrame).
   - For `link_only` / `link_and_dedupe`, source-dataset names come from the
     `dataset_display_name` argument at registration, not positional ordering.
   - Use `table_name=` when you need to control the registered internal table
     name separately from the dataset label shown in outputs.
   - For DuckDB file inputs, register a relation created by the same backend
     connection, e.g. `db_api.register(db_api.duckdb_con.read_parquet(path))`.
     Do not treat a bare file path as an existing table name or re-register
     a SplinkDataFrame from a different connection as raw data.

2. PANDAS / NUMPY ARE OPTIONAL.
   - Do not assume pandas is installed. If code calls `.as_pandas_dataframe()`
     and only needs Python data, prefer `.as_record_list()` (list[dict]),
     `.as_dict()` (columnar), or `.as_pyarrow_table()`. Keep
     `.as_pandas_dataframe()` only where a pandas DataFrame is genuinely
     required (and ensure pandas is installed separately).
   - BREAKING RENAME (PR #3150): replace every `.as_record_dict(...)` with
     `.as_record_list(...)`. There is no alias. Return shape (list[dict]) and
     the `limit` argument are unchanged. Do not rename `.as_dict()`, which
     returns a columnar dict[str, list].
   - New outputs available on SplinkDataFrame: `as_pyarrow_table()`,
     `as_duckdbpyrelation()` (DuckDB only), `as_spark_dataframe()` (Spark only).
     Install `splink[pyarrow]` for Arrow conversions and dict/list registration;
     PyArrow is not a required runtime dependency.

3. `query_sql` now returns a SplinkDataFrame by default.
   - `linker.misc.query_sql(...)` used to return a pandas DataFrame. If the code
     relies on a pandas DataFrame, pass `output_type="pandas"`.
   - `SplinkDataFrame` now also has a `.query_sql(sql)` method that runs SQL
     against that table (referenced in the SQL as `{this}`) and returns a new
     `SplinkDataFrame`, so you can chain SQL off an existing result without
     going back through the linker.

4. MATCH WEIGHTS replace BAYES FACTORS.
   - The setting `bayes_factor_column_prefix` (default `"bf_"`) is replaced by
     `match_weight_column_prefix` (default `"mw_"`).
   - Any downstream code reading `bf_<col>` columns must read `mw_<col>` instead.
     Values are now log-base-2 match weights, so the Bayes factor equivalent is
     `2 ** df["mw_<col>"]`.

5. LOGGING — `set_up_basic_logging` is REMOVED, replaced by `log_level`.
   - Splink 4 used `Linker(..., set_up_basic_logging=True)` / the standalone
     `set_up_basic_logging(...)` helper. Both are GONE; passing
     `set_up_basic_logging=` to the Linker now raises `TypeError`.
   - Splink 5: pass `log_level=` to the Linker (default `logging.INFO`). It
     accepts a stdlib level int, a standard name (`"INFO"`, `"DEBUG"`), or one of
     Splink's custom names: `"VERBOSE"` (15), `"PIPELINE"` (7), `"SQL"` (5).
     Example: `Linker(df, settings, log_level="SQL")` to log emitted SQL.
   - Splink no longer calls `logging.basicConfig()` or mutates the root logger;
     it configures a dedicated handler on the `"splink"` logger (stderr,
     `propagate=False`). Remove any logging setup that only existed because
     Splink used to reconfigure the root logger.
   - For finer control use the new module: `import splink.logging`, then
     `splink.logging.enable(level=...)` / `splink.logging.disable()`.

6. REMOVED FEATURES — delete or migrate.
   - `salting_partitions` argument on `block_on()` / blocking rules: REMOVED.
     Delete it.
   - `use_cache` argument and implicit caching: REMOVED. Use explicit
     `linker.table_management` methods (`compute_tf_table`,
     `register_term_frequency_lookup`, `register_table_predict`,
     `register_blocked_pairs_for_predict`, `invalidate_cache`).
   - Table-registration helpers now accept a `SplinkDataFrame` ONLY and the
     `overwrite` argument is REMOVED: `register_table_predict`,
     `register_term_frequency_lookup`, `register_blocked_pairs_for_predict`,
     `register_labels_table`. Call `db_api.register(raw_data)` first to obtain a
     SplinkDataFrame, then pass that. Passing a pandas/pyarrow/dict object or
     `overwrite=` now fails; to replace a registration, invalidate/recompute it.
   - `materialise_blocked_pairs` argument on `predict()`: REMOVED (blocked pairs
     are always materialised). Delete it.
   - Athena backend (`AthenaAPI`): REMOVED. Migrate to DuckDB (which can read S3
     parquet via httpfs) or Spark.
   - Legacy JSON-schema settings validator files: gone (validation is in code).
   - Blocking-analysis functions renamed/removed:
     `count_comparisons_from_blocking_rule` (singular) ->
     `count_comparisons_from_blocking_rules` (plural);
     `cumulative_comparisons_to_be_scored_from_blocking_rules_data` ->
     `count_comparisons_from_blocking_rules`;
     `cumulative_comparisons_to_be_scored_from_blocking_rules_chart` ->
     `chart_comparisons_from_blocking_rules`. The plural function takes either a
     single rule or an iterable and always returns a list of records.

7. INFERENCE API RENAMES (PR #3104).
   - `linker.inference.compare_two_records(...)` is REMOVED. Replace a
     single-pair call with `linker.inference.score_pair(...)`.
   - `linker.inference.score_pair(...)` scores one explicit left/right pair and
     accepts either record dicts or `SplinkDataFrame`s. Use one record on each
     side; multi-row frames form a cartesian product, not a zip of rows.
   - `linker.inference.score_pairs(...)` scores the cartesian product of two
     inputs with NO blocking. It accepts `list[dict]` collections or
     `SplinkDataFrame`s.
   - `score_pair(...)` / `score_pairs(...)` take
     `include_found_by_blocking_rules=False`. This does NOT filter rows: when
     `True`, an extra boolean `found_by_blocking_rules` column is added to the
     output indicating whether each scored pair would have been generated by the
     model's blocking rules.
   - `linker.inference.find_matches_to_new_records(...)` is REMOVED. For an
     unblocked search, register the existing/new records, call
     `score_pairs(existing, new_records)`, and filter on `match_weight`.
     Preserve blocking and thresholds in a previously blocked search by using
     `predict_between(existing, new_records, ...)` with equivalent rules.
   - New experimental methods exist if blocked ad-hoc prediction is wanted:
     `linker.inference.predict_within(dfs, ...)` and
     `linker.inference.predict_between(left, right, ...)`. These require TF
     tables to be registered (or hardcoded `tf_*` columns); they do not derive
     TF values from the data.
   - `predict_between(...)` supports exploding array blocking rules.
   - Precompute/load any required TF lookups through
     `linker.table_management.compute_tf_table(...)` or
     `register_term_frequency_lookup(...)`, not `linker.training`.

8. NEW OPTIONAL PERFORMANCE LEVERS (apply only if the original code was
   struggling with large data; otherwise leave defaults).
   - `predict(num_chunks_left=L, num_chunks_right=R)` runs prediction in chunks
     (chunk results are UNION ALL'd, giving the same output as an unchunked
     `predict()`).
     DuckDB automatically materialises source subsets needed by blocked pairs
     for effective chunks and registered-pair scoring. No extra argument is
     needed; full-input/registered TF values are preserved.
   - Distributed / persisted blocked-pairs workflow:
     `compute_blocked_pairs_for_predict()` materialises all blocked pairs and
     `compute_blocked_pairs_for_predict_chunk(left_chunk, right_chunk)`
     materialises one chunk. Persist the result, then later
     `register_blocked_pairs_for_predict(sdf)` with a single SplinkDataFrame,
     followed by `predict()` to score it.
     Use `predict_chunk(left_chunk, right_chunk, ...)` only to have Splink both
     block and score a single chunk on one worker (it errors if blocked pairs
     are already registered).
     All workers must use the same backend/version, IDs, dataset labels,
     input snapshot, model, and chunk grid.
   - Random sampling (`estimate_u`, clustering) uses hash filters in place of
     backend-native TABLESAMPLE. For fixed data/IDs, seed, and backend version,
     it is repeatable. Hash functions differ across backends: do not assume
     identical DuckDB/Spark/Postgres/SQLite samples or chunk membership.
   - `estimate_u_using_random_sampling(..., min_count_per_level=100,
     num_chunks=10)` — early-stopping is on by default; pass
     `min_count_per_level=None` to process the full sampled workload, not to
     recreate the exact Splink 4 random sample or estimates.
   - `estimate_parameters_using_expectation_maximisation(..., max_pairs=...)` to
     cap EM training size.
   - `count_comparisons_from_blocking_rules(...)` (note: PLURAL name) now
     estimates from a 5% sample by default; pass `record_sample_proportion=1.0`
     for exact counts. `estimate_probability_two_random_records_match(...,
     record_sample_proportion=...)` can likewise sample for speed. A
     `linker.blocking_analysis` component also exists with
     `count_comparisons_from_blocking_rules` / `chart_comparisons_from_blocking_rules`
     / `n_largest_blocks` defaulting to the linker's own inputs and rules.
     Pass an iterable when overriding the linker component's count/chart rules,
     e.g. `blocking_rules=[block_on("email")]`; their standalone counterparts
     still accept a single rule directly. n_largest_blocks() takes one rule.
   - `predict(warning_mode="auto"|"always"|"never")` controls untrained-model
     warnings.
   - Optional DuckDB Parquet materialisation (PR #3274): configure the API with
     `materialisation="parquet"`, a required local `materialisation_dir`, and
     optionally `ParquetWriteOptions` imported from `splink.backends.duckdb`.
     Options are compression, compression_level, per_thread_output (True by
     default), file_size_bytes, and row_group_size. Results remain
     SplinkDataFrames over views; normal dropping/cleanup removes their backing
     files. Export durable results before cleanup. This is backend-wide, not
     a predict() argument or a remote-storage/distributed scheduler feature.
   - The DuckDB guide documents optional reuse of expensive comparison
     functions via a custom null level. Do not apply this automatically: the
     function must be safe on every input and the level must retain
     `.configure(is_null_level=True)`.

9. PROFILING (optional). For debugging slow jobs, `DuckDBAPIWithProfiling` /
   `SparkAPIWithProfiling` are drop-in replacements that write per-query
   profiles to a directory.
   - Import the DuckDB API from `splink.backends.duckdb`. It profiles the
     actual materialisation with a single execution. Defaults are
     enable_profiling="json", profiling_mode="standard", and
     profiling_coverage="ALL". Profiles have timestamped filenames; select
     "query_tree" / "query_tree_optimizer" if text output is preferred.
     "no_output" collects without a file; "detailed" adds planner timings;
     "all" requires a supporting DuckDB version. Older DuckDB versions without
     profiling_coverage use their native coverage.
   - Parquet options are also accepted by DuckDBAPIWithProfiling, and profiles
     describe the COPY. Spark's profiler writes physical plans and runtime metrics.

10. PACKAGING / ENV.
   - Splink 5 requires Python 3.10+ (3.9 dropped).
   - Use extras declared at the target commit: `splink[pyarrow]`,
     `splink[spark]` / `splink[pyspark]`, `splink[sqlite]`, `splink[postgres]`.
     Install pandas separately when needed. The repository's pandas, spark-3,
     and spark-4 groups are not published extras. Select Splink 5 with a
     version constraint such as >=5,<6. Pin PySpark explicitly if the
     application needs a particular major version.
   - In Splink contributor environments, demos now have their own project and
     lock under docs/demos. Use that project or `make demos DEMO_PATH=...`.
     The docs group needs Python 3.13+.
   - mypy no longer permits implicit optional types. ty errors now fail CI
     (unresolved imports are ignored); --exit-zero has been removed.

11. CHARTS AND DASHBOARDS.
   - profile_columns() returns ProfileColumnsChart (a SplinkChart). Access its
     .chart_dict for Vega-Lite, .altair_chart for Altair operations, and
     .subcharts for individual column charts. Both combined and individual
     charts support .save(...) and .save_offline_chart(...).
   - Replace checks for a None result with handling for SplinkException when
     no non-null column expressions can be profiled.
   - DuckDB table-mode comparison viewer examples are selected by rowid for
     speed. Do not rely on their being a random sample or retaining the old
     order. Parquet mode and other backends keep the general SQL path.

12. SPARK COMPARISONS.
   - CosineSimilarityLevel and CosineSimilarityAtThresholds now support
     numeric arrays on Spark through native SQL operations. Remove old
     backend exclusions where relevant; keep application-specific input
     validation and thresholds. Zero norms produce SQL NULL.

PROCESS:
- Search the codebase for: `Linker(`, `db_api=`, `set_up_basic_logging`,
  `compare_two_records`, `find_matches_to_new_records`,
  `materialise_blocked_pairs`, `use_cache`, `salting_partitions`,
  `bayes_factor_column_prefix`, `bf_`, `AthenaAPI`, `query_sql`,
  `as_pandas_dataframe`, `as_record_dict`, `register_table_predict`,
  `register_term_frequency_lookup`, `register_blocked_pairs_for_predict`,
  `register_labels_table`, `count_comparisons_from_blocking_rule`,
  `cumulative_comparisons_to_be_scored_from_blocking_rules`, `profile_columns`,
  `DuckDBAPIWithProfiling`, `query_profiling_dir`, `CosineSimilarity`,
  `compute_tf_table`, `splink[pandas]`, `splink[spark-3]`, `splink[spark-4]`,
  `implicit_optional`, `--exit-zero`.
- For each hit, apply the relevant transformation above.
- After editing, ensure imports are correct and the code type-checks.
  Validate representative input/output formats, chart consumers, and scoring
  results on the installed Splink 5 version.
- Produce a short summary of every change and flag anything that needs a human
  decision (e.g. ambiguous `as_pandas_dataframe()` usage, Athena migrations).
Method	Returns	Requirements
`as_record_list(limit=None)`	`list[dict]`	no pandas or Arrow needed for this conversion
`as_dict(limit=None)`	`dict[str, list]`	no pandas or Arrow needed for this conversion
`as_pyarrow_table(limit=None)`	`pyarrow.Table`	PyArrow installed
`as_pandas_dataframe(limit=None)`	`pandas.DataFrame`	pandas installed
`as_duckdbpyrelation(limit=None)`	`duckdb.DuckDBPyRelation`	native relation on DuckDB
`as_spark_dataframe()`	`pyspark.sql.DataFrame`	Spark backend
Splink 4 function removed in Splink 5	Replacement in Splink 5
`count_comparisons_from_blocking_rule`	`count_comparisons_from_blocking_rules`
`cumulative_comparisons_to_be_scored_from_blocking_rules_data`	`count_comparisons_from_blocking_rules`
`cumulative_comparisons_to_be_scored_from_blocking_rules_chart`	`chart_comparisons_from_blocking_rules`
Splink 4	Splink 5
`bayes_factor_column_prefix`, default `"bf_"`	`match_weight_column_prefix`, default `"mw_"`
Predict output columns such as `bf_first_name`, `bf_surname`	Predict output columns such as `mw_first_name`, `mw_surname`
Final aggregate column `match_weight`	Unchanged
Dashboards reference `bf_` columns	Dashboards now reference `mw_` columns
Level	Value	Output
`INFO`	20	User-facing training/progress messages (default)
`VERBOSE`	15	Timing and parameter-estimation detail
`DEBUG`	10	Names of the SQL statements executed
`PIPELINE`	7	Names of the components of each SQL pipeline
`SQL`	5	The SQL statements themselves
Splink 4 code or workflow	Splink 5
`sdf.as_record_dict(...)`	`sdf.as_record_list(...)`; same `list[dict]` result
Standalone blocking analysis	New linker count/chart methods default to the model settings; pass overrides as a list of rules
`profile_columns(...)` used as an Altair object/dict	Use `.altair_chart` / `.chart_dict`; access individual charts via `.subcharts`
`profile_columns(...) is None` check	Handle `SplinkException` when no non-null expressions can be profiled
Investigating slow SQL pipelines	Use the new DuckDB or Spark profiling API; DuckDB defaults to JSON profiles
Running example notebooks	Use the `docs/demos` project or `make demos DEMO_PATH=...`
Large DuckDB materialisations	Optionally configure `materialisation="parquet"` and a local `materialisation_dir` on the API
Spark vector comparisons	`CosineSimilarityAtThresholds` / `CosineSimilarityLevel` are now supported
Theme	PRs
Input contract	#2863, #2865, #2866, #3109
Pandas/numpy optional	#2883, #2937, #2956, #2969, #2985, #2987
New SDF outputs and record-list rename	#2916, #3150
`query_sql` defaults	#2970
Chunked predict and DuckDB source pruning	#2850, #2915, #2957, #3267
Blocked-pairs compute/register API and source pruning	#3127, #3128, #3267
Faster `estimate_u`	#2870, #3025
Deterministic hash-based sampling	#3122
`max_pairs` for EM training	#3050, #3090
Blocking analysis, sampled counts, and iterable overrides	#3088, #3163
Match weights	#2851, #2952
TF at predict and on pruned source inputs	#3010, #3267
Reworked inference API	#3104
Reworked logging API	#3110
SQL profiling and native DuckDB JSON profiles	#3021, #3265, #3274
Removals: cache, salting, Athena	#2847, #2848, #2849, #2858
Predict warnings	#3003
Blocking SQL and Python SQL-construction speedups	#2972, #2973, #2974, #3016, #3162, #3267
Splink 4 ports / merge-forward	#2974, #2975, #3108
Charts, column profiling, and comparison viewer	#2940, #2941, #2956, #3161, #3175, #3274
Spark execution, test configuration, and cosine similarity	#2928, #3021, #3146, #3147, #3176
Drop Python 3.9	#3053
Docs, examples, tutorials, and notebook rendering	#3112, #3114, #3133, #3134, #3136, #3139, #3149, #3150, #3165, #3203, #3249, #3256, #3280
Dev, packaging, type checking, and CI	#2861, #2899, #2908, #2971, #3005, #3007, #3012, #3014, #3015, #3018, #3027, #3079, #3092, #3093, #3115, #3140, #3152, #3153, #3154, #3163, #3165
DuckDB direct Parquet materialisation	#3274
DuckDB comparison-function performance guidance	#3141
Contributor scope and AI-use guidance	#3209
Dependency locks: DuckDB, SQLGlot, Altair, PyArrow	#3130, #3156, #3217, #3221, #3222, #3226, #3237, #3244, #3246, #3282
Dependency locks: testing and development tools	#3129, #3143, #3145, #3172, #3214, #3228, #3245, #3248, #3279
Dependency locks: documentation tooling	#3144, #3155, #3183, #3196, #3218, #3219, #3247, #3251, #3260, #3261, #3264, #3275
Dependency locks: separate demo project	#3262, #3271, #3272, #3273
CI actions and CodeQL maintenance	#3131, #3188, #3192, #3204, #3208, #3212, #3220, #3231, #3233, #3234, #3241, #3242, #3243, #3259, #3276