uproot-5.0.0-release-notes.md

Uproot version 5.0.0

Uproot version 5 has a few major new features, one removal (uproot.lazy), and is based on Awkward Array version 2 instead of version 1.

uproot.lazy → uproot.dask

@kkothari2001 upgraded Uproot from Awkward version 1 to version 2, the major part of which was replacing uproot.lazy, which is based on Awkward 1's virtual and partitioned lazy arrays, with the new Dask collection, dask-awkward. The entry point for this function is uproot.dask.

@kkothari2001 also simplified Uproot's Pandas backend, which used to "explode" ragged arrays from ROOT into Pandas DataFrames with a non-trivial MultiIndex. Now, it takes advantage of awkward-pandas to put ragged (and more complex) Awkward Arrays directly into Pandas columns.

If you want the old behavior, you can read data using library="ak" to get an Awkward Array, and use ak.to_dataframe to "explode" the data into a MultiIndex.

TTree-reading with AwkwardForth

@aryan26roy added a new code path to the TTree-reading routines to read them with AwkwardForth instead of pure Python. Users won't see any interface changes due to this code, but the performance of reading TBranches with AsObject or AsStrings Interpretations should be orders of magnitude faster. For example, std::vector<std::vector<float>> reading is now 400× faster.

Reading RNTuples

@Moelf added a complete reader of RNTuple data with most of an RNTuple-writer in an unmerged pull request (#705). Although the RNTuple format is still in development, this is a very good start at reading RNTuple data, whose structure is a close match to Awkward Arrays (so the translation is more one-to-one than it is for TTrees, for instance).

New features

• feat: move to hatchling by @henryiii in scikit-hep/uproot5#688
• feat: from_map like optimization for dask arrays by @kkothari2001 in scikit-hep/uproot5#679
• feat: Finalizing AwkwardForth reader for Uproot by @aryan26roy in scikit-hep/uproot5#644
• feat: implemented NON-memberwise deserialization for AsMap. by @jpivarski in scikit-hep/uproot5#746
• feat: Added column_projection optimization by @kkothari2001 in scikit-hep/uproot5#755
• feat: support categorical axes on boost histograms by @lobis in scikit-hep/uproot5#764
• feat: warn about TBranch name, alias name conflict. by @jpivarski in scikit-hep/uproot5#776
• feat: any Mapping assigned to a WritableDirectory is interpreted as a TTree or failure, no fall-through. by @jpivarski in scikit-hep/uproot5#779
• feat: add 'interp_options' mechanism and ak_add_doc. by @jpivarski in scikit-hep/uproot5#784
• feat: Use awkward pandas, instead of the existing code that explodes Pandas Dataframes by @kkothari2001 in scikit-hep/uproot5#734
• feat: made 'very optional' arguments keyword-only by @jpivarski in scikit-hep/uproot5#787
• feat: adjust for name change in scikit-hep/awkward#1919. by @jpivarski in scikit-hep/uproot5#788

Bug-fixes and performance

• fix: depend on packaging, not setuptools vendored packaging by @henryiii in scikit-hep/uproot5#684
• fix: Avoid triggering temporary dask-awkward/awkward incompatibility. by @jpivarski in scikit-hep/uproot5#694
• fix: Do not write incorrect fSumw2 in histograms (v5). by @jpivarski in scikit-hep/uproot5#698
• fix: Fixes uproot.dask bug with empty branches by @kkothari2001 in scikit-hep/uproot5#700
• fix: Use from_map optimization for delayed numpy arrays and add tests with empty branches for the same by @kkothari2001 in scikit-hep/uproot5#703
• fix: use ctx manager to ensure resources are freed by @agoose77 in scikit-hep/uproot5#713
• fix: ReadOnlyDirectory should provide the largest abs(cycle) when cycle is unspecified, not the largest cycle. by @jpivarski in scikit-hep/uproot5#715
• fix: regularize ROOT type aliases to C fundamental type names. by @jpivarski in scikit-hep/uproot5#717
• fix: avoid empty TBasket issue in embedded TBasket by @jpivarski in scikit-hep/uproot5#751
• fix: don't use Awkward in test_0751 that doesn't need it by @jpivarski in scikit-hep/uproot5#753
• fix: working TList serialization by @lobis in scikit-hep/uproot5#763
• fix: histogram weights not handled correctly in hist / boost conversion by @lobis in scikit-hep/uproot5#774
• perf: streamline metadata handling for TBranch name lookup and uproot.dask by @jpivarski in scikit-hep/uproot5#772
• fix: ensure AwkwardForth fallback path is tested without history. by @jpivarski in scikit-hep/uproot5#780
• fix: all AwkwardForth Forms now agree with awkward_form method output. by @jpivarski in scikit-hep/uproot5#790

Other

• Manually add a Model for TMatrixTSym. by @jpivarski in scikit-hep/uproot5#484
• Updating docs and test in response to the removal of uproot.lazy in Uproot5 by @kkothari2001 in scikit-hep/uproot5#615
• changed arguments for awkward_form by @aryan26roy in scikit-hep/uproot5#617
• Completed the Forth based AsStrings reader. by @aryan26roy in scikit-hep/uproot5#616
• Actually pass user-specified 'awkward_form' arguments into context. by @jpivarski in scikit-hep/uproot5#622
• Fix annoying gaps in test files. by @jpivarski in scikit-hep/uproot5#625
• Cleaning up string generation in streamers.py by @aryan26roy in scikit-hep/uproot5#629
• chore: add dependabot for actions by @henryiii in scikit-hep/uproot5#631
• Bump actions/checkout from 2 to 3 by @dependabot in scikit-hep/uproot5#632
• Bump actions/upload-artifact from 2 to 3 by @dependabot in scikit-hep/uproot5#633
• Bump pypa/gh-action-pypi-publish from 1.4.2 to 1.5.0 by @dependabot in scikit-hep/uproot5#634
• Bump actions/download-artifact from 2 to 3 by @dependabot in scikit-hep/uproot5#635
• Forth based ROOT reader (revised) by @aryan26roy in scikit-hep/uproot5#636
• Set up tests for AsObjects, for the AwkwardForth reader by @jpivarski in scikit-hep/uproot5#637
• Iterate over objects in TDirectory in linear time. by @jpivarski in scikit-hep/uproot5#638
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in scikit-hep/uproot5#641
• Awkward v2 update by @kkothari2001 in scikit-hep/uproot5#620
• docs: add aryan26roy as a contributor for code by @allcontributors in scikit-hep/uproot5#645
• docs: add kkothari2001 as a contributor for code by @allcontributors in scikit-hep/uproot5#646
• docs: add Moelf as a contributor for code by @allcontributors in scikit-hep/uproot5#647
• chore: cleanup flake8 by @henryiii in scikit-hep/uproot5#527
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in scikit-hep/uproot5#650
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in scikit-hep/uproot5#654
• Bump pypa/gh-action-pypi-publish from 1.5.0 to 1.5.1 by @dependabot in scikit-hep/uproot5#656
• Primitive Support for RNTuple by @Moelf in scikit-hep/uproot5#630
• Set ReadOnlyDirectory attributes when fSeekKeys == 0 by @kakwok in scikit-hep/uproot5#660
• docs: add kakwok as a contributor for code by @allcontributors in scikit-hep/uproot5#663
• Dask awkward support for uproot.dask by @kkothari2001 in scikit-hep/uproot5#652
• pathlib.Path drops '//' (naturally), but it's sometimes used for URLs by @jpivarski in scikit-hep/uproot5#670
• Gets the number of overflow bins for hist.axis.IntCategory, at least. by @jpivarski in scikit-hep/uproot5#671
• Prevent std::pair from being AsStridedObjects. by @jpivarski in scikit-hep/uproot5#673
• Implement transformed axis from boost-histogram/hist. by @jpivarski in scikit-hep/uproot5#675
• AsDynamic has no self._header. by @jpivarski in scikit-hep/uproot5#674
• Fixed TTree write_anew in a subdirectory (consistent caches). by @jpivarski in scikit-hep/uproot5#677
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in scikit-hep/uproot5#657
• Implement stl containers for RNTuple by @Moelf in scikit-hep/uproot5#662
• Multiple clusters support for RNTuple by @Moelf in scikit-hep/uproot5#682
• ci: Get test dependencies from one source by @jpivarski in scikit-hep/uproot5#686
• ci: autocancel repeated runs by @henryiii in scikit-hep/uproot5#685
• ci: use mamba by @henryiii in scikit-hep/uproot5#683
• ci: Lint PR titles according to conventional commits by @jpivarski in scikit-hep/uproot5#689
• docs: Installation requirements and error text for dask/dask-awkward in extras.py by @jpivarski in scikit-hep/uproot5#690
• ci: use concurrency group for semantic-pr-title by @agoose77 in scikit-hep/uproot5#691
• docs: Add image for dask docs by @kkothari2001 in scikit-hep/uproot5#708
• ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in scikit-hep/uproot5#706
• chore: dask_awkward.test_utils moved in ContinuumIO/dask-awkward#76 by @jpivarski in scikit-hep/uproot5#714
• test: adjust for boost-histogram 1.3.2's _storage_type deprecation. by @jpivarski in scikit-hep/uproot5#719
• ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in scikit-hep/uproot5#716
• add veprbl as a contributor by @allcontributors in scikit-hep/uproot5#728
• add nikoladze as a contributor by @allcontributors in scikit-hep/uproot5#729
• add klieret as a contributor by @allcontributors in scikit-hep/uproot5#730
• add dcervenkov as a contributor by @allcontributors in scikit-hep/uproot5#731
• add beojan as a contributor by @allcontributors in scikit-hep/uproot5#732
• add agoose77 as a contributor by @allcontributors in scikit-hep/uproot5#733
• docs: add a CITATION.cff for Uproot. by @jpivarski in scikit-hep/uproot5#726
• chore: drop Python 3.6. by @jpivarski in scikit-hep/uproot5#742
• ci: remove GHA workaround for macOS Python 3.11 by @henryiii in scikit-hep/uproot5#743
• chore: some cleanup inpsired by refurb by @henryiii in scikit-hep/uproot5#745
• chore(deps): update pre-commit hooks by @pre-commit-ci in scikit-hep/uproot5#737
• chore(deps): bump amannn/action-semantic-pull-request from 4 to 5 by @dependabot in scikit-hep/uproot5#757
• chore: update Uproot to require Awkward 2.0.0rc1. by @jpivarski in scikit-hep/uproot5#765
• ci: update to Python 3.11 final by @henryiii in scikit-hep/uproot5#766
• docs: add lobis as a contributor for code by @allcontributors in scikit-hep/uproot5#771
• refactor: refactor Forth generation by @aryan26roy in scikit-hep/uproot5#710
• chore: remove Identifier and "uproot" parameter. by @jpivarski in scikit-hep/uproot5#770
• ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in scikit-hep/uproot5#773
• refactor: final refactoring for Forth generation by @aryan26roy in scikit-hep/uproot5#749
• docs: fix TRef.py doc urls by @veprbl in scikit-hep/uproot5#782
• ci: pre-commit autoupdate by @pre-commit-ci in scikit-hep/uproot5#783
• docs: uproot.dask docs by @kkothari2001 in scikit-hep/uproot5#702
• test: make tests parallelizable (custom_classes in uproot.open). by @jpivarski in scikit-hep/uproot5#786
• ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in scikit-hep/uproot5#789
• chore(deps): bump pypa/gh-action-pypi-publish from 1.5.1 to 1.6.1 by @dependabot in scikit-hep/uproot5#792
• ci: [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in scikit-hep/uproot5#793

New Contributors

• @aryan26roy made their first contribution in scikit-hep/uproot5#617
• @dependabot made their first contribution in scikit-hep/uproot5#632
• @Moelf made their first contribution in scikit-hep/uproot5#630
• @agoose77 made their first contribution in scikit-hep/uproot5#691
• @lobis made their first contribution in scikit-hep/uproot5#763

Full Changelog: https://github.com/scikit-hep/uproot5/compare/4.2.4...v5.0.0