Kevin Liu kevinjqliu
kevinjqliu
/ user_token.py
Created
August 14, 2025 23:28
Make Entra/AAD token work as `DefaultAzureCredential` in Fabric notebook
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from azure.core.credentials import AccessToken, TokenCredential | |
| from datetime import datetime, timedelta | |
| class UserTokenCredential(TokenCredential): | |
| def __init__(self, token: str, expires_on: datetime = None): | |
| self._token = token | |
| self._expires_on = expires_on or (datetime.utcnow() + timedelta(hours=1)) | |
| def get_token(self, *scopes, **kwargs): | |
| return AccessToken(self._token, int(self._expires_on.timestamp())) |
kevinjqliu
/ .zshrc
Created
July 20, 2025 19:08
clean up local branch when remote ref is gone (for all remotes)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # git | |
| clean_branch() { | |
| echo "🔄 Fetching and pruning all deleted remote branches..." | |
| # fetch all remote branches and prune deleted ones | |
| # Use --dry-run to avoid actual deletion | |
| gone_branches=("${(@f)$(git fetch -p --all --dry-run 2>&1 | awk ' | |
| /\[deleted\]/ { | |
| split($5, parts, "/") # output is in the form of "remote/branch_name" | |
| remote = parts[1] | |
| branch_name = substr($5, length(remote)+2) |
kevinjqliu
/ pyarrow_complex_type.ipynb
Created
April 24, 2025 18:53
Pyarrow doesnt support complex type comparison and join
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
kevinjqliu
/ duckdb_iceberg_rest_catalog.sh
Created
April 4, 2025 23:59
Connect DuckDB to local Iceberg Rest Catalog
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| -- https://duckdb.org/2025/03/14/preview-amazon-s3-tables.html | |
| FORCE INSTALL aws FROM core_nightly; | |
| FORCE INSTALL httpfs FROM core_nightly; | |
| FORCE INSTALL iceberg FROM core_nightly; | |
| CREATE SECRET minio_secret ( | |
| TYPE S3, | |
| KEY_ID 'admin', | |
| SECRET 'password', | |
| REGION 'us-east-1', |
kevinjqliu
/ Iceberg + Delta - Collaborating Across Table Formats.ipynb
Last active
March 7, 2025 03:38
Iceberg + Delta - Collaborating Across Table Formats
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
kevinjqliu
/ datafusion-python-45.2.0-RC2.log
Created
February 18, 2025 00:49
`datafusion-python` 45.2.0 RC2 verification
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| python/tests/test_dataframe.py ................................................................................ssss.............FFFatal Python error: Aborted | |
| Current thread 0x00000001fb1c4840 (most recent call first): | |
| File "/Users/kevinliu/repos/datafusion-python/python/tests/test_dataframe.py", line 1088 in test_write_compressed_parquet | |
| File "/Users/kevinliu/repos/datafusion-python/.venv/lib/python3.12/site-packages/_pytest/python.py", line 159 in pytest_pyfunc_call | |
| File "/Users/kevinliu/repos/datafusion-python/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall | |
| File "/Users/kevinliu/repos/datafusion-python/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec | |
| File "/Users/kevinliu/repos/datafusion-python/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__ | |
| File "/Users/kevinliu/repos/datafusion-python/.venv/lib/python3.12/site-packages/_pytest/python.py", line 1627 in runtest | |
| File "/Users/kevinliu/repos/datafusion-python |
kevinjqliu
/ All About Iceberg and Its Catalogs.ipynb
Created
October 1, 2024 04:22
All About Iceberg and Its Catalogs - DEML Summit 2024
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| pyiceberg-py3.11➜ iceberg-python git:(main) ✗ PYTEST_ARGS="-q" make test-integration | |
| docker compose -f dev/docker-compose-integration.yml kill | |
| WARN[0000] /Users/kevinliu/repos/iceberg-python/dev/docker-compose-integration.yml: `version` is obsolete | |
| [+] Killing 5/5 | |
| ✔ Container pyiceberg-spark Killed 0.3s | |
| ✔ Container pyiceberg-mc Killed 0.2s | |
| ✔ Container pyiceberg-minio Killed 0.3s | |
| ✔ Container pyiceberg-rest Killed 0.2s | |
| ✔ Container hive Killed 0.3s | |
| docker compose -f dev/docker-compose-integration.yml rm |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import pyarrow.fs as fs | |
| # List of URIs to test | |
| file_uris = [ | |
| "file:some/thing.csv", | |
| "file://some/thing.csv", | |
| "file:/some/thing.csv", | |
| "file:///some/thing.csv" | |
| ] |
kevinjqliu
/ gist:c8310b6253beab52cce93391df03bfe4
Created
September 11, 2024 00:11
`make-integration` error
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ============================================= short test summary info ============================================== | |
| FAILED tests/integration/test_reads.py::test_table_scan_empty_table[session_catalog_hive] - OSError: When getting information for key 'hive/test_table_scan_empty_table/metadata/00000-3959f563-e734-4011-b... | |
| FAILED tests/integration/test_writes/test_partitioned_writes.py::test_query_filter_appended_null_partitioned[2-timestamptz] - OSError: When initiating multiple part upload for key 'default/arrow_table_v2_appended_with_null_partitioned_on... | |
| FAILED tests/integration/test_writes/test_partitioned_writes.py::test_query_filter_appended_null_partitioned[2-timestamp] - OSError: When initiating multiple part upload for key 'default/arrow_table_v2_appended_with_null_partitioned_on... | |
| FAILED tests/integration/test_writes/test_partitioned_writes.py::test_query_filter_v1_v2_append_null[binary] - OSError: When initiating multiple part upload for key 'default/arrow_table_v1_v2_appended_with_null_partition |
NewerOlder