ianmcook

Arrow JS IPC compression interoperability

This example demonstrates two different results for Arrow IPC streams with compressed body buffers:

1. Arrow JS 21.2.0 can read a compressed IPC stream written by PyArrow.
2. PyArrow cannot read a compressed IPC stream written by Arrow JS 21.2.0.

The same behavior occurs with both allowed codecs (ZSTD and LZ4 Frame). The example below uses ZSTD.

ianmcook

{
  "statement": "SELECT * FROM MYTABLE",
  "resultSetMetaData": {
    "format": "arrowv1"
  },
  "timeout": 60,
  "database": "MYDATABASE",
  "schema": "MYSCHEMA",
  "warehouse": "MYWAREHOUSE",
  "role": "MYROLE"

ianmcook

C++

AdbcError error = {};
AdbcDatabase database = {};
AdbcDatabaseNew(&database, &error);

// AdbcDatabaseSetOption(&database, "driver", "<driver>", &error) // should be able to omit this...

AdbcDatabaseSetOption(&database, "uri", "<driver>:<scheme>://<address>", &error); // ...and include only this

ianmcook

To create games.duckdb, download games.txt from https://github.com/ianmcook/coursera-datasets/blob/master/games.txt, open duckdb and run:

ATTACH 'games.duckdb' as main_db (BLOCK_SIZE 16384);
USE main;
CREATE TABLE games (
    id INTEGER,
    name VARCHAR,
    inventor VARCHAR,
 year VARCHAR,

ianmcook

library(adbcdrivermanager)

drv <- adbc_driver("snowflake")

db <- adbc_database_init(
  drv,
  username="USER",
  password="PASS",
  adbc.snowflake.sql.account="ACCOUNT-IDENT",
  adbc.snowflake.sql.warehouse="MY_WAREHOUSE",

ianmcook

{
  "statement": "SELECT * FROM MYTABLEONE; SELECT * FROM MYTABLETWO",
  "parameters": {
      "MULTI_STATEMENT_COUNT": "2"
  },
  "resultSetMetaData": {
    "format": "arrowv1"
  },
  "timeout": 60,
  "database": "MYDATABASE",

ianmcook

adbc_driver_flightsql 
adbc_driver_manager
adbc_driver_postgresql
adbc_driver_sqlite
furo
numpydoc
pandas
polars
sphinx
sphinx-copybutton

ianmcook

First start an HTTP server to serve Arrow IPC stream data. You can do this using one of the server examples in HTTP GET Arrow Data: Simple Examples or simply by starting a Python HTTP server in the same directory where you have an Arrow IPC stream file (named file.arrows in this example).

python -m http.server 8008

Download the attached Python script script.py. You might need to do chmod +x script.py to make it executable.

ianmcook

#include <iostream>
#include <arrow/api.h>
#include <arrow/result.h>
#include <arrow/compute/api.h>
#include <arrow/compute/exec/exec_plan.h>

arrow::Status ExecutePlanAndCollectAsTable(
  std::shared_ptr<arrow::compute::ExecPlan> plan,
  std::shared_ptr<arrow::Schema> schema,
  arrow::AsyncGenerator<std::optional<arrow::compute::ExecBatch>> sink_gen) {

ianmcook

import pandas as pd
import pyarrow as pa

file_path = 'fruit.arrow'
stream_path = 'fruit.arrows'

df = pd.DataFrame(data={'fruit': ['apple', 'apple', 'apple', 'orange', 'orange', 'orange'],
                        'variety': ['gala', 'honeycrisp', 'fuji', 'navel', 'valencia', 'cara cara'],
                        'weight': [134.2 , 158.6, None, 142.1, 96.7, None]})