Navigation Menu

Skip to content

Instantly share code, notes, and snippets.

@xhochy

xhochy/PyConDE18-arrow-jdbc.ipynb

Created November 9, 2018 10:28
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save xhochy/2f0b497e474f4c97031e1e92e9ca7a76 to your computer and use it in GitHub Desktop.
Save xhochy/2f0b497e474f4c97031e1e92e9ca7a76 to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
PyConDE18-arrow-jdbc.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@dclong
Copy link

dclong commented Nov 21, 2020

I followed your example here to use pyarrow.jvm to query a Hive database.
However,
after running the following code

batch = jpype.JPackage("org").apache.arrow.adapter.jdbc.JdbcToArrow.sqlToArrow(
    connection,
    query,
    ra
)

it returns only 1024 row.
Basically, batch.getRowCount() return 1024. The table I queries is a huge which has way more than 1024 rows.
Do you have an idea what might have caused the issue?
Do I have to use customized configuration?

@xhochy
Copy link
Author

xhochy commented Dec 8, 2020

@dclong I'm also seeing this behaviour with the newest Arrow release. Did you open an upstream bug about this already?

@dclong
Copy link

dclong commented Dec 9, 2020

No. I figured out that the default configuration has changed. So, one way to fix the issue is to customize the underlying configuration.

@ortap
Copy link

ortap commented Jul 21, 2021

This looks awesome and I am trying to implement this but I run into errors. Like @dclong , I am trying to connect to hive. I am successfully able to get the "batch" (VectorSchemeRoot object). However, when I try to pass the VectorSchemeRoot through pyarrow.jvm.record_batch. It gives me the following error: expected bytes, java.lang.String found.

Have either of you tried this gist out recently? Would you know what the issue might be?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment