Skip to content

Instantly share code, notes, and snippets.

@agalea91

agalea91/ssh_pd_read_csv.py

Last active January 5, 2021 00:17
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save agalea91/7f8aa75fd8a2c03805192a9d643a0a48 to your computer and use it in GitHub Desktop.
Save agalea91/7f8aa75fd8a2c03805192a9d643a0a48 to your computer and use it in GitHub Desktop.
Download ZIP
Read CSV file on remote server over SSH, using local system RSA auth
Raw
ssh_pd_read_csv.py
SSH_HOST = ""
SSH_USER = ""
SSH_FILE_PATH = ""
COMPRESSION = "zip"
from paramiko import SSHClient
client = SSHClient()
client.load_system_host_keys()
client.connect(SSH_HOST, username=SSH_USER)
sftp_client = client.open_sftp()
f = sftp_client.open(SSH_FILE_PATH)
df_chunks = pd.read_csv(f, chunksize=100000, compression=COMPRESSION)
for i, df in enumerate(df_chunks):
display(df.head())
print(i)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment