Skip to content

Instantly share code, notes, and snippets.

View wesm's full-sized avatar
💭
➡️ ➡️ ➡️

Wes McKinney wesm

💭
➡️ ➡️ ➡️
View GitHub Profile
@wesm
wesm / PandasTour.ipynb
Created February 11, 2013 19:54
Pandas Tour
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@wesm
wesm / parquet-benchmark-20170210.py
Created February 10, 2017 18:07
Parquet multithreaded benchmarks
import gc
import os
import time
import numpy as np
import pandas as pd
from pyarrow.compat import guid
import pyarrow as pa
import pyarrow.parquet as pq
import snappy
@wesm
wesm / CInterfaceExample.ipynb
Created April 2, 2020 01:23
Example of round-tripping Arrow data through the new C ABI/Interface
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@wesm
wesm / gist:1079577
Created July 13, 2011 01:50
Summarizing an Interactive Brokers activity statement
"""
A little script for summarizing my Interactive Broker statements
"""
from pandas import *
from pandas.util.testing import set_trace as st
import numpy as np
from BeautifulSoup import BeautifulSoup
@wesm
wesm / arrow-toolchain.sh
Created July 29, 2018 23:16
Wes's Apache Arrow helper bash scripts
#!/usr/bin/env bash
export ARROW_CLANG_VERSION=6.0
export ARROW_GCC=gcc
export ARROW_GXX=g++
export ARROW_LLVM_VERSION=$ARROW_CLANG_VERSION
export PYARROW_WITH_ORC=1
export PYARROW_WITH_PARQUET=1
export PYARROW_WITH_PLASMA=1
@wesm
wesm / gist:5786965
Created June 15, 2013 04:57
StackOverflow Python Analytics 2009 to present
{
"metadata": {
"name": "StackoverflowAnalytics"
},
"nbformat": 3,
"nbformat_minor": 0,
"worksheets": [
{
"cells": [
{
@wesm
wesm / AfterArrow3246.ipynb
Created August 15, 2019 16:10
ARROW-3246 Benchmarks
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
+ export ARROW_BUILD_TOOLCHAIN=/opt/conda
+ ARROW_BUILD_TOOLCHAIN=/opt/conda
+ mkdir -p /build/lint
+ pushd /build/lint
/build/lint /
+ cmake -GNinja -DARROW_FLIGHT=ON -DARROW_GANDIVA=ON -DARROW_PARQUET=ON -DARROW_PYTHON=ON -DCMAKE_CXX_FLAGS=-D_GLIBCXX_USE_CXX11_ABI=0 -DCMAKE_EXPORT_COMPILE_COMMANDS=ON /arrow/cpp
-- Building using CMake version: 3.13.2
-- Arrow version: 0.13.0 (full: '0.13.0-SNAPSHOT')
-- clang-tidy found at /usr/bin/clang-tidy-7
-- clang-format found at /usr/bin/clang-format-7
/home/wesm/code/arrow/cpp/src/parquet/arrow/reader.h:133: warning: Found ';' while parsing initializer list! (doxygen could be confused by a macro call without semicolon)
/home/wesm/code/arrow/cpp/src/parquet/arrow/reader.h:151: warning: Found ';' while parsing initializer list! (doxygen could be confused by a macro call without semicolon)
/home/wesm/code/arrow/cpp/src/parquet/arrow/reader.h:244: warning: Found ';' while parsing initializer list! (doxygen could be confused by a macro call without semicolon)
/home/wesm/code/arrow/cpp/src/parquet/arrow/reader.h:282: warning: Found ';' while parsing initializer list! (doxygen could be confused by a macro call without semicolon)
/home/wesm/code/arrow/cpp/src/arrow/status.h:373: warning: documented symbol `Status arrow::Status::operator &' was not declared or defined.
/home/wesm/code/arrow/cpp/src/arrow/status.h:381: warning: documented symbol `Status arrow::Status::operator &' was not declared or defined.
/home/wesm/code/arrow/cpp/src/arrow/status.h:389: warning:
gandiva.dll!llvm::Value::setNameImpl(class llvm::Twine const &) C++
gandiva.dll!llvm::Value::setName(class llvm::Twine const &) C++
> [Inline Frame] gandiva.dll!llvm::ExtractValueInst::{ctor}(llvm::Value *) Line 2418 C++
[Inline Frame] gandiva.dll!llvm::ExtractValueInst::Create(llvm::Value *) Line 2356 C++
gandiva.dll!llvm::IRBuilder<llvm::ConstantFolder,llvm::IRBuilderDefaultInserter>::CreateExtractValue(llvm::Value * Agg, llvm::ArrayRef<unsigned int> Idxs, const llvm::Twine & Name) Line 1760 C++
gandiva.dll!gandiva::DecimalIR::ValueSplit::MakeFromStruct(gandiva::DecimalIR * decimal_ir, llvm::Value * dstruct) Line 355 C++
gandiva.dll!gandiva::DecimalIR::AddLarge(const gandiva::DecimalIR::ValueFull & x, const gandiva::DecimalIR::ValueFull & y, const gandiva::DecimalIR::ValueFull & out) Line 243 C++
[External Code]
[Inline Frame] gandiva.dll!std::_Func_class<llvm::Value *>::operator()() Line 279 C++
gandiva.dll!gandiva::FunctionIRBuilder::BuildIfElse(llvm::Value * condition, llvm::Type * ret