Skip to content

Instantly share code, notes, and snippets.

💭
➡️ ➡️ ➡️

Wes McKinney wesm

💭
➡️ ➡️ ➡️
View GitHub Profile
@wesm
wesm / CInterfaceExample.ipynb
Created Apr 2, 2020
Example of round-tripping Arrow data through the new C ABI/Interface
View CInterfaceExample.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@wesm
wesm / AfterArrow3246.ipynb
Created Aug 15, 2019
ARROW-3246 Benchmarks
View AfterArrow3246.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
View iwyu-20190214.txt
+ export ARROW_BUILD_TOOLCHAIN=/opt/conda
+ ARROW_BUILD_TOOLCHAIN=/opt/conda
+ mkdir -p /build/lint
+ pushd /build/lint
/build/lint /
+ cmake -GNinja -DARROW_FLIGHT=ON -DARROW_GANDIVA=ON -DARROW_PARQUET=ON -DARROW_PYTHON=ON -DCMAKE_CXX_FLAGS=-D_GLIBCXX_USE_CXX11_ABI=0 -DCMAKE_EXPORT_COMPILE_COMMANDS=ON /arrow/cpp
-- Building using CMake version: 3.13.2
-- Arrow version: 0.13.0 (full: '0.13.0-SNAPSHOT')
-- clang-tidy found at /usr/bin/clang-tidy-7
-- clang-format found at /usr/bin/clang-format-7
View gist:da6e7ba8cb6aa42875c4f3b98c6a9381
/home/wesm/code/arrow/cpp/src/parquet/arrow/reader.h:133: warning: Found ';' while parsing initializer list! (doxygen could be confused by a macro call without semicolon)
/home/wesm/code/arrow/cpp/src/parquet/arrow/reader.h:151: warning: Found ';' while parsing initializer list! (doxygen could be confused by a macro call without semicolon)
/home/wesm/code/arrow/cpp/src/parquet/arrow/reader.h:244: warning: Found ';' while parsing initializer list! (doxygen could be confused by a macro call without semicolon)
/home/wesm/code/arrow/cpp/src/parquet/arrow/reader.h:282: warning: Found ';' while parsing initializer list! (doxygen could be confused by a macro call without semicolon)
/home/wesm/code/arrow/cpp/src/arrow/status.h:373: warning: documented symbol `Status arrow::Status::operator &' was not declared or defined.
/home/wesm/code/arrow/cpp/src/arrow/status.h:381: warning: documented symbol `Status arrow::Status::operator &' was not declared or defined.
/home/wesm/code/arrow/cpp/src/arrow/status.h:389: warning:
View gist:dd629a5be4fe9fc3cde9b4aa584a73a5
gandiva.dll!llvm::Value::setNameImpl(class llvm::Twine const &) C++
gandiva.dll!llvm::Value::setName(class llvm::Twine const &) C++
> [Inline Frame] gandiva.dll!llvm::ExtractValueInst::{ctor}(llvm::Value *) Line 2418 C++
[Inline Frame] gandiva.dll!llvm::ExtractValueInst::Create(llvm::Value *) Line 2356 C++
gandiva.dll!llvm::IRBuilder<llvm::ConstantFolder,llvm::IRBuilderDefaultInserter>::CreateExtractValue(llvm::Value * Agg, llvm::ArrayRef<unsigned int> Idxs, const llvm::Twine & Name) Line 1760 C++
gandiva.dll!gandiva::DecimalIR::ValueSplit::MakeFromStruct(gandiva::DecimalIR * decimal_ir, llvm::Value * dstruct) Line 355 C++
gandiva.dll!gandiva::DecimalIR::AddLarge(const gandiva::DecimalIR::ValueFull & x, const gandiva::DecimalIR::ValueFull & y, const gandiva::DecimalIR::ValueFull & out) Line 243 C++
[External Code]
[Inline Frame] gandiva.dll!std::_Func_class<llvm::Value *>::operator()() Line 279 C++
gandiva.dll!gandiva::FunctionIRBuilder::BuildIfElse(llvm::Value * condition, llvm::Type * ret
View date32.py
In [1]: paste
import pandas as pd
import feather
data = {'date': ['2014-05-01 18:47:05.069722', '2014-05-01 18:47:05.119994',
'2014-05-02 18:47:05.178768', '2014-05-02 18:47:05.230071',
'2014-05-02 18:47:05.230071', '2014-05-02 18:47:05.280592']}
df = pd.DataFrame(data, columns = ['date'])
df['date'] = pd.to_datetime(df['date']).dt.date
View gist:4cc5c786c4fc37310b9af3b24a819fa2
$ ll /usr/lib/llvm-6.0/lib
total 163836
-rw-r--r-- 1 root root 14600 Oct 18 13:44 BugpointPasses.so
drwxr-xr-x 3 root root 4096 Jun 2 19:58 clang/
drwxr-xr-x 5 root root 4096 Jun 2 19:58 cmake/
lrwxrwxrwx 1 root root 40 Oct 18 13:44 libclang-6.0.so.1 -> ../../x86_64-linux-gnu/libclang-6.0.so.1
lrwxrwxrwx 1 root root 17 Oct 18 13:44 libclang.so.1 -> libclang-6.0.so.1
-rw-r--r-- 1 root root 591060 Oct 18 13:44 libFuzzer.a
lrwxrwxrwx 1 root root 39 Oct 18 13:44 libLLVM-6.0.1.so -> ../../x86_64-linux-gnu/libLLVM-6.0.so.1
lrwxrwxrwx 1 root root 39 Oct 18 13:44 libLLVM-6.0.1.so.1 -> ../../x86_64-linux-gnu/libLLVM-6.0.so.1
View gist:ac8f6f8072620e0bb4c9de07f7deb5f3
/arrow/r /
* checking for file './DESCRIPTION' ... OK
* preparing 'arrow':
* checking DESCRIPTION meta-information ... OK
* cleaning src
* running 'cleanup'
* checking for LF line-endings in source and make files and shell scripts
* checking for empty or unneeded directories
* building 'arrow_0.0.0.9000.tar.gz'
View parquet-changes.diff
diff --git a/cpp/src/parquet/.parquetcppversion b/cpp/src/parquet/.parquetcppversion
index d65937f10..f825f7c7f 100644
--- a/cpp/src/parquet/.parquetcppversion
+++ b/cpp/src/parquet/.parquetcppversion
@@ -1 +1 @@
-1.4.1-SNAPSHOT
+1.5.1-SNAPSHOT
diff --git a/cpp/src/parquet/arrow/arrow-reader-writer-test.cc b/cpp/src/parquet/arrow/arrow-reader-writer-test.cc
index 5f4e12349..086672711 100644
--- a/cpp/src/parquet/arrow/arrow-reader-writer-test.cc
View gist:7f5defba5fb0de3c3886259c68f51174
read_fastparquet()
File "/home/wesm/miniconda/envs/arrow-dev/bin/ipython", line 11, in <module>
sys.exit(start_ipython())
File "/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/IPython/__init__.py", line 125, in start_ipython
return launch_new_instance(argv=argv, **kwargs)
File "/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/traitlets/config/application.py", line 658, in launch_instance
app.start()
File "/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/IPython/terminal/ipapp.py", line 356, in start
self.shell.mainloop()
File "/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/IPython/terminal/interactiveshell.py", line 485, in mainloop
You can’t perform that action at this time.