Skip to content

Instantly share code, notes, and snippets.

💭
➡️ ➡️ ➡️

Wes McKinney wesm

💭
➡️ ➡️ ➡️
Block or report user

Report or block wesm

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
@wesm
wesm / CInterfaceExample.ipynb
Created Apr 2, 2020
Example of round-tripping Arrow data through the new C ABI/Interface
View CInterfaceExample.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@wesm
wesm / AfterArrow3246.ipynb
Created Aug 15, 2019
ARROW-3246 Benchmarks
View AfterArrow3246.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
View iwyu-20190214.txt
+ export ARROW_BUILD_TOOLCHAIN=/opt/conda
+ ARROW_BUILD_TOOLCHAIN=/opt/conda
+ mkdir -p /build/lint
+ pushd /build/lint
/build/lint /
+ cmake -GNinja -DARROW_FLIGHT=ON -DARROW_GANDIVA=ON -DARROW_PARQUET=ON -DARROW_PYTHON=ON -DCMAKE_CXX_FLAGS=-D_GLIBCXX_USE_CXX11_ABI=0 -DCMAKE_EXPORT_COMPILE_COMMANDS=ON /arrow/cpp
-- Building using CMake version: 3.13.2
-- Arrow version: 0.13.0 (full: '0.13.0-SNAPSHOT')
-- clang-tidy found at /usr/bin/clang-tidy-7
-- clang-format found at /usr/bin/clang-format-7
View gist:da6e7ba8cb6aa42875c4f3b98c6a9381
/home/wesm/code/arrow/cpp/src/parquet/arrow/reader.h:133: warning: Found ';' while parsing initializer list! (doxygen could be confused by a macro call without semicolon)
/home/wesm/code/arrow/cpp/src/parquet/arrow/reader.h:151: warning: Found ';' while parsing initializer list! (doxygen could be confused by a macro call without semicolon)
/home/wesm/code/arrow/cpp/src/parquet/arrow/reader.h:244: warning: Found ';' while parsing initializer list! (doxygen could be confused by a macro call without semicolon)
/home/wesm/code/arrow/cpp/src/parquet/arrow/reader.h:282: warning: Found ';' while parsing initializer list! (doxygen could be confused by a macro call without semicolon)
/home/wesm/code/arrow/cpp/src/arrow/status.h:373: warning: documented symbol `Status arrow::Status::operator &' was not declared or defined.
/home/wesm/code/arrow/cpp/src/arrow/status.h:381: warning: documented symbol `Status arrow::Status::operator &' was not declared or defined.
/home/wesm/code/arrow/cpp/src/arrow/status.h:389: warning:
View gist:dd629a5be4fe9fc3cde9b4aa584a73a5
gandiva.dll!llvm::Value::setNameImpl(class llvm::Twine const &) C++
gandiva.dll!llvm::Value::setName(class llvm::Twine const &) C++
> [Inline Frame] gandiva.dll!llvm::ExtractValueInst::{ctor}(llvm::Value *) Line 2418 C++
[Inline Frame] gandiva.dll!llvm::ExtractValueInst::Create(llvm::Value *) Line 2356 C++
gandiva.dll!llvm::IRBuilder<llvm::ConstantFolder,llvm::IRBuilderDefaultInserter>::CreateExtractValue(llvm::Value * Agg, llvm::ArrayRef<unsigned int> Idxs, const llvm::Twine & Name) Line 1760 C++
gandiva.dll!gandiva::DecimalIR::ValueSplit::MakeFromStruct(gandiva::DecimalIR * decimal_ir, llvm::Value * dstruct) Line 355 C++
gandiva.dll!gandiva::DecimalIR::AddLarge(const gandiva::DecimalIR::ValueFull & x, const gandiva::DecimalIR::ValueFull & y, const gandiva::DecimalIR::ValueFull & out) Line 243 C++
[External Code]
[Inline Frame] gandiva.dll!std::_Func_class<llvm::Value *>::operator()() Line 279 C++
gandiva.dll!gandiva::FunctionIRBuilder::BuildIfElse(llvm::Value * condition, llvm::Type * ret
View date32.py
In [1]: paste
import pandas as pd
import feather
data = {'date': ['2014-05-01 18:47:05.069722', '2014-05-01 18:47:05.119994',
'2014-05-02 18:47:05.178768', '2014-05-02 18:47:05.230071',
'2014-05-02 18:47:05.230071', '2014-05-02 18:47:05.280592']}
df = pd.DataFrame(data, columns = ['date'])
df['date'] = pd.to_datetime(df['date']).dt.date
View gist:4cc5c786c4fc37310b9af3b24a819fa2
$ ll /usr/lib/llvm-6.0/lib
total 163836
-rw-r--r-- 1 root root 14600 Oct 18 13:44 BugpointPasses.so
drwxr-xr-x 3 root root 4096 Jun 2 19:58 clang/
drwxr-xr-x 5 root root 4096 Jun 2 19:58 cmake/
lrwxrwxrwx 1 root root 40 Oct 18 13:44 libclang-6.0.so.1 -> ../../x86_64-linux-gnu/libclang-6.0.so.1
lrwxrwxrwx 1 root root 17 Oct 18 13:44 libclang.so.1 -> libclang-6.0.so.1
-rw-r--r-- 1 root root 591060 Oct 18 13:44 libFuzzer.a
lrwxrwxrwx 1 root root 39 Oct 18 13:44 libLLVM-6.0.1.so -> ../../x86_64-linux-gnu/libLLVM-6.0.so.1
lrwxrwxrwx 1 root root 39 Oct 18 13:44 libLLVM-6.0.1.so.1 -> ../../x86_64-linux-gnu/libLLVM-6.0.so.1
View gist:ac8f6f8072620e0bb4c9de07f7deb5f3
/arrow/r /
* checking for file './DESCRIPTION' ... OK
* preparing 'arrow':
* checking DESCRIPTION meta-information ... OK
* cleaning src
* running 'cleanup'
* checking for LF line-endings in source and make files and shell scripts
* checking for empty or unneeded directories
* building 'arrow_0.0.0.9000.tar.gz'
View parquet-changes.diff
diff --git a/cpp/src/parquet/.parquetcppversion b/cpp/src/parquet/.parquetcppversion
index d65937f10..f825f7c7f 100644
--- a/cpp/src/parquet/.parquetcppversion
+++ b/cpp/src/parquet/.parquetcppversion
@@ -1 +1 @@
-1.4.1-SNAPSHOT
+1.5.1-SNAPSHOT
diff --git a/cpp/src/parquet/arrow/arrow-reader-writer-test.cc b/cpp/src/parquet/arrow/arrow-reader-writer-test.cc
index 5f4e12349..086672711 100644
--- a/cpp/src/parquet/arrow/arrow-reader-writer-test.cc
View gist:7f5defba5fb0de3c3886259c68f51174
read_fastparquet()
File "/home/wesm/miniconda/envs/arrow-dev/bin/ipython", line 11, in <module>
sys.exit(start_ipython())
File "/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/IPython/__init__.py", line 125, in start_ipython
return launch_new_instance(argv=argv, **kwargs)
File "/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/traitlets/config/application.py", line 658, in launch_instance
app.start()
File "/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/IPython/terminal/ipapp.py", line 356, in start
self.shell.mainloop()
File "/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/IPython/terminal/interactiveshell.py", line 485, in mainloop
You can’t perform that action at this time.