niyue

1. clone arrow git repo locally (under a macOS in my case)
2. open arrow/python folder with vscode
3. create a Dockerfile for pyarrow development. pyarrow provides such Dockerfile, so you only need to do some linking ln -s examples/minimal_build/Dockerfile.ubuntu Dockerfile
4. Follow the steps here to open a remote container as dev env in vscode
   1. Run "Remote-Containers: Open Folder in Container..." in vscode
   2. Choose Dockerfile you just linked
   3. vscode will build the container and open it later, just wait
5. Enter the container for building pyarrow
6. docker exec -it bash

niyue

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://gist.githubusercontent.com/niyue/664da0717e379fca3c9167b4b258578c/raw/piemaker.schema.json",
  "title": "Piemaker JSON schema",
  "description": "piemaker project descriptor",
  "type": "object",
  "properties": {
    "namespace": {
      "description": "The namespace for the project",
      "type": "string"

niyue

arrow IPC file

magic ARROW1

padding to 8 bytes

streaming format with EOS

Schema

fields (Field[])

• name (string)
• nullable (bool)
• type (Type)
• List

niyue

graph TD
    count --> <%= k1 %>

niyue

https://splunk.box.com/s/9k6jdfck5s86oq0kwcyhiagh2ue9ezws

SHOW schemas 
FROM splunk

SELECT * 
FROM splunk.saved_search."top five sourcetypes"

SELECT *
FROM splunk.search."search index=_internal sourcetype=splunkd_ui_access | head 100 | fields method, spent, status"