snowch

Rust for DataFusion

A complete Rust tutorial for beginners, covering only what DataFusion actually uses.

This assumes you can program — you know what a function, a loop, and a data structure are — but not that you know any Rust. It starts at installation and ends with you writing a working DataFusion operator.

Why a special tutorial?

Rust is a big language, and DataFusion uses a narrow, unusual slice of it. It's saturated with one particular sharing pattern (Arc), leans hard on trait objects, and — importantly — has been deliberately architected to avoid the lifetime puzzles that dominate most Rust teaching. A general Rust course will spend your first two months on skills this codebase engineered around, and skip idioms that appear on every page of it.

snowch

snowch

snowch

############# Markdown parser in Jupyter notebooks ##################

<div class="alert alert-block alert-info"> <b>NOTE</b> Use blue boxes for Tips and notes. </div>

<div class="alert alert-block alert-success"> Use green boxes sparingly, and only for some specific purpose that the other boxes can't cover. For example, if you have a lot of related content to link to, maybe you decide to use green boxes for related links from each section of a notebook.

snowch

2024-11-19T11:57:48.454616Z DEBUG awscrt::libcrypto_resolve: searching process and loaded modules    
2024-11-19T11:57:48.454649Z DEBUG awscrt::libcrypto_resolve: found static aws-lc HMAC symbols    
2024-11-19T11:57:48.454653Z DEBUG awscrt::libcrypto_resolve: found static aws-lc libcrypto 1.1.1 EVP_MD symbols    
2024-11-19T11:57:48.454670Z DEBUG awscrt::libcrypto_resolve: Compiled with libcrypto AWS-LC 1.36.1, linked to libcrypto AWS-LC 1.36.1    
2024-11-19T11:57:48.454679Z  INFO awscrt::tls-handler: static: Initializing TLS using s2n.    
2024-11-19T11:57:48.454964Z DEBUG awscrt::tls-handler: ctx: Based on OS, we detected the default PKI path as /etc/ssl/certs, and ca file as /etc/ssl/certs/ca-certificates.crt    
2024-11-19T11:57:48.455002Z  INFO awscrt::event-loop: id=0x55b812b32a30: Initializing edge-triggered epoll    
2024-11-19T11:57:48.455010Z  INFO awscrt::event-loop: id=0x55b812b32a30: Using eventfd for cross-thread notifications.    
2024-11-19T11:57:48.455016Z  INFO awscrt::event-loop: id=0x55b

snowch

2024-11-14T11:37:16.555306Z DEBUG mountpoint_s3::cli: waiting up to 30 seconds for child process to be ready
2024-11-14T11:37:16.555705Z  INFO mountpoint_s3::cli: mount-s3 1.10.0
2024-11-14T11:37:16.555724Z DEBUG mountpoint_s3::cli: CliArgs { bucket_name: "csnow-bucket", mount_point: "/home/jovyan/s3", prefix: None, region: Some("VAST"), endpoint_url: Some("http://172.200.201.1:80"), force_path_style: false, transfer_acceleration: false, dual_stack: false, requester_pays: false, bucket_type: None, no_sign_request: false, profile: None, read_only: false, storage_class: None, allow_delete: true, allow_overwrite: false, auto_unmount: false, allow_root: false, allow_other: false, maximum_throughput_gbps: None, max_threads: 16, part_size: 8388608, read_part_size: None, write_part_size: None, uid: Some(1000), gid: Some(100), dir_mode: Some(509), file_mode: Some(436), foreground: false, expected_bucket_owner: None, log_directory: Some("/home/jovyan/s3.log"), log_metrics: false, debug: true, debug_crt: true, no_log: 

snowch

export EXT_HOST_NAME="vastdb-ingest"
export EXT_HOST_IP="10.71.212.111"
export STORE_PASSWORD="abcdef"

if [ -d ./certs ]; then
  echo "Directory ./certs already exists. Exiting."
  exit 1
fi

mkdir ./certs

snowch

snowch

sudo apt-get install qemu qemu-system-x86
sudo apt-get install libc6-dev:i386
git clone https://github.com/mit-pdos/xv6-public.git xv6
chmod 700 -R xv6
cd xv6
make
make qemu

snowch

Definition

(from: https://en.wikipedia.org/wiki/Linear_independence)

A sequence of vectors $\mathbf{v}_1, \mathbf{v}_2, \dots, \mathbf{v}_k$ from a vector space ${{mvar|V}}$ is said to be 'linearly dependent', if there exist scalars $a_1, a_2, \dots, a_k,$ not all zero, such that : $a_1\mathbf{v}_1 + a_2\mathbf{v}_2 + \cdots + a_k\mathbf{v}_k = \mathbf{0},$ where $\mathbf{0}$ denotes the zero vector.

Understanding Linear Dependence