Skip to content

Instantly share code, notes, and snippets.

View shamb0's full-sized avatar
🎯
Focusing

RK shamb0

🎯
Focusing
  • Rust Enthusiast Open to Hire | Exploring Web3, WASM, and Cloud-Native
  • India
  • X @0shamb0
View GitHub Profile

ChessMate: Chess + Teammate (AI companion), Learn Languages Through Chess - Complete Technical Blueprint

📖 Introduction

What is the System?

"Learn Languages Through Chess" is an innovative AI-powered educational platform that combines chess instruction with language learning. The system uses adaptive artificial intelligence to provide personalized chess coaching in multiple languages, adjusting both the complexity of chess concepts and language difficulty based on the learner's cognitive stage and linguistic proficiency.

Why Are We Creating It?

🎯 Core Problem Statement

Groq, templates/exp01-groq-cfg1-kwaak.toml

[

language = "rust"
tavily_api_key = "env:TAVILY_API_KEY"
tool_executor = "docker"

[commands]
test = "cargo test --no-fail-fast --color=never"

image

image

thread 'rustc' panicked at src/helpers.rs:718:9:
assertion failed: base_layout.size >= offset + value_layout.size
stack backtrace:
Auto Sales Benchmarks/Total Sales/Local Disk Cache Disabled
                        time:   [3.1286 ms 3.3002 ms 3.5393 ms]
                        thrpt:  [2.8254 Melem/s 3.0301 Melem/s 3.1963 Melem/s]
                 change:
                        time:   [+2.0064% +9.0733% +15.690%] (p = 0.02 < 0.05)
                        thrpt:  [-13.562% -8.3185% -1.9670%]
                        Performance has regressed.
2024-09-14 09:34:15.444  INFO cache_performance: cache_benchmark finished
  • Code snippet from the module pg_analytics/tests/test_mlp_auto_sales.rs
#[rstest]
async fn test_duckdb_object_cache_performance(
    #[future] s3: S3,
    mut conn: PgConnection,
    parquet_path: PathBuf,
2024-08-29 22:05:13.804 IST [629116] LOG:  received fast shutdown request
2024-08-29 22:05:13.810 IST [629116] LOG:  aborting any active transactions
2024-08-29 22:05:13.813 IST [629116] LOG:  background worker "logical replication launcher" (PID 629139) exited with exit code 1
2024-08-29 22:05:13.850 IST [629124] LOG:  shutting down
2024-08-29 22:05:13.856 IST [629124] LOG:  checkpoint starting: shutdown immediate
2024-08-29 22:05:13.879 IST [629124] LOG:  checkpoint complete: wrote 0 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.001 s, sync=0.001 s, total=0.029 s; sync files=0, longest=0.000 s, average=0.000 s; distance=0 kB, estimate=4350 kB; lsn=0/91BE9940, redo lsn=0/91BE9940
2024-08-29 22:05:13.889 IST [629116] LOG:  database system is shut down
2024-08-29 22:05:20.348 IST [754139] WARNING:  pga:: extension is being initialized
2024-08-29 22:05:20.362 IST [754139] LOG:  starting PostgreSQL 16.4 (Ubuntu 16.4-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1

Multi-level Partitioned Tables: TL;DR

Concept

  • Divides large tables into smaller, hierarchical partitions
  • Example: Two-level partitioning by year and manufacturer for auto sales data
  • Benefits: Improved query performance, efficient data management, parallel query execution, simplified maintenance, and better data organization

Demo Setup and Implementation Details

  • Components: PostgreSQL, S3 Storage, pg_analytics Foreign Data Wrapper (FDW), DuckDB
  • Key steps:

Multi-level Partitioned Tables

Concept

Multi-level partitioned tables are a powerful feature in database systems that allow us to divide large tables into smaller, more manageable pieces called partitions. These partitions are organized in a hierarchical structure, with each level representing a different category of data.

In our auto sales dataset scenario, we use a two-level partitioning strategy:

  1. First level: Partitioned by year
  2. Second level: Partitioned by manufacturer
2024-08-20T05:45:24.330137Z  INFO test_mlp_auto_sales::datasets::auto_sales: Completed data upload to S3
2024-08-20T05:45:24.330531Z  INFO sqlx::postgres::notice: table "auto_sales_partitioned" does not exist, skipping
2024-08-20T05:45:24.330715Z  INFO sqlx::postgres::notice: server "auto_sales_server" does not exist, skipping
2024-08-20T05:45:24.330874Z  INFO sqlx::postgres::notice: foreign-data wrapper "parquet_wrapper" does not exist, skipping
2024-08-20T05:45:24.331018Z  INFO sqlx::postgres::notice: server "auto_sales_server" does not exist, skipping
2024-08-20T05:45:25.205625Z  INFO test_mlp_auto_sales::datasets::auto_sales: Starting assert_total_sales test with query: 
            SELECT year, manufacturer, ROUND(SUM(price)::numeric, 4)::float8 as total_sales
            FROM auto_sales_partitioned
            WHERE year BETWEEN 2020 AND 2024