Skip to content

Instantly share code, notes, and snippets.

View shenli's full-sized avatar

Li Shen shenli

View GitHub Profile
@shenli
shenli / csv_reader.py
Created February 10, 2024 17:50
A CSV file reader that could assign text field name
"""
Based on Simple CSV reader. https://github.com/run-llama/llama-hub/blob/2c95b021246b54b0542bf9ed9289828cc9da6654/llama_hub/file/simple_csv/base.py
A parser for tabular data files.
"""
from pathlib import Path
from typing import Any, Dict, List, Optional
from llama_index.readers.base import BaseReader
Feature/Functionality Operational Data Layer Data Warehouse
Purpose Real-time operational support In-depth analysis and reporting
Data Latency Near real-time Typically batch-loaded (daily/weekly)
Data Freshness Current, transient data Historical, aggregated data
Update Frequency Frequently (real-time or near real-time) Infrequently (during ETL processes)
Schema Design Typically normalized Star, snowflake, or galaxy schema
Primary Users Business operations, front-line applications Business analysts, data scientists
Query Complexity Simple, operational queries Complex, ad-hoc analytical queries
Data Lifespan Short-term, constantly updated Long-term, provides historical view
Integration Process Minimal transformation, quick integration Extensive ETL processes, deeper integration
// Copyright 2016 PingCAP, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
```
[BEGIN] 2018/1/9 10:07:49
(gdb) thread apply all bt
Thread 75 (LWP 42463):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:439
#1 0x00000000004293f2 in runtime.futexsleep (addr=0x1b13d20 <runtime.mheap_>, val=2, ns=-1)
at /usr/local/go/src/runtime/os_linux.go:45
#2 0x000000000040f721 in runtime.lock (l=0x1b13d20 <runtime.mheap_>) at /usr/local/go/src/runtime/lock_futex.go:102
#3 0x00000000004222ff in runtime.(*mheap).alloc_m (h=0x1b13d20 <runtime.mheap_>, npage=1, spanclass=12 '\f', large=false,
@shenli
shenli / bench_test.go
Last active August 15, 2016 06:00
Batch bench
package tidb
import (
"math/rand"
"testing"
)
var (
totalCount = int64(1000000)
batchSize = int64(2000)
@shenli
shenli / time.go
Last active March 22, 2016 03:47
MySQL Time test
package main
import (
"database/sql"
"fmt"
_ "github.com/go-sql-driver/mysql"
"time"
)
func main() {
@shenli
shenli / time.go
Created March 22, 2016 03:26
MySQL Time test
package main
import (
"database/sql"
"fmt"
_ "github.com/go-sql-driver/mysql"
"time"
)
func main() {