Skip to content

Instantly share code, notes, and snippets.

View openinx's full-sized avatar

openinx openinx

View GitHub Profile
@openinx
openinx / gist:6026089
Created July 18, 2013 01:43
A MySQL Client Ternimal implemented by pure PYTHON.
import time
import pymysql
import prettytable
import itertools
import traceback
import ConfigParser
from nepo.common import utils
import nepo.web.constant as web_const
@openinx
openinx / gist:9486743
Created March 11, 2014 14:22
PyMySQL如何一般执行SELECT * FROM BIG_TABLE语句一边fetch数据呢?
conn = pymysql.connect(user='root',host='127.0.0.1', port=3306, db='test')
try:
cursor = conn.cursor(pymysql.cursors.SSCursor)
cursor.execute('select * from sbtest')
while True:
row = cursor.fetchone()
if not row:
break
print row
finally:
@openinx
openinx / gist:9757208
Created March 25, 2014 08:17
use a dict cursor.
import pymysql
from contextlib import contextmanager
#from contextlib import contextmanager
@contextmanager
def create_dict_cursor():
with pymysql.connect(user='root', passwd='goodboy', host='127.0.0.1', cursorclass=pymysql.cursors.DictCursor) as cursor:
yield cursor
@openinx
openinx / compare_dict.py
Created April 23, 2014 16:10
How to compare two dict.
def assertDictMatch(self, d1, d2, approx_equal=False, tolerance=0.001):
"""Assert two dicts are equivalent.
This is a 'deep' match in the sense that it handles nested
dictionaries appropriately.
NOTE:
If you don't care (or don't know) a given value, you can specify
the string DONTCARE as the value. This will cause that dict-item
@openinx
openinx / ycsb-data.py
Created January 8, 2019 01:59
Load the qps and latency of ycsb log into MySQL.
#!/usr/bin/python
import sys
import time
import datetime
import re
import pymysql
import pymysql.cursors
# create table `ycsb`(
@openinx
openinx / stacktrace.java
Created December 26, 2023 05:46
stacktrace
spark-sql> select * from default.test_table_1;
23/12/26 13:37:52 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0) (core-1-2.emr-967e350d6501c1edbb51.cn-beijing.emr-volces.com executor 1): java.lang.UnsupportedOperationException: Unsupported type: UTF8String
at org.apache.iceberg.arrow.vectorized.ArrowVectorAccessor.getUTF8String(ArrowVectorAccessor.java:81)
at org.apache.iceberg.spark.data.vectorized.IcebergArrowColumnVector.getUTF8String(IcebergArrowColumnVector.java:138)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:760)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:364)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890)
at org.apache.spark.rdd.RDD.$anonfun$mapParti