Skip to content

Instantly share code, notes, and snippets.

View ctlllll's full-sized avatar
🚀
Building

Tianle Cai ctlllll

🚀
Building
View GitHub Profile
@ctlllll
ctlllll / longest_chinese_tokens_gpt4o.py
Created May 13, 2024 19:53
Longest Chinese tokens in gpt4o
import tiktoken
import langdetect
T = tiktoken.get_encoding("o200k_base")
length_dict = {}
for i in range(T.n_vocab):
try:
length_dict[i] = len(T.decode([i]))
except:
@ctlllll
ctlllll / Sqrt
Created October 10, 2017 07:18
Naive Sqrt Algorithm
def Sqrt(x):
r1=x
r2=x+1
while abs(r1-r2)>1e-10:
r1=r2
r2=(r2+x/r2)/2
return r2