Skip to content

Instantly share code, notes, and snippets.

View wallacegsouza's full-sized avatar

Wallace Gomes de Souza wallacegsouza

View GitHub Profile
@wallacegsouza
wallacegsouza / token_report.py
Created October 30, 2025 13:22
Report on the number of tokens in each file of the project.
import sys, os, argparse, fnmatch, json, csv
from collections import defaultdict
# Optional plotting (enabled via --chart). We import lazily in render_charts().
# Backend will be set to 'Agg' for headless environments.
try:
import tiktoken
except ImportError:
print("Please install tiktoken first: pip install tiktoken", file=sys.stderr); sys.exit(1)