Sanket Kedia sanketkedia
sanketkedia
/ wikipedia_cohere_dataset.py
Created
April 14, 2026 21:12
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| """Shared Hugging Face helpers for Cohere Wikipedia embedding parquet shards.""" | |
| from __future__ import annotations | |
| import logging | |
| import random | |
| import time | |
| from pathlib import Path | |
| from typing import List |
sanketkedia
/ ingest_250m2.py
Created
April 14, 2026 20:36
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env python3 | |
| from __future__ import annotations | |
| import argparse | |
| import json | |
| import logging | |
| import os | |
| import queue | |
| import random | |
| import signal |