Skip to content

Instantly share code, notes, and snippets.

@KyMidd

KyMidd/lines_per_shard_and_split.sh Secret

Created April 17, 2023 19:40
Show Gist options
  • Save KyMidd/fe22354f8d1b0e912ed8d2873ce17c93 to your computer and use it in GitHub Desktop.
Save KyMidd/fe22354f8d1b0e912ed8d2873ce17c93 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
lines_per_shard_and_split.sh
# Number of lines per shard, add 1 to round up fractional shards
# Lines aren't duplicated, last shard is slightly short of even with others, which is fine
LINES_PER_SHARD=$((ALL_REPOS_LENGTH / SHARD_COUNT))
LINES_PER_SHARD=$((LINES_PER_SHARD + 1))
# Shard the list based on the shard fraction
# This outputs files like all_repos_00, all_repos_01, all_repos_02, etc
split -l $LINES_PER_SHARD -d all_repos all_repos_
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment