Skip to content

Instantly share code, notes, and snippets.

@zoltanctoth

zoltanctoth/split_text.py

Last active November 29, 2023 17:59
Show Gist options
  • Save zoltanctoth/6f0f69fa00adc086d47bbc95ea4a43e9 to your computer and use it in GitHub Desktop.
Save zoltanctoth/6f0f69fa00adc086d47bbc95ea4a43e9 to your computer and use it in GitHub Desktop.
Download ZIP
Split text in max 5000-character-long parts
Raw
split_text.py
long_text = 3000 * "Lorem ipsum dolor sit amet, consectetur adipiscing elit. "
COMPREHEND_LIMIT = 5000
lines = long_text.split(".")
current_text = ""
for line in lines:
if len(current_text + line) > COMPREHEND_LIMIT:
# EXECUTE COMPREHEND
print(f"Executing Comprehend on {len(current_text)} characters")
current_text = ""
current_text = current_text + line
if current_text != "":
# EXECUTE COMPREHEND
print(f"Executing Comprehend on {len(current_text)} characters)")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment