Skip to content

Instantly share code, notes, and snippets.

@hdary85
Created September 28, 2023 02:07
Show Gist options
  • Save hdary85/f0135b958d137c448bd8358d551b0540 to your computer and use it in GitHub Desktop.
Save hdary85/f0135b958d137c448bd8358d551b0540 to your computer and use it in GitHub Desktop.
with open('your_file.txt', 'r') as file:
paragraphs = file.read().split('$')
# Remove leading and trailing whitespace from each paragraph
paragraphs = [p.strip() for p in paragraphs]
# Remove duplicates while preserving order
unique_paragraphs = []
seen = set()
for paragraph in paragraphs:
if paragraph not in seen:
unique_paragraphs.append(paragraph)
seen.add(paragraph)
# Write the unique paragraphs back to a new file with the same delimiter
with open('output_file.txt', 'w') as file:
file.write('$'.join(unique_paragraphs))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment