Skip to content

Instantly share code, notes, and snippets.

@a1exlism

a1exlism/large_file_read.py

Last active October 7, 2019 06:28
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save a1exlism/bdf2185772c2612f14a5ee72d638ab64 to your computer and use it in GitHub Desktop.
Save a1exlism/bdf2185772c2612f14a5ee72d638ab64 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
large_file_read.py
# LINK https://segmentfault.com/a/1190000005356779
#!/usr/bin/env python
# coding=utf-8
import sys
import time
start = time.time()
lines = 0
for each_line in sys.stdin:
lines += 1
stop = time.time()
print 'The total lines:', lines, ', it\'s takes', stop - start, 'seconds'
# TIPS bash commands
data du - sh access.log
# 3.1G access.log
data cat access.log | python filter.py
# The total lines: 11024424, it's takes 87.4551298618 seconds
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment