Skip to content

Instantly share code, notes, and snippets.

@glwagner
Last active June 3, 2017 04:05
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save glwagner/4ea389cafb3b4a33586cfcf032cfec45 to your computer and use it in GitHub Desktop.
Save glwagner/4ea389cafb3b4a33586cfcf032cfec45 to your computer and use it in GitHub Desktop.
Provides a python function with an associated generator object that rapidly counts line in a specified text file. This method only works in python 3.
def linecount_generator(reader):
"""Yield a generator object for the purpose of counting lines in a file
at speed. See
https://stackoverflow.com/questions/
845058/how-to-get-line-count-cheaply-in-python/27518377#27518377
"""
b = reader(1024*1024)
while b:
yield b
b = reader(1024*1024)
def countlinesfast(filepath):
"""Count the lines in a file using the generator function
linecount_generator."""
with open(filepath, 'rb') as countfile:
bufgen = linecountgen(countfile.raw.read)
nlines = sum( buf.count(b'\n') for buf in bufgen )
return nlines
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment