https://twitter.com/dancow/status/1298632074849587204 reply to dancow

1298632074849587204.py

import pandas as pd
# https://burntsushi.net/stuff/worldcitiespop.csv
df = pd.read_csv('worldcitiespop.csv',low_memory=False)
df = df.query('Country == "us"')
print(len(df))
# prints 141989

Python 3.8.5
Pandas 1.1.1

takes less than 5 seconds on a 2020 macbook i5

slightly optimized code still takes more than 4 seconds:

import pandas as pd
#url = 'https://burntsushi.net/stuff/worldcitiespop.csv'
url = 'worldcitiespop.csv'
iter_csv = pd.read_csv('worldcitiespop.csv', iterator=True, chunksize=10000)
df = pd.concat([chunk.query('Country == "us"') for chunk in iter_csv])
print(len(df))

Approaching 3 seconds with this optimization

import pandas as pd
#url = 'https://burntsushi.net/stuff/worldcitiespop.csv'
url = 'worldcitiespop.csv'
iter_csv = pd.read_csv('worldcitiespop.csv', iterator=True, chunksize=100000, low_memory=False)
df = pd.concat([chunk.query('Country == "us"') for chunk in iter_csv])
print(len(df))