pandas-vs-modin-in-concate

pandas-vs-modin-in-concate.md

pandas vs modin in concate

datasets

system:

• i5-6400
• 24gb ram 2400mhz
• samsung 860 evo ssd
• gtx 10603gb

import time
import pandas as pd

s = time.time()

esea_master_dmg_demos_1 = pd.read_csv("esea_master_dmg_demos.part1.csv")
df = pd.concat([esea_master_dmg_demos_1 for _ in range(5)])

e = time.time()
print("Pandas Loading Time = {}".format(e-s))

Pandas Loading Time = 36.65340209007263

del esea_master_dmg_demos_1
del df
import modin.pandas as pd

s = time.time()

esea_master_dmg_demos_1 = pd.read_csv("esea_master_dmg_demos.part1.csv")
df = pd.concat([esea_master_dmg_demos_1 for _ in range(5)])

e = time.time()

print("Modin Loading Time = {}".format(e-s))

Modin Loading Time = 7.940576076507568