Skip to content

Instantly share code, notes, and snippets.

@rhoboro
Last active March 16, 2019 03:08
Show Gist options
  • Save rhoboro/6b0d5cec76f07b866528c75a2d104b60 to your computer and use it in GitHub Desktop.
Save rhoboro/6b0d5cec76f07b866528c75a2d104b60 to your computer and use it in GitHub Desktop.
リストとジェネレータのメモリ効率を比較

リストとジェネレターの比較

1MBの10000個のファイルをコピーする処理でリストとジェネレータのメモリ効率を比較します。

実行方法

$ chmod +x mkdummy.sh
$ docker build -t copy:latest .
# リストで実行
$ docker run --rm -it copy:latest
# ジェネレータで実行
$ docker run --rm -it copy:latest --gen
# メモリを600Mに制限して実行
$ docker run --rm -it -m "600MB" copy:latest --gen
import argparse
@profile
def main(is_gen):
if is_gen:
call_gen()
else:
call_list()
@profile
def call_list():
for i, dummy in enumerate(do_list('dummy', 10000)):
with open('dest' + str(i), 'w') as f:
f.write(dummy)
def do_list(src, n):
rv = []
for i in range(n):
with open(src + str(i)) as f:
rv.append(f.read())
return rv
@profile
def call_gen():
for i, dummy in enumerate(do_gen('dummy', 10000)):
with open('dest' + str(i), 'w') as f:
f.write(dummy)
def do_gen(src, n):
for i in range(n):
with open(src + str(i)) as f:
yield f.read()
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('--gen', action="store_true")
args = parser.parse_args()
main(args.gen)
FROM python:3.7
WORKDIR /usr/src/app
RUN pip install psutil memory-profiler
COPY . .
RUN ./mkdummy.sh
ENTRYPOINT ["python", "-m", "memory_profiler", "copy_file.py"]
for i in `seq 0 10000`
do
dd if=/dev/zero of=dummy$i bs=1M count=1
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment