Skip to content

Instantly share code, notes, and snippets.

@tvass
Created August 20, 2018 14:04
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save tvass/4106056e3d645d5b226e2ef67100f67e to your computer and use it in GitHub Desktop.
Save tvass/4106056e3d645d5b226e2ef67100f67e to your computer and use it in GitHub Desktop.
Performance issue with Ansible when using 'stat' module
pwd : /home/tvass/Documents/work/ansible-perf
uname : Linux XXXXX 4.17.9-200.fc28.x86_64 #1 SMP Mon Jul 23 21:41:29 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Ansible version : ansible 2.6.2 (same issue w/ 2.5.8 and 2.4.6.0)
1) Generate 300 random files
---
#!/bin/bash
# Generates ten data files, each 300 MB in size and filled with random data.
[[ ! -d "data/" ]] && mkdir "data/"
for i in {1..300}; do
dd if="/dev/urandom" of="data/file${i}" bs=1024 count=10240
done
---
2) Time with sha256sum
$ time sha256sum data/file*
[...]
real 0m9.400s
user 0m7.469s
sys 0m0.608s
3) Time with Python
---
#!/usr/bin/env python
import hashlib
import sys
def sha256_checksum(filename, block_size=65536):
sha256 = hashlib.sha256()
with open(filename, 'rb') as f:
for block in iter(lambda: f.read(block_size), b''):
sha256.update(block)
return sha256.hexdigest()
def main():
for f in sys.argv[1:]:
checksum = sha256_checksum(f)
print(checksum + ' ' + f)
if __name__ == '__main__':
main()
---
$ time ./hash.py data/*
[...]
real 0m8.292s
user 0m6.800s
sys 0m0.630s
4) Time with Ansible
---
- hosts: localhost
gather_facts: false
strategy: free
tasks:
- name: Check if package is present
stat:
path: "{{ item }}"
checksum_algorithm: sha256
get_attributes: no
get_mime: no
get_md5: no
with_fileglob:
- "/home/tvass/Documents/work/ansible-perf/data/*"
---
$ time ansible-playbook cs.yaml
[...]
real 0m41.888s
user 1m10.612s
sys 0m11.982s
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment