Skip to content

Instantly share code, notes, and snippets.

@akoumjian
Last active November 6, 2022 00:51
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save akoumjian/5cb22aa83602e7f152e75e142154e9dc to your computer and use it in GitHub Desktop.
Save akoumjian/5cb22aa83602e7f152e75e142154e9dc to your computer and use it in GitHub Desktop.
Generate CRC32 base64 encoded hash the same way google does
"""
Google makes it difficult to verify if local files match the ones on their cloud storage.
This snippet takes a file path to run the CRC32C hashing and then base64 encode it so it matches the results
from the google-cloud-storage python blob api
You will need pip install:
- google-cloud-storage
"""
import google_crc32c
import collections
def generate_file_crc32c(path, blocksize=2**20):
"""
Generate a base64 encoded crc32c checksum for a file to compare with google cloud storage.
Returns a string like "4jvPnQ=="
Compare with a google storage blob instance:
blob.crc32c == generate_file_crc32c("path/to/local/file.txt")
"""
crc = google_crc32c.Checksum()
read_stream = open(path, "rb")
collections.deque(crc.consume(read_stream, blocksize), maxlen=0)
read_stream.close()
return base64.b64encode(crc.digest()).decode("utf-8")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment