Skip to content

Instantly share code, notes, and snippets.

@omegaml
Last active January 14, 2021 19:03
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save omegaml/d366231c5c2522e71e9cd113f80590be to your computer and use it in GitHub Desktop.
Save omegaml/d366231c5c2522e71e9cd113f80590be to your computer and use it in GitHub Desktop.
helper function for overly long dataset names (overcoming a monogdb limitation)
def put_longname(store, obj, name, **kwargs):
""" helper function to overcome mongodb limitation on namespace length
Only use to store Pandas dataframes and series
Usage:
Copy/paste this function into your code base
# instead of
meta = om.datasets.put(obj, name)
# use
meta = put_longname(om.datasets, obj, name)
How it works:
1. Generates a hashed name which is safe to
use within the mongodb namespace length limits
2. stores the object
3. sets the metadata name to the given name,
which can be arbitrary length (within mongodb
key lenght limits)
See also:
https://github.com/omegaml/omegaml/issues/153
"""
from hashlib import md5
if not kwargs.get('append') or kwargs.get('replace'):
kwargs.pop('append', False)
kwargs.pop('replace', False)
store.drop(name, force=True)
if store.metadata(name) is None:
safe_name = md5(name.encode('utf8')).hexdigest()
meta = store.put(obj, safe_name, **kwargs)
meta.name = name
meta.save()
else:
meta = store.put(obj, name, **kwargs)
return meta
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment