Last active
January 14, 2021 19:03
-
-
Save omegaml/d366231c5c2522e71e9cd113f80590be to your computer and use it in GitHub Desktop.
helper function for overly long dataset names (overcoming a monogdb limitation)
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
def put_longname(store, obj, name, **kwargs): | |
""" helper function to overcome mongodb limitation on namespace length | |
Only use to store Pandas dataframes and series | |
Usage: | |
Copy/paste this function into your code base | |
# instead of | |
meta = om.datasets.put(obj, name) | |
# use | |
meta = put_longname(om.datasets, obj, name) | |
How it works: | |
1. Generates a hashed name which is safe to | |
use within the mongodb namespace length limits | |
2. stores the object | |
3. sets the metadata name to the given name, | |
which can be arbitrary length (within mongodb | |
key lenght limits) | |
See also: | |
https://github.com/omegaml/omegaml/issues/153 | |
""" | |
from hashlib import md5 | |
if not kwargs.get('append') or kwargs.get('replace'): | |
kwargs.pop('append', False) | |
kwargs.pop('replace', False) | |
store.drop(name, force=True) | |
if store.metadata(name) is None: | |
safe_name = md5(name.encode('utf8')).hexdigest() | |
meta = store.put(obj, safe_name, **kwargs) | |
meta.name = name | |
meta.save() | |
else: | |
meta = store.put(obj, name, **kwargs) | |
return meta |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment