Skip to content

Instantly share code, notes, and snippets.

@hornc

hornc/unicode

Last active May 10, 2017 11:57
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save hornc/cd4d476b39f6d6a4fa9f0c10a3151e5d to your computer and use it in GitHub Desktop.
Save hornc/cd4d476b39f6d6a4fa9f0c10a3151e5d to your computer and use it in GitHub Desktop.
Download ZIP
Raw
unicode
from unicodedata import normalize
def norm(s):
return normalize('NFC', s)
def is_nfc(s):
norm = normalize('NFC', s)
print "%s => %s" % (s, norm)
return s == norm
def fix_half(s):
return s.replace(u'\ufe21', '')
def fix_unicode(record):
for k,v in record.iteritems():
if type(v) == unicode:
record[k] = norm(fix_half(v))
record['_comment'] = "fix unicode"
return record
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment