Skip to content

Instantly share code, notes, and snippets.

@u1735067
Created September 18, 2019 15:46
Show Gist options
  • Save u1735067/3944bcd0cce095cc6a2d23e8c9be2aad to your computer and use it in GitHub Desktop.
Save u1735067/3944bcd0cce095cc6a2d23e8c9be2aad to your computer and use it in GitHub Desktop.
Chardet (https://github.com/chardet/chardet) automatic open (from CLI example)
from chardet.universaldetector import UniversalDetector
def open_chardet(file, *args, **kwargs):
detector = UniversalDetector()
with open(file, mode='rb') as fh:
for line in fh:
line = bytearray(line)
detector.feed(line)
if detector.done:
break
detector.close()
if detector.result['encoding']:
print('Detected encoding: {} (confidence {})'.format(
detector.result['encoding'],
detector.result['confidence']
))
kwargs['encoding'] = detector.result['encoding']
return open(file, *args, **kwargs)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment