Created
August 10, 2014 16:30
-
-
Save codersquid/bb6bfab7209b786b081e to your computer and use it in GitHub Desktop.
ia-wrapper UnicodeEncodeError example using http://video.ep14.c3voc.de/3.mp4
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ | |
"3": { | |
"abstract": "Lessons learned when building a client for a fully distributed system and trying to minimize context-switching pains when using multiple languages.", | |
"collection": "opensource_movies", | |
"conference": "EuroPython 2014", | |
"date": "2014-07-24T14:00:00+0200", | |
"description": "Last year we decided to create official clients for the most popular languages, Python included.\r\n\r\nSome of the goals were:\r\n\r\n* support the complete API of elasticsearch including all parameters\r\n* provide a 1-to-1 mapping to the rest API to avoid having opinions and provide a familiar interface to our users consistent across languages and evironments\r\n* degrade gracefully when the es cluster is changing (nodes dropping out or being added)\r\n* flexibility - allow users to customize and extend the clients easily to suit their, potentially unique, environment\r\n\r\nIn this talk I would like to take you through the process of designing said client, the challenges we faced and the solutions we picked. Amongst other things I will touch on the difference between languages (and their respective communities), the architecture of the client itself, mapping out the API and making sure it stays up to date and integrating with existing tools.\r\n", | |
"extent": "00:30", | |
"is-part-of": "https://ep2014.europython.eu/en/", | |
"language": "en", | |
"location": "Berlin, Germany", | |
"mediatype": "movies", | |
"schedule_event_id": "3", | |
"schedule_event_type": "lecture", | |
"speakers": [ | |
"Honza Kr\u00e1l" | |
], | |
"title": "Lessons learned from building Elasticsearch client", | |
"type": "conference", | |
"year": "2014" | |
} | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#! /usr/bin/env python | |
# -*- coding: utf-8 -*- | |
import json | |
import os | |
from internetarchive import get_item | |
ACCESS_KEY="access" | |
SECRET_KEY="secret" | |
if __name__ == '__main__': | |
with open('example.json') as fh: | |
schedule = json.load(fh) | |
md = schedule['3'] | |
item = get_item('europython_2014_event_3') | |
# this will throw a UnicodeEncodeError | |
item.upload(os.path.abspath('videos/3.mp4'), metadata=md, access_key=ACCESS_KEY, secret_key=SECRET_KEY) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
./example.py ~/upload | |
Traceback (most recent call last): | |
File "./example.py", line 18, in <module> | |
item.upload(os.path.abspath('videos/3.mp4'), metadata=md, access_key=ACCESS_KEY, secret_key=SECRET_KEY) | |
File "/home/sheila/.virtualenvs/upload/src/internetarchive/internetarchive/item.py", line 579, in upload | |
resp = self.upload_file(body, key=key, **kwargs) | |
File "/home/sheila/.virtualenvs/upload/src/internetarchive/internetarchive/item.py", line 494, in upload_file | |
prepared_request = request.prepare() | |
File "/home/sheila/.virtualenvs/upload/src/internetarchive/internetarchive/iarequest.py", line 51, in prepare | |
queue_derive=self.queue_derive, | |
File "/home/sheila/.virtualenvs/upload/src/internetarchive/internetarchive/iarequest.py", line 69, in prepare | |
self.prepare_headers(headers, metadata, queue_derive) | |
File "/home/sheila/.virtualenvs/upload/src/internetarchive/internetarchive/iarequest.py", line 113, in prepare_headers | |
if any(c in str(value) for c in ['\n', '\r']): | |
File "/home/sheila/.virtualenvs/upload/src/internetarchive/internetarchive/iarequest.py", line 113, in <genexpr> | |
if any(c in str(value) for c in ['\n', '\r']): | |
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 8: ordinal not in range(128) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment