- Only call
force_encoding
on BINARY Strings. - When receiving a BINARY string from the network or file system, make sure to
force_encode
it to its correct encoding.- In general, the encoding information is provided in an out-of-band channel, such as the Content-Type header in HTTP
- If you don't know the encoding, the String is BINARY forever and should not be concatenated with non-BINARY strings
- When calling
force_encoding
on a BINARY String, immediately call encode! afterwards. This will transcode the String to thedefault_internal
encoding - When using a regular expression with /u, make sure that only Unicode Strings are possible
- When using a regular expression with /n, make sure that only BINARY Strings are possible
- If you get an incompatible encoding between BINARY (ASCII-8BIT) and another encoding, the correct debugging approach is to identify where the BINARY String came from. Usually, this means that a library read in BINARY data from the network and didn't give it an encoding.
- In app code, never use
force_encoding
to convert BINARY data into a particular encoding. By the time you've reached app code, you have lost the information about which encoding is being used. Instead, find where the String came into Ruby, and fix it to set up the encoding based on the information it knows. - In library code, only use
force_encoding
to convert BINARY data into an encoding if you have information about what encoding is being used. This means that you have a header in network protocols or a magic comment in templates (like ERB) or source files. - Only include the magic comment in source files that actually contain characters from that encoding
- To combine two Strings with known, but different encodings, use
encode
to transcode the Strings into the same encoding, then combine them.
-
-
Save wycats/83c011e40e1970df0ef4 to your computer and use it in GitHub Desktop.
Comment deleted.
lonny: BINARY is an official encoding in Ruby 1.9. It is synonymous with ASCII-8BIT:
ruby-1.9.2-p180 :001 > "abc".encode('binary').encoding
=> #Encoding:ASCII-8BIT
Yehuda, your concise notes and musings have been an inspiration to a few projects of mine. The biggest is TinyTDS which was the first backend connection mode for SQL Server that handled encodings correctly. Both it and the adapter will be released in the upcoming Rails Installer project. I'll be bookmarking this for reference anytime I have to think about encodings again. Keep it up and thanks again for sharing!
thanks :)
use force_encoding to convert
It might be worth a slight wording change to denote that force_encoding
isn't actually "converting" anything, rather switching the encoding flag on the string. No bytes are actually modified and the String isn't even read at all with force_encoding
.
Just to make it absolutely clear what it's doing vs encode
and encode!
at least?
love your new blog design bro