Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
TextEncoder/TextDecoder polyfills for utf-8
// TextEncoder/TextDecoder polyfills for utf-8 - an implementation of TextEncoder/TextDecoder APIs
// Written in 2013 by Viktor Mukhachev <vic99999@yandex.ru>
// To the extent possible under law, the author(s) have dedicated all copyright and related and neighboring rights to this software to the public domain worldwide. This software is distributed without any warranty.
// You should have received a copy of the CC0 Public Domain Dedication along with this software. If not, see <http://creativecommons.org/publicdomain/zero/1.0/>.
// Some important notes about the polyfill below:
// Native TextEncoder/TextDecoder implementation is overwritten
// String.prototype.codePointAt polyfill not included, as well as String.fromCodePoint
// TextEncoder.prototype.encode returns a regular array instead of Uint8Array
// No options (fatal of the TextDecoder constructor and stream of the TextDecoder.prototype.decode method) are supported.
// TextDecoder.prototype.decode does not valid byte sequences
// This is a demonstrative implementation not intended to have the best performance
// http://encoding.spec.whatwg.org/#textencoder
// http://encoding.spec.whatwg.org/#textencoder
function TextEncoder() {
}
TextEncoder.prototype.encode = function (string) {
var octets = [];
var length = string.length;
var i = 0;
while (i < length) {
var codePoint = string.codePointAt(i);
var c = 0;
var bits = 0;
if (codePoint <= 0x0000007F) {
c = 0;
bits = 0x00;
} else if (codePoint <= 0x000007FF) {
c = 6;
bits = 0xC0;
} else if (codePoint <= 0x0000FFFF) {
c = 12;
bits = 0xE0;
} else if (codePoint <= 0x001FFFFF) {
c = 18;
bits = 0xF0;
}
octets.push(bits | (codePoint >> c));
c -= 6;
while (c >= 0) {
octets.push(0x80 | ((codePoint >> c) & 0x3F));
c -= 6;
}
i += codePoint >= 0x10000 ? 2 : 1;
}
return octets;
};
function TextDecoder() {
}
TextDecoder.prototype.decode = function (octets) {
var string = "";
var i = 0;
while (i < octets.length) {
var octet = octets[i];
var bytesNeeded = 0;
var codePoint = 0;
if (octet <= 0x7F) {
bytesNeeded = 0;
codePoint = octet & 0xFF;
} else if (octet <= 0xDF) {
bytesNeeded = 1;
codePoint = octet & 0x1F;
} else if (octet <= 0xEF) {
bytesNeeded = 2;
codePoint = octet & 0x0F;
} else if (octet <= 0xF4) {
bytesNeeded = 3;
codePoint = octet & 0x07;
}
if (octets.length - i - bytesNeeded > 0) {
var k = 0;
while (k < bytesNeeded) {
octet = octets[i + k + 1];
codePoint = (codePoint << 6) | (octet & 0x3F);
k += 1;
}
} else {
codePoint = 0xFFFD;
bytesNeeded = octets.length - i;
}
string += String.fromCodePoint(codePoint);
i += bytesNeeded + 1;
}
return string
};
@agnivade

This comment has been minimized.

Copy link

@agnivade agnivade commented Aug 31, 2018

Hi, can you please attach a license to this code ? We would like to use it in the Go project here - https://go-review.googlesource.com/c/go/+/131718.

@anonyco

This comment has been minimized.

Copy link

@anonyco anonyco commented Apr 22, 2020

These are not polyfills. They do not provide standards-compliant behavior and overwrite the native implementation when TextEncoder/TextDecoder are natively supported.

@Yaffle

This comment has been minimized.

Copy link
Owner Author

@Yaffle Yaffle commented Apr 22, 2020

@anonyco, The work to not overwrite the native is "left to the readers as an exercise".
The behavior is almost standard-compliant. The decoding/encoding is good enough, the options are not supported (but old Firefox does not support them as well), the result is an array, not typed array/view.

@anonyco

This comment has been minimized.

Copy link

@anonyco anonyco commented Apr 22, 2020

Please at least include the following disclaimer in a comment at the top of the code:

  • Native TextEncoder/TextDecoder implementation is overwritten
  • String.prototype.codePointAt polyfill not included
  • TextEncoder.prototype.encode returns a regular array instead of Uint8Array
  • This is a demonstrative implementation not intended to have the best performance

Nobody likes disclaimers about their code, but disclaimers help prevent code from being misused and people from being misled.

Please also switch from the Unlicense to CC0: https://creativecommons.org/2011/04/15/plaintext-versions-of-creative-commons-licenses-and-cc0/. Both licenses have the same intentions, but there is a big difference: CC0 is battle-tested, thorough, and recognized everywhere, whereas the Unlicense is rather vague, untested, and not universally recognized.

Thank you.

@Yaffle

This comment has been minimized.

Copy link
Owner Author

@Yaffle Yaffle commented Apr 22, 2020

@anonyco, OK, thanks

@anonyco

This comment has been minimized.

Copy link

@anonyco anonyco commented Apr 22, 2020

Thank you so much. You are doing a great service for people inexperienced with JavaScript who are trying to feel their way around. I just want to suggest not calling them "issues." That word has negative connotations. One man's "issue" can be another man's treasure, so please do something like

    // Some important notes about the polyfill below:
    // N.B. that
    // Ponies to watch out for:

Thank you.

@Yaffle

This comment has been minimized.

Copy link
Owner Author

@Yaffle Yaffle commented Apr 22, 2020

@anonyco, thank you

@preludium975

This comment has been minimized.

Copy link

@preludium975 preludium975 commented Nov 10, 2020

huge thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.