Skip to content

Instantly share code, notes, and snippets.

@patch
Last active December 28, 2015 01:19
Show Gist options
  • Save patch/7419410 to your computer and use it in GitHub Desktop.
Save patch/7419410 to your computer and use it in GitHub Desktop.

Comment on Mojolicious Unicode Normalization Plugin Released by chromatic at Modern Perl Programming (since the local comments are broken):

Nice! In the past I was looking at writing a PerlIO layer for implicit Unicode normalization on I/O. ikegami has worked on writing PerlIO::via::Normalize but it's been on GitHub for two years without a CPAN release.

Although I definitely have a need for normalization forms other than NFC, I use NFC for general input normalization and then explicitly normalize to another form on the rarer cases when I need them, such as some text indexing and information retrieval (NFKC) as well as some regexes (NFD).

Have you thought about the option to normalize on output? There are two reasons that this can be desired: 1) applications with public-facing APIs should always output data in NFC, but some people, such as tchrist, want to normalize to NFD on input; and 2) even when normalizing on input, denormalized text can be created within the application.

I recently spoke at the Internationalization & Unicode Conference and folks were very impressed with the Unicode functionality available in both the Perl language and core modules. Most other programming languages don't have a standard core way to perform Unicode normalization.

Thanks for the exciting new module! I imagine other frameworks and projects will take notice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment