Skip to content

Instantly share code, notes, and snippets.

@ferventcoder
Last active August 29, 2015 14:23
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save ferventcoder/ffd08eed77c8d9039b77 to your computer and use it in GitHub Desktop.
Save ferventcoder/ffd08eed77c8d9039b77 to your computer and use it in GitHub Desktop.
I was attempting to enforce UTF-8, this only contains a few encodings but could do more
def fix_encodings(files)
files.each do |file|
next if File.directory?(file)
[Encoding::UTF_8, Encoding::UTF_16LE, Encoding::UTF_16, Encoding::ISO_8859_1, Encoding::US_ASCII, Encoding::default_external].each do |encoding|
contents = File.read(file, :mode =>'rb', :encoding => encoding)
next unless contents.valid_encoding?
break if contents.valid_encoding? && encoding == Encoding::UTF_8
if contents.valid_encoding?
#if has_valid_utf8?(file)
begin
puts "Converting #{file} to from #{encoding} to UTF-8 encoding"
File.open(file,'w:UTF-8') do |f|
f.write contents.encode('UTF-8') #.gsub(/\r\n?/u,"\n")
end
break
rescue => e
puts "Whoops, not able to convert #{file}. Try it manually"
end
end
end
end
end
@ferventcoder
Copy link
Author

@ferventcoder
Copy link
Author

It sounds like I could have just used iconv for similar haha - http://stackoverflow.com/questions/1033104/how-do-i-convert-a-ucs2-string-into-utf8

@ferventcoder
Copy link
Author

This also may not even work correctly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment