Fix those annoying "invalid byte sequence in UTF-8" errors when parsing CSVs, etc. Just strips out undef/invalid chars. Use with caution.

fix_utf8.rb

#Fix those annoying "invalid byte sequence in UTF-8" errors when parsing CSVs, etc.
#prints to STDOUT where you can redirect to a new file, etc.
#NOTE: this just strips out invalid/undef characters which may be a Very Bad Thing™ - YMMV
#suggestions welcome...

#also fix annoying newlin chars created by excel... replace \r with \r (weird but works)

input_file = ARGV[0] or raise "No input file specified"
puts IO.read(input_file).encode('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '').gsub(/\r/,"\r")