Skip to content

Instantly share code, notes, and snippets.

@nicStuff

nicStuff/mifcs.rb

Last active Mar 23, 2020
Embed
What would you like to do?
Simple ruby script that renames all .jpg/.mov/.nef files in the given folder to a name containing the exif creation timestamp (for jpg/nef) and modification time (for mov). Scales images using imagemagick in a folder
##########################################################################################################
##########################################################################################################
##########################################################################################################
###### Mass Image Filename Corrector and Scaler
##########################################################################################################
################################################ [rename] Renaming
######
###### Renames jpeg/nef/mov/mp4 files from xxxx.jpg, to a timestamp filename, like
###### 2015.05.28_15.33.22.jpg, using the date in exif field Exif.Photo.DateTimeOriginal for
###### images, and using file modification time for .mov/.mp4
######
###### * Video's wrong timestamp
######
###### File modification times for videos can be wrong, for instance if they are copy-pasted from a
###### device to the computer, renames are logged in a text file so possible mistakes can be then
###### reversed. Furthermore, skips renaming if there are 6 or more digits in the filename since
###### these could be a date in the minima format ddmmyy.
######
###### To get the right timestamps you can ls -lah the videos in a folder with a path like the following
###### `/run/user/1000/gvfs/mtp:host=%5Busb%3A003%2C005%5D/`: access it with the current user, not root
######
###### * How to recovery videos dates
######
###### - if the camera uses the same increment to assign videos/photos the filename you can
###### get the video date by looking at the nearby images' exif data
######
################################################ [scale] Scaling
######
###### Scales jpg and png images to the specified length of the long side. Features:
######
###### * doesn't upscale: if the specified long side is greater than the current one
###### it doesn't do anything;
###### * preserves aspect ratio and rotation;
###### * processes in parallel: keeps running more or less n threads concurrently until
###### the end of the process, where n is the number of logical cores of the machine.
######
################################################ [ronef] Removing Orphan NEFs
######
###### Soft-deletes NEF files which don't have anymore a jpg file: useful for when, filtering images,
###### you find images that you don't like and delete the jpg, but not the nef. The orphan NEFs will
###### be moved to a subfolder for further inspection or deletion.
######
##########################################################################################################
###### => Gist: https://gist.github.com/nicStuff/4feea847f328996e8f3c
##########################################################################################################
##########################################################################################################
##########################################################################################################
###### => Requires the following linux programs:
######### => exiv2 (for renaming with the exif timestamp)
######### => imagemagick (with jpg support) (for jpg scaling)
###### => Requires the following Ruby libraries
######### => fileutils (for renaming the file)
######### => etc (for getting the number of logical cores)
##########################################################################################################
##########################################################################################################
##########################################################################################################
###### Misc low level logic
# processes the given folder, recursive if specified
def process_folder (target_folder, recursive, level = 0, &behaviour)
tabs = ''; level.times { tabs << "\t" }
printf tabs + '*** processing folder ' + target_folder; puts
folders = []
# getting list of files and sorting it by the number in the filename
sorted_files = []
Dir.foreach(target_folder) { |f| sorted_files << f }
sorted_files.sort.each do |f|
full_file_path = target_folder + '/' + f
# checks
next if (f == '.' || f == '..')
if (Dir.exist?(full_file_path))
folders << full_file_path
next
end
next if (f =~ /.*\.(jpe?g|nef|mov|mp4)/i).nil?
# calling logic on each file
yield f, full_file_path, target_folder
end
return if !recursive
# calling for each subdir
folders.each do |f|
process_folder f + '/', recursive, level + 1, &behaviour
end
end
def rename_file source, target, ctime, logger, dry = false
ctime ||= File.new(source).mtime
# 5.1. skipping if target file exists
if File.exists?(target)
logger.info "target file #{target} exists, skipping"
return
end
# 5.2. renaming
logger.info "#{dry ? '[dry] ' : ''}#{source} ----> #{target}"
File.rename(source, target) unless dry
# 6. correcting access time (for .mov we don't want that at every launch of the tool the filename changes)
File.utime(ctime, ctime, target) unless dry
end
def config_enabled name
ARGV.each {|arg| return true if '--' + name == arg}
false
end
###### Init and configuration
require 'fileutils'
if (ARGV.length < 1 || ARGV[0] == '')
puts "specify command <rename [folder path]|scale [target_width] [folder path]|ronef [folder path]> [--dry] [--recursive]"
exit
end
directory = nil
recursive = config_enabled 'recursive'
dry = config_enabled 'dry'
command = ARGV[0]
######## Processing
puts "************\n******\tMass Image Filename Corrector and Scaler\n************\n#{dry ? '[dry] ' : ''}applying command #{command} #{recursive ? 'recursive' : ''}\n"
case command
when "rename"
if ARGV.length < 2 || !Dir.exist?(ARGV[1])
puts "Invalid directory specified"
exit 1
end
require 'logger'
directory = ARGV[1]
# tracking dates used in filenames: this way we add a suffix to images taken in the same second (happens e.g. with continuous shooting)
puts "\tLogging renames in file #{log_fpath}"
suffix_indexes = {
# filename => suffix index
}
process_folder directory, recursive do |filename, full_file_path, target_folder|
log_fpath = File.join(target_folder, 'mifcs-renames.log')
logger = Logger.new(log_fpath)
# skipping if the filename matches the pattern
next if filename =~ /\d{4}\.\d{2}\.\d{2}_\d{2}\.\d{2}\.\d{2}.*/
if filename.gsub(/[^0-9]/, '').length > 5 # considering the minimal date format ddmmyy, which has 6 numbers
logger.info "Skipping filename #{filename} which has too many numbers in it, which could be a date"
next
end
begin
# getting modification time (default, works for .mov)
ctime = File.new(full_file_path).mtime
original_date = ctime.strftime("%Y.%m.%d_%H.%M.%S")
# 3. nef or jpg: getting exif creation time
unless (filename =~ /.*\.(jpg|nef)/i).nil?
# 3.1. getting exif data
exif_data = `exiv2 -pt "#{full_file_path}"`
# 3.2. Getting original date
original_date = exif_data.match(/.*Exif\.Photo\.DateTimeOriginal.*?(\d{4}:\d{2}:\d{2}\s+\d{2}:\d{2}:\d{2}).*/m)[1].gsub(/:/, ".").gsub(/ /, "_")
end
# 4. Getting filename
file_extension = File.extname(full_file_path)
dest_file_path = target_folder + '/' + original_date + file_extension
# 4. computing target filename
# 4.1. suffix handling
# if there is a suffix index it means that this filename has already been find earlier: we increment the suffix and change this filename
fpath_suffix = ''
if suffix_indexes.key?(dest_file_path)
fpath_suffix = suffix_indexes[dest_file_path]
suffix_indexes[dest_file_path] = fpath_suffix + 1
dest_file_path.sub!(file_extension, '-' + fpath_suffix.to_s + file_extension)
# otherwise, this is the first time we find the file, we se the index
else
suffix_indexes[dest_file_path] = 1
end
# 5. renaming
rename_file full_file_path, dest_file_path, ctime, logger, dry
rescue => error
puts "[WARN] Failed to process #{filename} in #{target_folder} " + error.to_s
end
end
when "dummy"
puts ARGV.inspect
puts (dry)
puts (recursive)
when "scale"
if ARGV.length < 3
puts "Specify target long side width and file path"
exit 1
end
target_lside = ARGV[1].to_i
directory = ARGV[2]
require 'etc'
cores = Etc.nprocessors
tg = ThreadGroup.new
puts "concurrently (#{cores} threads more or less at a time) scaling to #{target_lside} long side #{dry ? ' dry' : ''}"
process_folder directory, recursive do |filename, full_file_path|
# If thread group is full, wait a bit
sleep 0.2 while (tg.list.size >= cores)
tg.add (Thread.new do
# Processing only images
next if (filename =~ /.*\.(jpe?g|png)/i).nil?
width = `convert "#{full_file_path}" -format "%[fx:w]" info:`.to_i
height = `convert "#{full_file_path}" -format "%[fx:h]" info:`.to_i
long_side = (width > height ? width : height)
short_side = (width < height ? width : height)
threadname_for_logs = "[Thread#{Thread.current.object_id}] "
## Checks
# Avoiding upscale or processing if already at desired size
hysteresis = 10
if ((target_lside - hysteresis) >= long_side) || ((target_lside + hysteresis) >= long_side)
puts "[noop] #{threadname_for_logs}#{full_file_path} #{long_side} +/- 10\ <= #{target_lside}"
next
end
## Processing
# assessing length of short side of target image
target_sside = ((target_lside.to_f / long_side.to_f) * short_side.to_f).round
# getting number of pixels
npixels_original = width * height
npixels_resized = target_lside * target_sside
puts "#{dry ? '[dry] ' : ''}#{threadname_for_logs}#{full_file_path} #{long_side}x#{short_side} ---> #{target_lside}x#{target_sside}"
## scale commands (http://www.imagemagick.org/Usage/resize/#percent)
# convert orig.jpg -resize 12000000@\> res_orig.jpg
# => means: resize orig.jpg to a number of pixels of 12 millions (keeps aspect ratio) only if the image has more than 12 million pixels (i.e. doesn't enlarge), and put the result in res_orig.jpg
result_file_path = full_file_path #+ '-rsz'
size_before = File.size full_file_path
# http://blog.honeybadger.io/capturing-stdout-stderr-from-shell-commands-via-ruby/: for having stderr we should use another module, and we won't
`convert "#{full_file_path}" -resize #{npixels_resized}@\\> "#{result_file_path}"` unless dry
size_after = File.size result_file_path
shrink_percent = - (((size_before - size_after) * 100) / size_before).round(2)
puts "* #{threadname_for_logs}#{size_before} byte -> #{size_after} byte (#{shrink_percent}%)" unless dry
end)
end
when 'ronef' # Remove Orhpan NEFs
if ARGV.length < 2 || !Dir.exist?(ARGV[1])
puts "Invalid directory specified"
exit 1
end
require 'io/console'
directory = ARGV[1]
SOFT_DELETE_FOLDER_NAME = '.mifcs-deleted-ronef'
folder_fn_map = {
# folder_name => {
# <filename_without_extension> => {
# :jpg_fp => <jpg full file path>,
# :nef_fp => <nef full file path>
# },
# <filename_without_extension> => {
# :jpg_fp => <jpg full file path>,
# :nef_fp => <nef full file path>
# },
# [...]
# }
}
# 1. get all files, mapped to directory, then filename
puts "# RoNEF-s - 1 - Mapping all files to folders"
process_folder directory, recursive do |filename, full_file_path|
next if (filename =~ /.*\.(jpe?g|nef)$/i).nil?
next if filename == SOFT_DELETE_FOLDER_NAME
fname_key = (filename =~ /.*\.jpe?g$/i ? :jpg_fp : :nef_fp)
extless_fname = filename.gsub(/(.+)\..+/, '\1')
folder_path = File.dirname(full_file_path)
# getting/preparing objects
folder_fn_map[folder_path] = {} unless folder_fn_map.key? folder_path
folder_fn_map[folder_path][extless_fname] = {} unless folder_fn_map[folder_path].key? extless_fname
folder_fn_map[folder_path][extless_fname][fname_key] = full_file_path
end
# 2. getting the list of nefs to delete, mapped by folder
puts "# RoNEF-s - 2 - Deleting"
folder_fn_map.each_pair do |folder_path, extfn_jn|
puts "\tDetecting orphan NEFs in folder #{folder_path}"
jpgs_found = false
nefs_to_delete = []
# collecting the NEFs to delete
extfn_jn.each_pair do |extless_fname, content|
puts "\t\tprocessing #{extless_fname}, files #{content}"
jpgs_found = true if content.key? :jpg_fp
if content.key?(:nef_fp) && !content.key?(:jpg_fp)
nefs_to_delete << content[:nef_fp]
end
end
# [checks] only nefs found: we assume that those are the only images in the folder, we tell it and remove the nefs from the images to delete
unless jpgs_found
puts "\t\t* NOOP: Only NEFs found, skipping"
next
end
# [checks] no nefs found
if nefs_to_delete.empty?
puts "\t\t* NOOP: No NEFs to delete, skipping"
next
end
# shows NEFs to delete, asks for confirmation
puts "\t\t* Found #{nefs_to_delete.count} NEFs to delete:\t#{nefs_to_delete}"
# Soft-deleting
soft_delete_folder = File.join(folder_path, SOFT_DELETE_FOLDER_NAME)
FileUtils.mkdir_p soft_delete_folder unless dry
unless dry
nefs_to_delete.each do |nef_td_path|
FileUtils.mv(nef_td_path, File.join(soft_delete_folder, File.basename(nef_td_path)))
end
end
puts "\t\t\t#{dry ? ' [DRY] ' : ''}OK, moved to #{soft_delete_folder}"
end
else
puts "Invalid command #{command}"
exit
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment