Skip to content

Instantly share code, notes, and snippets.

@raytroop
Last active October 13, 2018 13:33
Show Gist options
  • Save raytroop/20f3c5e2d741fffb654b04049ef84d88 to your computer and use it in GitHub Desktop.
Save raytroop/20f3c5e2d741fffb654b04049ef84d88 to your computer and use it in GitHub Desktop.
Display the source blob
Display the rendered blob
Raw
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

ema.variables_to_restore dont rely on ema.apply and is not part of graph. it solely output a dict

variables_to_restore(moving_avg_variables=None) moving_avg_variables defaults to variables.moving_average_variables() + variables.trainable_variables()


tensorflow/tensorflow#11839 (comment)

This behavior is confusing but it is not a bug. When calling variables_to_restore, you are supposed to indicate what variables have moving averages through its moving_avg_variables parameter. ExponentialMovingAverage doesn't actually use its own stored averages to determine what variables have moving averages, but instead relies on moving_avg_variables. moving_avg_variables defaults to variables.moving_average_variables() + variables.trainable_variables(), which is why Variable c is included.

I believe the reason variables_to_restore acts this way is that it is typically called during evaluation, while the moving averages were set during training. During evaluation, no moving averages are set, so variables_to_restore cannot rely on the ExponentialMovingAverage's stored moving averages, which is why it uses moving_avg_variables instead.

@sguada, I think we should change the description to something like "If a variable is in moving_avg_variables, use its moving average variable name as the restore name; otherwise, use the variable name." What do you think?



a = tf.Variable(tf.constant(1.0), name='a')
b = tf.Variable(tf.constant(3.0), name='b')
c = tf.Variable(tf.constant(5.0), name='c')
ema = tf.train.ExponentialMovingAverage(decay=0.9999)
# ema.apply([a, b])

# print(tf.get_default_graph().get_all_collection_keys())
# print(tf.get_collection('moving_average_variables'))
# print(tf.global_variables())
variables_to_restore = ema.variables_to_restore([a, b])
d = tf.Variable(tf.constant(7.0), name='d')
for mv, var in variables_to_restore.items():
    print(mv, ' : ', var)

print(tf.get_collection(tf.GraphKeys.MOVING_AVERAGE_VARIABLES))
c  :  <tf.Variable 'c:0' shape=() dtype=float32_ref>
b/ExponentialMovingAverage  :  <tf.Variable 'b:0' shape=() dtype=float32_ref>
a/ExponentialMovingAverage  :  <tf.Variable 'a:0' shape=() dtype=float32_ref>
[]
@raytroop
Copy link
Author

ema.average_name(var) is similiar with ma.variables_to_restore([var]),
but ema.average(var) return None if var is not applied with ema

x = tf.get_variable(name='var', initializer=1.0)
ema = tf.train.ExponentialMovingAverage(0.9)
print(ema.average_name(x))
print(ema.average(x))

var/ExponentialMovingAverage
None

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment