12-1_CNN.ipynb

ExponentialMovingAverage.md

ema.variables_to_restore dont rely on ema.apply and is not part of graph. it solely output a dict

variables_to_restore(moving_avg_variables=None) moving_avg_variables defaults to variables.moving_average_variables() + variables.trainable_variables()

tensorflow/tensorflow#11839 (comment)

This behavior is confusing but it is not a bug. When calling variables_to_restore, you are supposed to indicate what variables have moving averages through its moving_avg_variables parameter. ExponentialMovingAverage doesn't actually use its own stored averages to determine what variables have moving averages, but instead relies on moving_avg_variables. moving_avg_variables defaults to variables.moving_average_variables() + variables.trainable_variables(), which is why Variable c is included.

I believe the reason variables_to_restore acts this way is that it is typically called during evaluation, while the moving averages were set during training. During evaluation, no moving averages are set, so variables_to_restore cannot rely on the ExponentialMovingAverage's stored moving averages, which is why it uses moving_avg_variables instead.

@sguada, I think we should change the description to something like "If a variable is in moving_avg_variables, use its moving average variable name as the restore name; otherwise, use the variable name." What do you think?

a = tf.Variable(tf.constant(1.0), name='a')
b = tf.Variable(tf.constant(3.0), name='b')
c = tf.Variable(tf.constant(5.0), name='c')
ema = tf.train.ExponentialMovingAverage(decay=0.9999)
# ema.apply([a, b])

# print(tf.get_default_graph().get_all_collection_keys())
# print(tf.get_collection('moving_average_variables'))
# print(tf.global_variables())
variables_to_restore = ema.variables_to_restore([a, b])
d = tf.Variable(tf.constant(7.0), name='d')
for mv, var in variables_to_restore.items():
    print(mv, ' : ', var)

print(tf.get_collection(tf.GraphKeys.MOVING_AVERAGE_VARIABLES))

c  :  <tf.Variable 'c:0' shape=() dtype=float32_ref>
b/ExponentialMovingAverage  :  <tf.Variable 'b:0' shape=() dtype=float32_ref>
a/ExponentialMovingAverage  :  <tf.Variable 'a:0' shape=() dtype=float32_ref>
[]

ema.average_name(var) is similiar with ma.variables_to_restore([var]),
but ema.average(var) return None if var is not applied with ema

x = tf.get_variable(name='var', initializer=1.0)
ema = tf.train.ExponentialMovingAverage(0.9)
print(ema.average_name(x))
print(ema.average(x))

var/ExponentialMovingAverage
None