Gramian matrix

gramian_matrix.py

import numpy as np
from scipy.spatial.distance import cosine

def constant(x):
    return x

# We compute the gramian matrix like so
def gram_matrix(x1,x2) :
    assert( x1.shape == x2.shape )
    return np.matmul(np.transpose(x1),x2) / ( x1.shape[0] * x1.shape[1] )

# But this doesn't give us a loss.  
# To compute the loss we check the gramian against the gramian of X1 against itself
def gram_matrix_loss(x1,x2) :
    x1 = constant(x1) # where constant is a wrapper around x1 to stop gradient descent
    difference = ( gram_matrix(x1,x2) - gram_matrix(x1,x1) )
    return np.power(difference,2).mean()  # return the sum or mean of the square of the difference

a = (( np.array(range(12),dtype=np.float32).reshape([1,12]) - 6.0 ) / 6.0).T

b = a*1.1
c = 1 - a
d = (np.ones([1,12],dtype=np.float32)/2.0).T

for name,var in zip("a,b,c,d".split(","),[a,b,c,d]):
    print "gram_matrix(a,"+name+") = ",gram_matrix(a,var)
    print "gram_matrix_loss(a,"+name+") = ",gram_matrix_loss(a,var)
    print "cosine_distance(a,"+name+") = ",cosine(a,var)
    print ""

Gram Loss (a.k.a Gramian Matrix, Gram Matrix) is a method to compute the linear independence of two matrices. This has an application in computing the difference between the embedding layers of two networks. A specific application would be that you have one network N1 that already does a good job detecting fur and you want to use it as the critic of network N2 that you are training to generate realistic fur. You suspect the fur detection of N1 is buried somewhere in around layer X in network N1 and N2. Using gram loss to compare X1 and X2 will help train N2 to make realistic fur, if fur is encoded in that layer.