Skip to content

Instantly share code, notes, and snippets.

@wiseodd
Created March 13, 2018 19:36
Show Gist options
  • Save wiseodd/1c9f5006310f5ee03bd4682b4c03020a to your computer and use it in GitHub Desktop.
Save wiseodd/1c9f5006310f5ee03bd4682b4c03020a to your computer and use it in GitHub Desktop.
Natural Gradient Descent for Logistic Regression
import numpy as np
from sklearn.utils import shuffle
# Data comes from y = f(x) = [2, 3].x + [5, 7]
X0 = np.random.randn(100, 2) - 1
X1 = np.random.randn(100, 2) + 1
X = np.vstack([X0, X1])
t = np.vstack([np.zeros([100, 1]), np.ones([100, 1])])
X, t = shuffle(X, t)
X_train, X_test = X[:150], X[:50]
t_train, t_test = t[:150], t[:50]
# Model
W = np.random.randn(2, 1) * 0.01
def sigm(x):
return 1/(1+np.exp(-x))
def NLL(y, t):
return -np.mean(t*np.log(y) + (1-t)*np.log(1-y))
alpha = 0.1
# Training
for it in range(5):
# Forward
z = X_train @ W
y = sigm(z)
loss = NLL(y, t_train)
# Loss
print(f'Loss: {loss:.3f}')
m = y.shape[0]
dy = (y-t_train)/(m * (y - y*y))
dz = sigm(z)*(1-sigm(z))
dW = X_train.T @ (dz * dy)
grad_loglik_z = (t_train-y)/(y - y*y) * dz
grad_loglik_W = grad_loglik_z * X_train
F = np.cov(grad_loglik_W.T)
# Step
W = W - alpha * np.linalg.inv(F) @ dW
# W = W - alpha * dW
# print(W)
y = sigm(X_test @ W).ravel()
acc = np.mean((y >= 0.5) == t_test.ravel())
print(f'Accuracy: {acc:.3f}')
@DrSleep
Copy link

DrSleep commented Dec 10, 2019

looks like this

X_train, X_test = X[:150], X[:50]
t_train, t_test = t[:150], t[:50]

must be this

X_train, X_test = X[:150], X[150:] # or X[:150], X[-50:]
t_train, t_test = t[:150], t[150:] # or t[:150], t[-50:] 

as otherwise train and test subsets overlap

@chongyi-zheng
Copy link

chongyi-zheng commented Dec 15, 2019

In your blog "Natural Gradient Descent", there exists a sentance "Covariance and precision matrix are different to each other (up to special condition, e.g. identity matrix), even though it induces the same Gaussian. ". How the same Gaussian has the differenct Covaiance matrix?

I think the author means the covariance and precision matrix are two different ways to parameterize the same Gaussian distribution. Same distribution space, but different parameter spaces. I guess there should be a transformation between the two different parameter spaces.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment