Skip to content

Instantly share code, notes, and snippets.

@iskandr
Created April 17, 2017 20:24
Show Gist options
  • Save iskandr/a874e4cf358697037d14a17020304535 to your computer and use it in GitHub Desktop.
Save iskandr/a874e4cf358697037d14a17020304535 to your computer and use it in GitHub Desktop.
Since Keras 2.0 removed the Highway Network layer, here's my attempt at implementing something equivalent using the functional API
import keras.backend as K
from keras.layers import Dense, Activation, Multiply, Add, Lambda
import keras.initializers
def highway_layers(value, n_layers, activation="tanh", gate_bias=-3):
dim = K.int_shape(value)[-1]
gate_bias_initializer = keras.initializers.Constant(gate_bias)
for i in range(n_layers):
gate = Dense(units=dim, bias_initializer=gate_bias_initializer)(value)
gate = Activation("sigmoid")(gate)
negated_gate = Lambda(
lambda x: 1.0 - x,
output_shape=(dim,))(gate)
transformed = Dense(units=dim)(value)
transformed = Activation(activation)(value)
transformed_gated = Multiply()([gate, transformed])
identity_gated = Multiply()([negated_gate, value])
value = Add()([transformed_gated, identity_gated])
return value
@deepanwayx
Copy link

Thanks for the implementation. What is the utility of the "transformed" variable in line 14? As right in the next line you are defining a new expression for "transformed" which only uses the "value" variable.

@danFromTelAviv
Copy link

I believe line 15 should be : transformed = Activation(activation)(transformed)

@bipin-a
Copy link

bipin-a commented Nov 25, 2020

Hey, how could you add this layer into a CNN using Sequential in Keras?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment