Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Residual LSTM in Keras
def make_residual_lstm_layers(input, rnn_depth, rnn_dropout):
"""
The intermediate LSTM layers return sequences, while the last returns a single element.
The input is also a sequence. In order to match the shape of input and output of the LSTM
to sum them we can do it only for all layers but the last.
"""
for i in range(rnn_depth):
return_sequences = i < rnn_depth - 1
x_rnn = LSTM(rnn_width, dropout_W=rnn_dropout, dropout_U=rnn_dropout, return_sequences=return_sequences)(input)
if return_sequences:
# residual block
x = merge([x, x_rnn], mode='sum')
else:
# last layer does not return sequences and cannot be residual
x = x_rnn
return x
@PrithivirajDamodaran

This comment has been minimized.

Copy link

@PrithivirajDamodaran PrithivirajDamodaran commented Nov 27, 2018

Can you help me understand how this is a residual block ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment