RNN (Recurrent Neural nets) are used to model sequences. Unlike the usual feedforward nets which are stateless in terms on inputs, RNNs have memory. In particular, its inputs are the output of previous step and also new observation in current step.
The basic RNN are notoriously hard to train. LSTM (Long short term memory) networks are an adaptation of RNNs which make training long sequence dependencies feasible.
In addition to the deep learning book’s chapters here are some of the nice blog post which explains RNN:
- Andrej Karpathy’s blog on RNN’s effectiveness.
- Colah’s blog on how computations happen inside an LSTM networks.
- Training Recurrent Neural Networks, Ilya Sutskever, PhD Thesis, 2012.
- Deep Learning Book’s chapter on RNN
Manohar Kuse's Cyber 