The conference was a great experience but there didn't seem to be any groundbreaking work, mostly new tricks or combination of methods. Probabilistic modelling using neural networks and GAN's seem very popular and applying neural networks to new datasets/areas is still enough to get a NIPS poster. Some of the orals were good but personally I think most of them were only poster level while many posters were oral talk level though people more intelligent than me made that choice which probably means the selection process is quite random.
Differentiable Neural Computers (Memory Networks) will probably play a big role in the more complex tasks e.g. dialog systems and reasoning where the model needs to keep an internal representation of an entity and it's properties (also see the EntityNet from LeCun). As Graves explained it is conceptually a nice idea to separate the computation and the memory and it gives the model the capability to le