Skip to content

Instantly share code, notes, and snippets.

@rejuvyesh
rejuvyesh / lru.py
Created April 13, 2023 02:45 — forked from Ryu1845/lru.py
Linear Recurrent Unit (LRU) from the paper ["Resurrecting Recurrent Neural Networks for Long Sequences"](https://arxiv.org/abs/2303.06349)
"""
Simplified Implementation of the Linear Recurrent Unit
------------------------------------------------------
We present here a simplified JAX implementation of the Linear Recurrent Unit (LRU).
The state of the LRU is driven by the input $(u_k)_{k=1}^L$ of sequence length $L$
according to the following formula (and efficiently parallelized using an associative scan):
$x_{k} = \Lambda x_{k-1} +\exp(\gamma^{\log})\odot (B u_{k})$,
and the output is computed at each timestamp $k$ as follows: $y_k = C x_k + D u_k$.
In our code, $B,C$ follow Glorot initialization, with $B$ scaled additionally by a factor 2
to account for halving the state variance by taking the real part of the output projection.
@rejuvyesh
rejuvyesh / notebook.ipynb
Created July 19, 2017 00:52 — forked from eamartin/notebook.ipynb
Understanding & Visualizing Self-Normalizing Neural Networks
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@rejuvyesh
rejuvyesh / gist:deaf1f2176f255ac6a4b
Created December 19, 2015 05:09 — forked from debasishg/gist:b4df1648d3f1776abdff
another attempt to organize my ML readings ..
  1. Feature Learning
  1. Deep Learning
# convert multipage pdf to single page tiff
gs -q -dNOPAUSE -dBATCH -sDEVICE=tiffg4 -sOutputFile=%04d.tif source.pdf -c qui
# or use -sDEVICE=pgmraw to convert to pgm
# unpaper, rotate the logical page 90 degrees, each logical page contained two scanned physical pages, so we use --layout double (for input) and --output-pages 2 since we want to split these two pages.
unpaper -v --deskew-scan-deviation 3.0 --border-align top --deskew-scan-range 15 --no-grayfilter --no-blurfilter --no-noisefilter --overwrite --pre-rotate 90 --border-scan-step 4 --layout double --output-pages 2 %04d.pgm.pbm unpaper%04d.pbm
# trim the pages and convert the to single-page pdfs
find . -name 'unpaper*' | xargs -i -n1 -P6 convert -trim +repage {} {}.pdf
# unregister broken GHC packages. Run this a few times to resolve dependency rot in installed packages.
# ghc-pkg-clean -f cabal/dev/packages*.conf also works.
function ghc-pkg-clean() {
for p in `ghc-pkg check $* 2>&1 | grep problems | awk '{print $6}' | sed -e 's/:$//'`
do
echo unregistering $p; ghc-pkg $* unregister $p
done
}
# remove all installed GHC/cabal packages, leaving ~/.cabal binaries and docs in place.
# unregister broken GHC packages. Run this a few times to resolve dependency rot in installed packages.
# ghc-pkg-clean -f cabal/dev/packages*.conf also works.
function ghc-pkg-clean() {
for p in `ghc-pkg check $* 2>&1 | grep problems | awk '{print $6}' | sed -e 's/:$//'`
do
echo unregistering $p; ghc-pkg $* unregister $p
done
}
# remove all installed GHC/cabal packages, leaving ~/.cabal binaries and docs in place.