Metadata-Version: 2.1
Name: lmu
Version: 0.2.0
Summary: Legendre Memory Units
Home-page: https://www.appliedbrainresearch.com/lmu
Author: Applied Brain Research
Author-email: info@appliedbrainresearch.com
License: Free for non-commercial use
Description: Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks
        ----------------------------------------------------------------------------------
        
        `Paper <https://papers.nips.cc/paper/9689-legendre-memory-units-continuous-time-representation-in-recurrent-neural-networks.pdf>`_
        
        NengoLMU is a python software library containing various implementations of the Legendre Memory Unit (LMU). The LMU is a novel memory cell for recurrent neural networks that dynamically maintains information across long windows of time using relatively few resources. It has been shown to perform as well as standard LSTM or other RNN-based models in a variety of tasks, generally with fewer internal parameters (see `this paper <https://papers.nips.cc/paper/9689-legendre-memory-units-continuous-time-representation-in-recurrent-neural-networks.pdf>`_ for more details). For the Permuted Sequential MNIST (psMNIST) task in particular, it has been demonstrated to outperform the current state-of-the-art results. See the note below for instructions on how to get access to this model.
        
        The LMU is mathematically derived to orthogonalize its continuous-time history – doing so by solving *d* coupled ordinary differential equations (ODEs), whose phase space linearly maps onto sliding windows of time via the Legendre polynomials up to degree *d* − 1 (the example for *d* = 12 is shown below).
        
        .. image:: https://i.imgur.com/Uvl6tj5.png
           :target: https://i.imgur.com/Uvl6tj5.png
           :alt: Legendre polynomials
        
        A single LMU cell expresses the following computational graph, which takes in an input signal, **x**, and couples a optimal linear memory, **m**, with a nonlinear hidden state, **h**. By default, this coupling is trained via backpropagation, while the dynamics of the memory remain fixed.
        
        .. image:: https://i.imgur.com/IJGUVg6.png
           :target: https://i.imgur.com/IJGUVg6.png
           :alt: Computational graph
        
        The discretized **A** and **B** matrices are initialized according to the LMU 's mathematical derivation with respect to some chosen window length, **θ**. Backpropagation can be used to learn this time-scale, or fine-tune **A** and **B**, if necessary.
        
        Both the kernels, **W**, and the encoders, **e**, are learned. Intuitively, the kernels learn to compute nonlinear functions across the memory, while the encoders learn to project the relevant information into the memory (see `paper <https://papers.nips.cc/paper/9689-legendre-memory-units-continuous-time-representation-in-recurrent-neural-networks.pdf>`_ for details).
        
        .. note::
        
           The ``paper`` branch in the ``lmu`` GitHub repository includes a pre-trained Keras/TensorFlow model, located at ``models/psMNIST-standard.hdf5``, which obtains the current best-known psMNIST result (using an RNN) of **97.15%**. Note that the network is using fewer internal state-variables and neurons than there are pixels in the input sequence. To reproduce the results from `this paper <https://papers.nips.cc/paper/9689-legendre-memory-units-continuous-time-representation-in-recurrent-neural-networks.pdf>`_, run the notebooks in the ``experiments`` directory within the ``paper`` branch.
        
        Nengo Examples
        --------------
        
        * `LMUs in Nengo (with online learning) <https://www.nengo.ai/nengo/examples/learning/lmu.html>`_
        * `Spiking LMUs in Nengo Loihi (with online learning) <https://www.nengo.ai/nengo-loihi/examples/lmu.html>`_
        * `LMUs in NengoDL (reproducing SotA on psMNIST) <https://www.nengo.ai/nengo-dl/examples/lmu.html>`_
        
        Citation
        --------
        
        .. code-block::
        
           @inproceedings{voelker2019lmu,
             title={Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks},
             author={Aaron R. Voelker and Ivana Kaji\'c and Chris Eliasmith},
             booktitle={Advances in Neural Information Processing Systems},
             pages={15544--15553},
             year={2019}
           }
        
        ***************
        Release history
        ***************
        
        .. Changelog entries should follow this format:
        
           version (release date)
           ======================
        
           **section**
        
           - One-line description of change (link to Github issue/PR)
        
        .. Changes should be organized in one of several sections:
        
           - Added
           - Changed
           - Deprecated
           - Removed
           - Fixed
        
        0.2.0 (November 2, 2020)
        ========================
        
        **Added**
        
        - Added documentation for package description, installation, usage, API, examples,
          and project information. (`#20`_)
        - Added LMU FFT cell variant and auto-switching LMU class. (`#21`_)
        - LMUs can now be used with any Keras RNN cell (e.g. LSTMs or GRUs) through the
          ``hidden_cell`` parameter. This can take an RNN cell (like
          ``tf.keras.layers.SimpleRNNCell`` or ``tf.keras.layers.LSTMCell``) or a feedforward
          layer (like ``tf.keras.layers.Dense``) or ``None`` (to create a memory-only LMU).
          The output of the LMU memory component will be fed to the ``hidden_cell``.
          (`#22`_)
        - Added ``hidden_to_memory``, ``memory_to_memory``, and ``input_to_hidden`` parameters
          to ``LMUCell``, which can be used to enable/disable connections between components
          of the LMU. They default to disabled. (`#22`_)
        - LMUs can now be used with multi-dimensional memory components. This is controlled
          through a new ``memory_d`` parameter of ``LMUCell``. (`#22`_)
        - Added ``dropout`` parameter to ``LMUCell`` (which applies dropout to the input)
          and ``recurrent_dropout`` (which applies dropout to the ``memory_to_memory``
          connection, if it is enabled). Note that dropout can be added in the hidden
          component through the ``hidden_cell`` object. (`#22`_)
        
        **Changed**
        
        - Renamed ``lmu.lmu`` module to ``lmu.layers``. (`#22`_)
        - Combined the ``*_encoders_initializer``parameters of ``LMUCell`` into a single
          ``kernel_initializer`` parameter. (`#22`_)
        - Combined the ``*_kernel_initializer`` parameters of ``LMUCell`` into a single
          ``recurrent_kernel_initializer`` parameter. (`#22`_)
        
        **Removed**
        
        - Removed ``Legendre``, ``InputScaled``, ``LMUCellODE``, and ``LMUCellGating``
          classes. (`#22`_)
        - Removed the ``method``, ``realizer``, and ``factory`` arguments from ``LMUCell``
          (they will take on the same default values as before, they just cannot be changed).
          (`#22`_)
        - Removed the ``trainable_*`` arguments from ``LMUCell``. This functionality is
          largely redundant with the new functionality added for enabling/disabling internal
          LMU connections. These were primarily used previously for e.g. setting a connection to
          zero and then disabling learning, which can now be done more efficiently by
          disabling the connection entirely. (`#22`_)
        - Removed the ``units`` and ``hidden_activation`` parameters of ``LMUCell`` (these are
          now specified directly in the ``hidden_cell``. (`#22`_)
        - Removed the dependency on ``nengolib``. (`#22`_)
        - Dropped support for Python 3.5, which reached its end of life in September 2020.
          (`#22`_)
        
        .. _#20: https://github.com/abr/lmu/pull/20
        .. _#21: https://github.com/abr/lmu/pull/21
        .. _#22: https://github.com/abr/lmu/pull/22
        
        0.1.0 (June 22, 2020)
        =====================
        
        Initial release of NengoLMU 0.1.0! Supports Python 3.5+.
        
        The API is considered unstable; parts are likely to change in the future.
        
        Thanks to all of the contributors for making this possible!
        
Platform: UNKNOWN
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Nengo
Classifier: Intended Audience :: Science/Research
Classifier: License :: Free for non-commercial use
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python 
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Topic :: Scientific/Engineering 
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.6
Provides-Extra: all
Provides-Extra: docs
Provides-Extra: optional
Provides-Extra: tests
