Neural autoregressive models

Density modeling, i.e. the problem of learning the distribution that generated some available data, is one of the most general and fundamental problem in machine learning. One simple approach to tackle this problem is to define some directed graph over the input observations and then learn the conditional distribution of each observation given its parents in the graph.

In The Neural Autoregressive Distribution Estimator (NADE), Iain Murray and I proposed a neural network specifically tailored to this task of density or distribution modeling. Compared to many baselines, NADE was able to reach state of the art performances on several different datasets. Published at the AISTATS 2011 conference, the paper received a Notable Paper Award.

Once NADE is trained, we can also easily generate samples from its learned distribution. When trained on small images of handwritten digit characters (with binary pixels), NADE can reproduce images that resemble such handwritten characters:

I'm now interested in how this simple yet powerful approach can be further developed to model more complex data with arbitrary structure.


  • The Neural Autoregressive Distribution Estimator [pdf] [talk] [code]
    Hugo Larochelle and Iain Murray,
    Artificial Intelligence and Statistics, 2011
    Notable Paper Award