NIPS 2016 Workshop on Approximate Inference

Fri, 30 Sep 2016 00:00:00 -0400

We’re organizing a NIPS workshop on approximate inference. It is together with Tamara Broderick, Stephan Mandt, and James McInerney—and alongside an incredible cast of seminal researchers: David Blei, Andrew Gelman, Mike Jordan, and Kevin Murphy. [Workshop homepage]

This year, we set a theme based on what we believe are some of the most important challenges. In particular, there’s an emphasis on the practice of approximate inference, whether it be challenges which arise in applications or in software. Advances in both methodology and theory are of course crucial to achieve this end-goal; we also highly encourage such work.

Note: We have (quite a few!) travel awards. If you’re interested in applying, the travel award and early application deadline is October 7.

Call for papers below.

We invite researchers in machine learning and statistics to participate in the:

NIPS 2016 Workshop on Advances in Approximate Bayesian Inference
Friday 9 December 2016, Barcelona, Spain
www.approximateinference.org
Submission deadline: 1 November 2016

1. Call for Participation

We invite researchers to submit their recent work on the development, analysis, or application of approximate Bayesian inference. A submission should take the form of an extended abstract of 2-4 pages in PDF format using the NIPS style. Author names do not need to be anonymized and references may extend as far as needed beyond the 4 page upper limit. If authors' research has previously appeared in a journal, workshop, or conference (including the NIPS 2016 conference), their workshop submission should extend that previous work. Submissions may include a supplement/appendix, but reviewers are not responsible for reading any supplementary material.

Submissions will be accepted either as contributed talks or poster presentations. Extended abstracts should be submitted by 1 November; see website for submission details. Final versions of the extended abstract are due by 5 December, and will be posted on the workshop website.

2. Workshop Overview

Bayesian analysis has seen a resurgence in machine learning, expanding its scope beyond traditional applications. Increasingly complex models have been trained with large and streaming data sets, and they have been applied to a diverse range of domains. Key to this resurgence has been advances in approximate Bayesian inference. Variational and Monte Carlo methods are currently the mainstay techniques, where recent insights have improved their approximation quality, provided black box strategies for fitting many models, and enabled scalable computation.

In this year's workshop, we would like to continue the theme of approximate Bayesian inference with additional emphases. In particular, we encourage submissions not only advancing approximate inference but also regarding (1) unconventional inference techniques, with the aim to bring together diverse communities; (2) software tools for both the applied and methodological researcher; and (3) challenges in applications, both in non-traditional domains and when applying these techniques to advance current domains.

This workshop is a continuation of past years:
+ NIPS 2015 Workshop: Advances in Approximate Bayesian Inference
+ NIPS 2014 Workshop: Advances in Variational Inference
This workshop has been endorsed by the International Society for Bayesian Analysis (ISBA) and is supported by Disney Research.

3. Confirmed Speakers and Panelists

Invited speakers:
Barbara Engelhardt (Princeton University)
Surya Ganguli (Stanford University)
Jonathan Huggins (MIT)
Jeffrey Regier (UC Berkeley)
Matthew Johnson (Harvard University)

Panel: Software
TBA (Stan)
Noah Goodman (WebPPL; Stanford University)
Dustin Tran (Edward; Columbia University)
TBA (TensorFlow, BayesFlow; Google)
Michael Hughes (BNPy; Harvard University)

Panel: On the Foundations and Future of Approximate Inference
Ryan Adams (Harvard University, Twitter Cortex)
Barbara Engelhardt (Princeton University)
Philip Hennig (Max Planck Institute for Intelligent Systems)
Richard Turner (University of Cambridge)
Neil Lawrence (University of Sheffield)

4. Key Dates

Travel award application deadline: 7 October 2016
Early acceptance notification: 7 October 2016
Paper submission: 1 November 2016
Acceptance notification: 16 November 2016
Travel award notification: 16 November 2016
Final paper submission: 5 December 2016

Workshop organizers:
Tamara Broderick (MIT)
Stephan Mandt (Disney Research)
James McInerney (Columbia University)
Dustin Tran (Columbia University)

Advisory committee:
David Blei (Columbia University)
Andrew Gelman (Columbia University)
Michael Jordan (UC Berkeley)
Kevin Murphy (Google)

Discussion of "Fast Approximate Inference for Arbitrarily Large Semiparametric Regression Models via Message Passing"

Mon, 19 Sep 2016 00:00:00 -0400

This article is written with much help by David Blei. It is extracted from a discussion paper on “Fast Approximate Inference for Arbitrarily Large Semiparametric Regression Models via Message Passing”. [link]

We commend Wand (2016) for an excellent description of message passing (mp) and for developing it to infer large semiparametric regression models. We agree with the author in fully embracing the modular nature of message passing, where one can define “fragments” that enable us to compose localized algorithms. We believe this perspective can aid in the development of new algorithms for automated inference.

Automated inference. The promise of automated algorithms is that modeling and inference can be separated. A user can construct large, complicated models in accordance with the assumptions he or she is willing to make about their data. Then the user can use generic inference algorithms as a computational backend in a “probabilistic programming language,” i.e., a language for specifying generative probability models.

With probabilistic programming, the user no longer has to write their own algorithms, which may require tedious model-specific derivations and implementations. In the same spirit, the user no longer has to bottleneck their modeling choices in order to fit the requirements of an existing model-specific algorithm. Automated inference enables probabilistic programming systems, such as Stan (Carpenter et al., 2016), through methods like automatic differentiation variational inference (advi) (Kucukelbir, Tran, Ranganath, Gelman, & Blei, 2016) and no U-turn sampler (nuts) (Hoffman & Gelman, 2014).

Though they aim to apply to a large class of models, automated inference algorithms typically need to incorporate modeling structure in order to remain practical. For example, Stan assumes that one can at least take gradients of a model’s joint density. (Contrast this with other languages which assume one can only sample from the model.) However, more structure is often necessary: advi and nuts are not fast enough by themselves to infer very large models, such as hierarchical models with many groups.

We believe mp and Wand’s work could offer fruitful avenues for expanding the frontiers of automated inference. From our perspective, a core principle underlying mp is to leverage structure when it is available—in particular, statistical properties in the model—which provides useful computational properties. In mp, two examples are conditional independence and conditional conjugacy.

From conditional independence to distributed computation. As Wand (2016) indicates, a crucial advantage of message passing is that it modularizes inference; the computation can be performed separately over conditionally independent posterior factors. By definition, conditional independence separates a posterior factor from the rest of the model, which enables mp to define a series of iterative updates. These updates can be run asynchronously and in a distributed environment.

Figure 1. A hierarchical model, with latent variables $\alpha_k$ defined locally per group and latent variables $\phi$ defined globally to be shared across groups.

We are motivated by hierarchical models, which substantially benefit from this property. Formally, let $y_{nk}$ be the $n^{th}$ data point in group $k$ , with a total of $N_k$ data points in group $k$ and $K$ many groups. We model the data using local latent variables $\alpha_k$ associated to a group $k$ , and using global latent variables $\phi$ which are shared across groups. The model is depicted in Figure 1.

The posterior distribution of local variables $\alpha_k$ and global variables $\phi$ is

$p(\alpha,\phi\mid\mathbf{y}) \propto p(\phi\mid\mathbf{y}) \prod_{k=1}^K \Big[ p(\alpha_k\mid \beta) \prod_{n=1}^{N_K} p(y_{nk}\mid\alpha_k,\phi) \Big].$

The benefit of distributed updates over the independent factors is immediate. For example, suppose the data consists of 1,000 data points per group (with 5,000 groups); we model it with 2 latent variables per group and 20 global latent variables. Passing messages, or inferential updates, in parallel provides an attractive approach to handling all 10,020 latent dimensions. (In contrast, consider a sequential algorithm that requires taking 10,019 steps for all other variables before repeating an update of the first.)

While this approach to leveraging conditional independence is straightforward from the message passing perspective, it is not necessarily immediate from other perspectives. For example, the statistics literature has only recently come to similar ideas, motivated by scaling up Markov chain Monte Carlo using divide and conquer strategies (Huang & Gelman, 2005; Wang & Dunson, 2013). These first analyze data locally over a partition of the joint density, and second aggregate the local inferences. In our work in Gelman et al. (2014), we arrive at the continuation of this idea. Like message passing, the process is iterated, so that local information propagates to global information and global information propagates to local information. In doing so, we obtain a scalable approach to Monte Carlo inference, both from a top-down view which deals with fitting statistical models to large data sets and from a bottom-up view which deals with combining information across local sources of data and models.

From conditional conjugacy to exact iterative updates. Another important element of message passing algorithms is conditional conjugacy, which lets us easily calculate the exact distribution for a posterior factor conditional on other latent variables. This enables analytically tractable messages (c.f., Equations (7)-(8) of Wand (2016)).

Consider the same hierarchical model discussed above, and set

$p(y_k,\alpha_k\mid \phi) = h(y_k, \alpha_k) \exp\{\phi^\top t(y_k, \alpha_k) - a(\phi)\},$ $p(\phi) = h(\phi) \exp\{\eta^{(0) \top} t(\phi) - a(\eta_0)\} .$

The local factor $p(y_k,\alpha_k\mid\phi)$ has sufficient statistics $t(y_k,\alpha_k)$ and natural parameters given by the global latent variable $\phi$ . The global factor $p(\phi)$ has sufficient statistics $t(\phi) = (\phi, -a(\phi))$ , and with fixed hyperparameters $\eta^{(0)}$ , which has two components: $\eta^{(0)} = (\eta^{(0)}_1,\eta^{(0)}_2)$ .

This exponential family structure implies that, conditionally, the posterior factors are also in the same exponential families as the prior factors (Diaconis & Ylvisaker, 1979),

$p(\phi\mid\mathbf{y},\alpha) = h(\phi) \exp\{\eta(\mathbf{y},\alpha)^\top t(\phi) - a(\mathbf{y},\alpha)\},$ $p(\alpha_k\mid y_k, \phi) = h(\alpha_k) \exp\{\eta(y_k, \phi)^\top t(\alpha_k) - a(y_k, \phi)\} .$

The global factor’s natural parameter is $\eta(\mathbf{y},\alpha) = (\eta^{(0)}_1 + \sum_{k=1}^K t(y_k, \alpha_k), \eta^{(0)}_2 + \sum_{k=1}^K N_k)$ .

With this statistical property at play—namely that conjugacy gives rise to tractable conditional posterior factors—we can derive algorithms at a conditional level with exact iterative updates. This is assumed for most of the message passing of semiparametric models in Wand (2016). Importantly, this is not necessarily a limitation of the algorithm. It is a testament to leveraging model structure: without access to tractable conditional posteriors, additional approximations must be made. Wand (2016) provides an elegant way to separate out these nonconjugate pieces from the conjugate pieces.

In statistics, the most well-known example which leverages conditionally conjugate factors is the Gibbs sampling algorithm. From our own work, we apply the idea in order to access fast natural gradients in variational inference, which accounts for the information geometry of the parameter space (Hoffman, Blei, Wang, & Paisley, 2013). In other work, we demonstrate a collection of methods for gradient-based marginal optimization (Tran, Gelman, & Vehtari, 2016). Assuming forms of conjugacy in the model class arrives at the classic idea of iteratively reweighted least squares as well as the EM algorithm. Such structure in the model provides efficient algorithms—both statistically and computationally—for their automated inference.

Open Challenges and Future Directions. Message passing is a classic algorithm in the computer science literature, which is ripe with interesting ideas for statistical inference. In particular, mp enables new advancements in the realm of automated inference, where one can take advantage of statistical structure in the model. Wand (2016) makes great steps following this direction.

With that said, important open challenges still exist in order to realize this fusion.

First is about the design and implementation of probabilistic programming languages. In order to implement Wand (2016)’s message passing, the language must provide ways of identifying local structure in a probabilistic program. While that is enough to let practitioners use mp, a much larger challenge is to then automate the process of detecting local structure.

Second is about the design and implementation of inference engines. The inference must be extensible, so that users can not only employ the algorithm in Wand (2016) but easily build on top of it. Further, its infrastructure must be able to encompass a variety of algorithms, so that users can incorporate mp as one of many tools in their toolbox.

Third, we think there are innovations to be made on taking the stance of modularity to a further extreme. In principle, one can compose not only localized message passing updates but compose localized inference algorithms of any choice—whether it be exact inference, Monte Carlo, or variational methods. This modularity will enable new experimentation with inference hybrids and can bridge the gap among inference methods.

Finally, while we discuss mp in the context of automation, fully automatic algorithms are not possible. Associated to all inference are statistical and computational tradeoffs (Jordan, 2013). Thus we need algorithms along the frontier, where a user can explicitly define a computational budget and employ an algorithm achieving the best statistical properties within that budget; or conversely, define desired statistical properties and employ the fastest algorithm to achieve them. We think ideas in mp will also help in developing some of these algorithms.

References

Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., … Riddell, A. (2016). Stan: A probabilistic programming language. Journal of Statistical Software.
Diaconis, P., & Ylvisaker, D. (1979). Conjugate Priors for Exponential Families. The Annals of Statistics, 7(2), 269–281.
Gelman, A., Vehtari, A., Jylänki, P., Robert, C., Chopin, N., & Cunningham, J. P. (2014). Expectation propagation as a way of life. ArXiv Preprint ArXiv:1412.4869.
Hoffman, M. D., Blei, D. M., Wang, C., & Paisley, J. (2013). Stochastic Variational Inference. Journal of Machine Learning Research, 14, 1303–1347.
Hoffman, M. D., & Gelman, A. (2014). The no-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15, 1593–1623.
Huang, Z., & Gelman, A. (2005). Sampling for Bayesian Computation with Large Datasets. SSRN Electronic Journal.
Jordan, M. I. (2013). On statistics, computation and scalability. Bernoulli, 19(4), 1378–1390.
Kucukelbir, A., Tran, D., Ranganath, R., Gelman, A., & Blei, D. M. (2016). Automatic Differentiation Variational Inference. ArXiv Preprint ArXiv:1603.00788.
Tran, D., Gelman, A., & Vehtari, A. (2016). Gradient-based marginal optimization. Technical Report.
Wand, M. P. (2016). Fast Approximate Inference for Arbitrarily Large Semiparametric Regression Models via Message Passing. ArXiv Preprint ArXiv:1602.07412.
Wang, X., & Dunson, D. B. (2013). Parallelizing MCMC via Weierstrass sampler. ArXiv Preprint ArXiv:1312.4605.

Blog has migrated from Ghost to Jekyll

Thu, 11 Aug 2016 00:00:00 -0400

In the past few days I spent time migrating the blog from Ghost to Jekyll.

The theme builds off Type Theme, and is heavily inspired by Otoro, the New York Times, and the Rosenrot. The annals-like frontpage takes cue from Paul Graham’s essays and Cosma Shalizi’s notebooks.

Top: Old frontpage. Bottom: New frontpage.

Hooray for math! (using KaTeX)

$E(w_i,c_k,z) = -\sum_{j=1}^z w_{i,j} c_{k,j} + \Big[z\log a -\sum_{j=1}^z (w_{i,j}^2 + c_{k,j}^2)\Big].$

Hooray for code snippets! (taken from Edward)

def set_seed(x):
  """Set seed for both NumPy and TensorFlow.

  Parameters
  ----------
  x : int, float
    seed
  """
  np.random.seed(x)
  tf.set_random_seed(x)

Hooray for bibliographies! … Okay I don’t quite have that one set up yet. But it’s doable!

The blog’s permalinks are the same. The RSS feed is the same.

Why the change?

I liked Ghost, having migrated from Wordpress and with the impression of a sleeker and more modern content management system (CMS). But CMS’ have always been a bother for me because I could never understand the internals. It would always be a reliance on plugins in order to accomplish things. Even with Ghost some things were too automagical and hidden to the user.

Time to go even more bare-bones with a static site generator. All code and content for this blog is available.

Here’s to hoping the redesign motivates me to write more often!

Inference networks

Thu, 21 Jul 2016 00:00:00 -0400

I wrote a tutorial on inference networks on the Edward website. I normally don’t like writing outsourced blog posts, but I figure it’s better to link there than write duplicated content which may later become outdated.

Thanks go to Kevin Murphy for motivating the tutorial as it is based on our discussions, and also related discussion with Jaan Altosaar.

On Bayesian and frequentist, latent variables and parameters

Sun, 03 Jul 2016 00:00:00 -0400

I’ve been helping write tutorials that teach concepts such as black box variational inference in Edward. And as I’ve been editing, I’ve noticed the majority of my suggestions are about the writing rather than the code. Communication is important—arguably more important in papers than the idea itself. Communication evokes different lines of thinking about how to approach problems, and it always has subtleties regarding the different culture and set of problems one finds important. Here are a few subtleties that have come up recently and my stock answers to them.

“Bayesian models” framed as placing priors on parameters of a likelihood. This is a very frequentist -> Bayesian line of thinking. I strongly believe models should simply be framed as a joint distribution $p(x, z)$ for data $x$ and latent variables $z$ .¹ The other line of thinking can lead to misunderstandings about Bayesian analysis. For example, a classical argument (and one which I still hear among some of my statistician friends in Berkeley and Stanford) is that Bayesian analysis is subjective because the prior $p(z)$ is subjective. But in fact the entire model is subjective—both likelihood and prior as Andrew repeatedly states. The distinction away from the fact that you’re specifying a model is often meaningless. This relates to the old adage that “all models are wrong” (Box, 1976). Any model you work with is “subjective” according to the assumptions you’re willing to make in the model.
Bayesian versus frequentist models. The secret that no one wants to say publicly is that there’s no difference! A “latent variable model” or “hierarchical model” or “generative model” in Bayesian literature is just a “random effects model” in frequentist literature. I try to avoid attaching specific statistical methodologies to models because there is no difference. They are all just “probabilistic models”. The statistical methodology—whether it be Bayesian, frequentist, fiducial, whatever—is about how to reason with the model given data. That is, they are about “inference” and not the “model”. Why else do we do EM, a frequentist tool, on hierarchical models, typically labeled a Bayesian tool?

Frequentists may often use only the likelihood $p(x \mid z)$ as the model, so it is trivially a joint distribution $p(x, z)$ with a point mass distribution for $z$ . But it’s useful to keep in mind the joint distribution as it explicitly bakes in the modeling assumptions one makes.
Latent variables vs parameters. These days I try to stick to calling $z$ latent variables rather than parameters (and hence why I prefer to use $z$ rather than $\theta$ ). Parameters connote the idea of having only one setting, and it brings up the whole frequentist-Bayesian debacle about whether parameters can be random. Calling them latent variables instead simply state that the model uses random variables that remain unobserved during inference. We may be interested in inferring the latent variables, and thus it is natural to think about what $p(z \mid x)$ means, or what it means to approximate $p(z \mid x)$ with a point. Or we may not be directly interested in the latent variables, and thus it’s natural to integrate over them during inference.² Somehow I find this less convincing when thinking about $z$ as “parameters”. “Parameters” introduce a concept beyond random variables and their realizations, and which I can’t formally wrap my head around.³
Prior belief vs prior information. I’ve started to follow Andrew in using the latter and avoiding the former. Prior belief makes Bayesian analysis sound like a religion. Prior information comes not only in the context of prior information about the latent variables but prior information about the problem (e.g., data’s likelihood) in general.

¹ For clarity, I’m ignoring the nuance with probabilistic programs, in the same way we typically don’t communicate everything using measures, or by first describing the category we’re working in.

² Random effects (Eisenhart, 1947), incidental parameters (Neyman and Scott, 1948), nuisance parameters (Kalbfleisch and Sprott, 1970), and latent data (Rubin, 1976) are often explained in frequentist literature as something to integrate over. But I don’t like them because they don’t explain what it means to treat them as random variables.

³ In practice, we often work with “parameters” in the sense that we often work with point estimates during inference. This doesn’t mean that we shouldn’t continue working with them—it’s just that in principle it’s better to think about the optimal thing (inferring the full distribution) and viewing the point estimate as an approximation to the optimal thing. That is, what we should do in principle and what we should do in practice.

Variational auto-encoders do not train complex generative models

Thu, 23 Jun 2016 00:00:00 -0400

There is a tutorial on variational auto-encoders which popped up on my arXiv radar this week. Thanks to Carl Doersch for writing this tutorial! As a researcher in this area, I believe there is sorely a need for an exposition of recent developments in variational inference. Variational inference’s application for enabling deep generative models has exploded in the past few years. This tutorial is a great step in that direction. Inspired by this and discussions with others during ICML, I’d like to make a few comments to clarify misunderstandings that are pervasive among newcomers to variational inference, Bayesian analysis, and/or auto-encoders.

Variational auto-encoders are not a way to train generative models. The generative model is part of the variational auto-encoder, which is typically a deep latent Gaussian model (Rezende et al., 2014). Training generative models is done by inference, typically variational inference (Hinton and Van Camp, 1993; Waterhouse et al., 1996) with an inference network. This model-inference combination is what defines a variational auto-encoder (Kingma and Welling, 2014) and more classically a Helmholtz machine (Dayan et al., 1995). Making precise the separate components of model and inference is crucial for extensibility. For example, how can we build on the inference, leveraging Monte Carlo methods (MacKay and Gibbs, 1999; Salimans et al., 2015), or build on the modeling, leveraging recurrent compositions (Gregor et al., 2015)? If I learned anything at my time at Harvard statistics with people like Don Rubin, I learned that being precise about our language prevents our discourse from being ambiguous and our terms from becoming overloaded.

Generative models are usually motivated from their immediate application of generating data that look like the training data. Variational auto-encoders, like, say, generative adversarial networks, are most generally a way to both postulate and infer complex data generative processes. This is possibly for unsupervised tasks, possibly supervised tasks, semi-supervised tasks, or even causal inferences. They are part of an underlying statistical methodology.¹ This methodology simply frames models as distributions over data and latent variables, allowing models to address a broad array of downstream tasks with any size of data. For example, deep generative models can achieve “state-of-the-art” for supervised learning on small amounts of data (Damianou and Lawrence, 2013) and generalization with few examples (Salakhutdhinov et al., 2013).

The neural network used in the encoder (variational distribution) does not lead to any richer approximating distribution. It is a way to amortize inference such that the number of parameters does not grow with the size of the data (an incredible feat, but not one for expressivity!) (Stuhlmüller et al., 2013). This is better explained from the perspective of variational inference than auto-encoders. For example, suppose we have a model with latent variables that are Gaussian a priori (as in VAEs). We may choose a fully factorized Gaussian as the variational distribution, where each latent variable is rendered independent. The inference network takes data as input and outputs the local variational parameters relevant to each data point. The optimal inference network outputs the set of Gaussian parameters which maximizes the variational objective. This means a variational auto-encoder with a perfect inference network can only do as well as a fully factorized Gaussian with no inference network.²

¹ The underlying statistical methodology of latent variable models is typically Bayesian. However, the distinction between frequentist and Bayesian models is practically none, e.g., Harville (1977).

² Much recent work, including my own, has tried to improve this simple approximation to improve our approximate inference. I invite you to the incredible developments since the past year: Rezende and Mohamed, 2015; Salimans et al., 2015; Tran et al., 2015; Tran et al., 2016; Burda et al., 2016; Ranganath et al., 2016; Maaløe et al., 2016; Mnih and Rezende, 2016; Louizos et al., 2016; Johnson et al., 2016; Dinh et al., 2016; Kingma et al., 2016.

References

Burda, Y., Grosse, R., & Salakhutdinov, R. (2016). Importance Weighted Autoencoders. In International Conference on Learning Representations.
Damianou, A. C., & Lawrence, N. D. (2013). Deep Gaussian Processes. In Artificial Intelligence and Statistics.
Dayan, P., Hinton, G. E., Neal, R. M., & Zemel, R. S. (1995). The Helmholtz Machine. Neural Computation, 7(5), 889–904. http://doi.org/10.1162/neco.1995.7.5.889
Dinh, L., Sohl-Dickstein, J., & Bengio, S. (2016). Density estimation using Real NVP. arXiv.org.
Harville, D. A (1977). Maximum likelihood approaches to variance component estimation and to related problems. Journal of the American Statistical Association, 72(358):320–338.
Hinton, G. and Van Camp, D (1993). Keeping the neural networks simple by minimizing the description length of the weights. In Computational Learning Theory, pp. 5–13. ACM.
Johnson, M. J., Duvenaud, D., Wiltschko, A. B., Datta, S. R., & Adams, R. P. (2016). Composing graphical models with neural networks for structured representations and fast inference. arXiv.org.
Kingma, D. P., & Welling, M. (2014). Auto-Encoding Variational Bayes. In International Conference on Learning Representations.
Kingma, D. P., Salimans, T., & Welling, M. (2016). Improving Variational Inference with Inverse Autoregressive Flow. arXiv.org.
Louizos, C., & Welling, M. (2016). Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors. In International Conference on Machine Learning.
Maaløe, L., Sønderby, C. K., Sønderby, S. K., & Winther, O. (2016). Auxiliary Deep Generative Models. In International Conference on Machine Learning.
MacKay, D. J., & Gibbs, M. N. (1999). Density networks. Statistics and neural networks: advances at the interface. Oxford University Press, Oxford, 129-144.
Mnih, A., & Rezende, D. J. (2016). Variational inference for Monte Carlo objectives. In International Conference on Machine Learning.
Ranganath, R., Tran, D., & Blei, D. M. (2016). Hierarchical Variational Models. In International Conference on Machine Learning.
Rezende, D. J., Mohamed, S., & Wierstra, D. (2014). Stochastic Backpropagation and Approximate Inference in Deep Generative Models. In International Conference on Machine Learning.
Salimans, T., Kingma, D. P., & Welling, M. (2015). Markov Chain Monte Carlo and Variational Inference: Bridging the Gap. In International Conference on Machine Learning.
Salakhutdinov, R., Tenenbaum, J. B., and Torralba, A (2013). Learning with hierarchical-deep models. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 35 (8):1958–1971.
Stuhlmüller, A., Taylor, J., & Goodman, N. (2013). Learning Stochastic Inverses. In Neural Information Processing Systems.
Tran, D., Blei, D. M., & Airoldi, E. M. (2015). Copula variational inference. In Neural Information Processing Systems.
Tran, D., Ranganath, R., & Blei, D. M. (2016). The Variational Gaussian Process. International Conference on Learning Representations.
Waterhouse, S., MacKay, D., and Robinson, T (1996). Bayesian methods for mixtures of experts. In Neural Information Processing Systems.

A quick update: Edward, and some motivations

Mon, 30 May 2016 00:00:00 -0400

As you may (or may not) know, I’ve been busy lately spear-heading Edward, an open-source library for probabilistic modeling. It’s meant to help bridge the gap between what I view as two dichotomous approaches of statistical learning: one approach develops complex models in order to achieve the best results on a specific task; and the other approach adheres to simple models in order to understand every component of the analysis both empirically and theoretically. The former starts at the end-goal; the latter starts at the foundation.

There are many names you can append to one of these approaches, and certainly those names imply contrasting motivations due to culture, and thus contrasting views and contrasting applications. But ultimately, the goal is still the same. The approaches are not orthogonal, but a lack of awareness connecting the two make them seem to be. A neural network is a powerful aproach for modeling non-linear functions (I mean this not tongue-in-cheek; it’s difficult to summarize many decades of innovation in a sentence). A Bayesian linear model is a powerful approach for incorporating parameter uncertainty during supervision, and for accessing a basis on which to validate our models.

How do we assess model fit from a complex neural network architecture, and generalize to any setting, whether it be small data, simulation-based tasks, or even causal inferences? How do we build more expressive models when our tools tell us the generalized linear model fits okay, but is not nearly as fine-grained as we’d like it to be, or let our inferences scale to data that no longer fits in memory? If we use the many innovations of deep learning in concert with statistical analysis, or conversely, the century of statistical foundations in deep learning, we might just achieve something quite grand. (And I mean this one only a little tongue-in-cheek.)

Edward, broadly speaking, tries to combine efforts from both approaches. It’s a software library I’ve always wanted to develop but never had the right resources until now. Fast and distributed computation can be done using TensorFlow as a rich symbolic framework. Neural networks can be easily constructed with high-level libraries like Keras. Flexible probability models can be specified using languages such as Stan. And all of inference can be done using fast approximations via a built-in variational inference engine, with criticism techniques for both point prediction and distribution-based model assessments.

I gave an Edward talk a few days ago at Google Brain— which I quite positively butchered due to a lack of sleep and preparation. (If only I had known I would get a sizeable cast of the TensorFlow core developers, Geoff Hinton, Jeff Dean, and Kevin Murphy in one room!) But I think the work explained itself, and with a lot of excitement about why Bayesian deep learning might be the right thing.

There are a number of necessary developments in Edward to even make small steps in connecting these two approaches. I finally got to refactoring the code for variational auto-encoders; and designing the criticism API for posterior predictive checks; and getting much help from others for explaining how to build neural network-based probabilistic models; and many open problems to try to solve. It’s not even close to there for getting statisticians, machine learners, and deep learners to agree with one another. But I think Edward is making progress.

Video resources for machine learning

Sun, 03 Jan 2016 00:00:00 -0500

One feature unique to our field is the sheer amount of online resources. It’s one of the reasons I personally got into machine learning over other disciplines, as self-learning is much more accessible.

This is especially true for video resources. Part of this is because as a field we are very well integrated with computer science, and thus we’re technologically more motivated. (However, arguably no other computer science research domain has as many resources as us.) Whatever the reason, it’s important that we take advantage of all the information available in order to maximize learning.

I haven’t found a good collated list of machine learning videos, so I decided to compile my own. In general the following are excellent resources:

VideoLectures.NET is the primary video archive for machine learning. Browse the homepage and you’ll see almost all machine learning videos featured on the frontpage, ranging from conferences, workshops, lectures, and even discussions.
TechTalks.tv is the second most used archive. Notably, many ICML videos are located here.
Youtube contains a few, such as certain AISTATS and ICLR years. It’s best for collecting lectures, such as by Nando de Freitas and Alex Smola.

Here are all the video archives specific to conferences and workshops that I could find. ¹ Over this winter I’ve been relaxing and marathoning these as if they were TV shows—it’s quite a lot of fun!

Hope this is of use. I wish I had time to go further and collect resources regarding specific ideas or techniques, such as David Blei’s 2009 lecture on Topic models, or Peter Orbanz’s 2009 lecture on Foundations of Bayesian nonparametrics. Perhaps someone not me will do this in the future.

¹ If you have links to others not listed here and that you feel should be added, or if there is a broken link, please contact me at dustin@cs.columbia.edu. Any help is appreciated and would also help the community at large. :-)

Stochastic Expectation Propagation

Sun, 06 Dec 2015 00:00:00 -0500

Yingzhen Li, Jose Miguel Hernandez-Lobato, and Richard E. Turner. Stochastic Expectation Propagation. In Neural Information Processing Systems, 2015.

Summary

Expectation propagation (EP) is a popular technique for approximate Bayesian inference, although it has arguably lost favor in recent years to variational inference. Whereas variational inference was made scalable to massive data sets through stochastic variational inference (Hoffman et al., 2013), and thus attractive to machine learning practitioners, expectation propagation is limited by a memory overhead which scales with the number of data points. This paper proposes an extension of EP which removes the complexity requirement and thus enables it to scale to massive data.

Let $\mathbf{x}=\{\mathbf{x}_k\}$ be a data set split into $K$ components, and $p(\mathbf{x}\mid\mathbf{\theta})$ be a model likelihood with prior $p(\mathbf{\theta})$ . We aim to learn an approximating distribution $q(\mathbf{\theta})$ such that

$p(\mathbf{\theta}\mid \mathbf{x}) \propto p(\mathbf{\theta})\prod_{k=1}^K p(\mathbf{x}_k\mid\mathbf{\theta}) \approx q(\mathbf{\theta}) \propto p(\mathbf{\theta})\prod_{k=1}^K f_k(\mathbf{\theta}).$

Following the way Zoubin Ghahramani motivates EP, we would like to minimize $\mathrm{KL}(p(\mathbf{\theta}\mid \mathbf{x})\|q(\mathbf{\theta}))$ (I have no clue who first motivated EP from this way—Zoubin’s slides were how I first learned this). This can be interpreted as a reverse analog of the variational objective: whereas VI is restricted to the support of the posterior and is “mode-seeking” (Minka, 2005), this KL contains at least the support of the posterior and is “inclusive” (Frey, 2000). Minimizing this KL could proceed by iteratively updating the factors $f_k(\mathbf{\theta})$ . Denoting the leave-one-out posterior $p_{-k}(\mathbf{\theta})=p(\mathbf{\theta}\mid \mathbf{x})/p(\mathbf{x}_k\mid\mathbf{\theta})$ , we iteratively minimize $\mathrm{KL}(p(\mathbf{\theta}\mid \mathbf{x})\|p_{-k}(\mathbf{\theta})f_k(\mathbf{\theta}))$ for each $k=1,\ldots,K$ .

However, $\mathrm{KL}(p\|q)$ is intractable (and so is the iterative version) as it requires calculating the posterior density; therefore we minimize an approximation to it. Consider approximating the leave-one-out posterior with the cavity distribution $q_{-k}(\mathbf{\theta}) = q(\mathbf{\theta})/f_k(\mathbf{\theta})$ , which is the approximating distribution without the $k^{th}$ likelihood approximation. Define the tilted distribution $q_{\backslash k}(\mathbf{\theta})=q_{-k}(\mathbf{\theta}) p(\mathbf{x}_k\mid\mathbf{\theta})$ , which is the corresponding cavity distribution with the true likelihood factor plugged in. EP iteratively solves

$\min_{f_k}\mathrm{KL}(q_{\backslash k}(\mathbf{\theta}) \| q_{-k}(\mathbf{\theta}) f_k(\mathbf{\theta})),\quad k=1,\ldots,K.$

This objective is a good proxy to $\mathrm{KL}(p\|q)$ if the true likelihood factor $p(\mathbf{x}_k\mid\mathbf{\theta})$ contributes to the leave-one-out posterior in the same way as it does in the tilted distribution.

Calculating the cavity distribution for each $k$ requires explicit storage of the approximating factors $f_k(\mathbf{\theta})$ . This implies that EP has an $\mathcal{O}(K)$ memory requirement which prevents it from scaling to large data sets. To solve this, the authors consider storage of only a single factor $f(\mathbf{\theta})$ ; rather than trying to capture the individual effect of the likelihood factors, the single factor captures only the average effect. Stochastic expectation propagation (SEP) uses the same iterative KL objective to learn the next approximating factor $f_k(\mathbf{\theta})$ , and additionally updates $f(\mathbf{\theta})=f(\mathbf{\theta})^{1-1/K}f_k(\mathbf{\theta})^{1/K}$ . By storing only the average effect of the likelihood factors, SEP approximates the true cavity distributions $q_{-k}(\mathbf{\theta})\propto q(\mathbf{\theta})/f_k(\mathbf{\theta})$ with an averaged cavity distribution $q_{-1}(\mathbf{\theta})\propto q(\mathbf{\theta})/f(\mathbf{\theta})$ .

Discussion

I’m a big fan of the paper. The exposition on EP, assumed density filtering, and message passing is excellent, and it naturally motivates SEP following the technical developments of EP. The use of an averaged cavity distribution is an unfortunate compromise, but it seems necessary in order to scale local approximation techniques such as EP.

The authors make a cool connection to SVI. If the local minimizations in SEP are instead doing the reverse KL, then this is a form of “stochastic” variational message passing. Moreover, if the update for the single factor $f(\mathbf{\theta})$ follows $f(\mathbf{\theta})^{1 - \epsilon}f_n(\mathbf{\theta})^{\epsilon}$ where $\epsilon$ uses a Robbins-Monro schedule, then this is the same as stochastic variational inference with a mean-field approximation. An interesting extension then is to apply the equivalent of an adaptive learning rate in stochastic approximations for the geometric scale factor when updating $f(\mathbf{\theta})$ .

Hogwild (Nui et al., 2011) and asynchronous versions also naturally follow which is cool. I also think it would have been useful for the paper to fight for the streaming data stance. That is, it’s trivial for stochastic EP to learn on streaming data whereas it is impossible for vanilla EP.

The move to $\mathrm{KL}(p\|q)$ with SEP does require some compromises in contrast to SVI unfortunately. For instance, unlike VI to SVI, EP to SEP compromises the approximation quality in order to make it scalable. That is, SVI still performs the same global minimization of $\mathrm{KL}(q\|p)$ , and shares nice properties through theory of stochastic approximations (Robbins and Monro, 1951), whereas SEP does not preserve the same local minimization as EP. You can imagine scenarios where an averaged likelihood approximator will not work, or even the “minibatch” M of K extension of them. Additionally, it’s not immediately clear how to extend EP beyond approximations which preserve dependence between the local approximations.

Minor comment: like much of Tom Minka’s work, they note that the local KL minimizations can more generally use $\alpha$ -divergence measures. I still think this is a little restricting, as in general you just want some approximation; therefore something like a Laplace approximation (Smola et al., 2004) is valid as well, and similarly, nested EP (Riihimaki et al., 2013) is more conducive following this motivation.

I’m excited to hear more about stochastic EP from their poster and spotlight at NIPS! (The same authors have a number of interesting extensions of this paper at a few workshops as well.)

At NIPS 2015

Fri, 04 Dec 2015 00:00:00 -0500

I’m quite excited for NIPS this year, which starts this Monday.

It’s always interesting to look at the recent trends from browsing the schedule. Deep learning and reinforcement learning are of course a big highlight of NIPS, with three tutorials, a symposium, and many affiliated workshops. Approximate inference is becoming increasingly more mainstream as complex probabilistic models are fit with scalable inference strategies. Similarly, probabilistic programming—and the general concept of developing generic algorithms which fit broad classes of models—is becoming crucial to model prototyping. Gaussian processes are big just as they were in the last ICML: Bayesian optimization, Bayesian nonparametrics, and kernel methods are in general becoming more feasible and used in practice.

On personal details, I’ll be presenting a poster on Copula variational inference, which is joint work with Dave Blei and Edo Airoldi, trying to tackle the general problem of learning posterior correlation in variational inference. I’m also helping organize the Advances in Approximate Bayesian Inference workshop on Friday. We have an incredible list of talks lined up, and two panels which I’m sure will be insightful. On Saturday, I’ll be giving a talk at the Black Box Learning and Inference workshop. It covers my most recent research on developing a rich framework for variational inference (abstract below). The Stan team also has a webpage on NIPS.

Title: Variational models

Abstract:

Originally developed in the 1990s, variational inference has enjoyed renewed interest around scalable optimization for large datasets, black box strategies for inferring broad model classes, and neural networks for amortized inference. However, this research has been constrained by the lack of an expressive family of approximating distributions, which can generalize across many models. I will present the framework of variational models, which interpret the approximating family as a model of the posterior latent variables. This naturally leads to Bayesian constructs for developing expressive approximations. Following this, I will present a particular instantiation termed the variational Gaussian process. With applications to auto-encoders and deep recurrent attention models, I show that the variational model’s complexity grows efficiently and towards any distribution.

Joint work with Rajesh Ranganath and Dave Blei.

Denoising Criterion for Variational Auto-Encoding Framework

Tue, 01 Dec 2015 00:00:00 -0500

Daniel Jiwoong Im, Sungjin Ahn, Roland Memisevic, and Yoshua Bengio. Denoising Criterion for Variational Auto-Encoding Framework. arXiv preprint arXiv:1511.06406, 2015.

Summary

Auto-encoders are used to learn representations of observed data $x$ . They do so by minimizing the reconstruction error of a decoder $p(x\mid z)$ , given a characterization of hidden variables $z$ from an encoder $q(z\mid x)$ . Along with a penalty $P_\alpha$ for regularization, this can be written as maximizing the objective

$\mathcal{L}_{\text{ae}} = \mathbb{E}_{q(z\mid x)}[\log p(x\mid z)] - P_\alpha$

over the encoder and decoder’s parameters. However, it’s unclear what a principled choice of the penalty is, as well as the choice of the encoder for a given decoder $p(x\mid z)$ . Variational auto-encoders (Kingma and Welling, 2014; Rezende et al., 2014) reinterpret the evidence lower bound prominent in variational inference for this purpose:

$\mathcal{L}_{\text{vae}} = \mathbb{E}_{q(z\mid x)}[\log p(x\mid z)] - \mathrm{KL}(q(z\mid x)\| p(z)),$

where $p(z)$ is a model prior placed over the hidden variables of the encoder. The penalty term is automatically given as the KL divergence between the encoder and the prior, and the encoder can more generally be specified following variational inference literature. This is now a probabilistic encoder-decoder, with a principled objective given by a lower bound to the model evidence $\log p(x)$ .

In this paper, Im et al. take ideas from denoising auto-encoders (Vincent et al., 2008) and apply it to variational auto-encoders. In particular, they consider a corrupted distribution $q(x'\mid x)$ , such that

$q(z\mid x) = \int q(z\mid \tilde{x})q(\tilde{x}\mid x)\mathrm{d}\tilde{x}.$

With a distribution to model corrupted noise, the augmented $q(z\mid x)$ is more expressive than the original $q(z\mid x)$ without corruption: it is able to model additional uncertainty around the data inputs.

For inference, plugging this variational distribution into the ELBO leads to an analytically intractable objective, as the log-density $\log q(z\mid x)$ cannot be computed. As a proxy, they propose maximizing the objective function

$\mathcal{L}_{\text{dvae}} = \mathbb{E}_{q(\tilde{x}\mid x)}\Big[\mathbb{E}_{q(z\mid x)}\Big[\log\frac{p(x, z)}{q(z\mid\tilde{x})}\Big]\Big],$

which they show is in fact an upper bound to the original ELBO. They also claim it is still a lower bound to the model evidence $\log p(x)$ , and thus still a valid variational objective (I’ll comment on this later). Gradients proceed as typically done in the literature, either using the reparameterized gradient for differentiable latent variables, or the score function estimator in the more general case. They have an interesting Q&A format for discussing their experiments that I found kind of cool.

Discussion

This paper is eerily similar to a paper Rajesh Ranganath, Dave Blei, and I uploaded just two weeks earlier than theirs, on Hierarchical Variational Models. And that was uploaded only three weeks ago from today! Man, the field moves fast.

The idea is the same: place a prior on inputs to the variational distribution—treating them as variational latent variables—and proceed to marginalize them out. The new inputs are now the inputs to the prior distribution. In their case, the prior is on the data input for inference networks, which output local variational parameters. In our case, the prior is on variational parameters more generally. A prior on data input is the special case when the variational model uses an inference network: following our notation, we replace $q(z;\lambda)$ with the more powerful $q(z;\theta)=\int q(z\mid\lambda)q(\lambda;\theta)\mathrm{d}\lambda$ , and use an inference network $\phi$ to output prior hyperparameters $\theta=\phi(x)$ (see end of Section 5).
The source of inspiration is different: they follow the standard procedure in denoising auto-encoders. We follow the Bayesian framework, viewing the variational distribution as a model of the posterior latent variables. These different sources of inspiration motivate difference choices of complexity for the prior: they only consider Bernoulli or Gaussian, whereas from the Bayesian interpretation, it is natural to study more complex priors such as the normalizing flow (Rezende and Mohamed, 2015) as a prior; we show the latter leads to more expressive approximations and also more scalable computation. (And in our more recent paper on the Variational Gaussian Process, we use an even cooler prior :^).
The variational objectives are different, and I have concerns that their objective is not in fact a lower bound on $\log p(x)$ .¹ This means the values of the objective function are no longer a proxy for the model evidence. Among a few things, this invalidates the experimental results, as they compare to methods based on reported variational bounds. Further, maximizing their objective can lead to poor approximations, as there is no stopping the function to blow up to arbitrary values as the global maximum. Actually, it surprises me that the maximized variational objective values in the experiments are still close to those in $\mathcal{L}_{\text{vae}}$ . Would be very interested to hear from the authors on these issues.

Regardless of whether the theorems and experiments are correct, denoising variational auto-encoders is a much needed idea. It provides great intuition for why one should use hierarchical instantiations of variational models. Moreover it’s always interesting to see the same ideas inspired from a different field. These intuitions especially go nicely with our VGP paper, in which we make more explicit the auto-encoder connection for learning variational models with latent variables. Maybe we’ll add these details in a future revision.

Minor point: They claim that injecting noise both at the data input level (“denoising auto-encoder”) and the stochastic hidden layer level (“variational auto-encoder”) can be advantageous. However, they’re really one and the same, which I’m sure they’re aware of as they wrote Lemma 1 in this light. That is, the corrupted hidden variable $\tilde{x}$ is just another stochastic hidden layer, which is marginalized out. This makes the abstract and introduction a little confusing.

¹ In the proof of Lemma 1, when they try to show that $\log p(x)$ upper bounds $\mathcal{L}_{\text{dvae}}$ , there are three expectations expanded out in the first line. If the two expectations regarding the distribution $q(z'\mid x)$ are more explicitly written, it is clear why the $q$ ’s do not necessarily cancel:

$\mathbb{E}_{q_{\psi}(z''\mid x)} \mathbb{E}_{q_{\psi}(z'\mid x)} \Big[ \log \mathbb{E}_{q_{\varphi}(z\mid z'')} \Big[ \frac{p_\theta(x,z)}{q_{\varphi}(z\mid z')} \Big] \Big] \ne \mathbb{E}_{q_{\psi}(z''\mid x)} \mathbb{E}_{q_{\psi}(z'\mid x)} [ \log p_\theta(x)].$

When discussing this with Rajesh Ranganath, he points out a simple counterexample where $\mathcal{L}_{\text{dvae}}>\log p(x)$ : Consider a one-dimensional latent variable model, with $p(x,z)=\mathrm{Bernoulli}(z;0.5)$ . Let

$q(\tilde{x}\mid x) = \mathrm{Bernoulli}(\tilde{x}; 0.5),~ q(z=1\mid \tilde{x}=1) = 0.1,~ q(z=1\mid \tilde{x}=0) = 0.9.$

Then $q(z\mid x)$ is Bernoulli with probability $0.5\cdot0.1+0.5\cdot0.9=0.5$ , and

$\mathcal{L}_{\text{dvae}} = \mathbb{E}_{q(\tilde{x}\mid x)}\Big[\mathbb{E}_{q(z\mid x)}\Big[\log\frac{p(x, z)}{q(z\mid\tilde{x})}\Big]\Big]$ $= \mathbb{E}_{q(\tilde{x}\mid x)}\Big[0.5\log\frac{0.5}{q(z=1\mid\tilde{x})}+0.5\log\frac{0.5}{q(z=0\mid\tilde{x})}\Big]$ $= 0.5\Big[0.5\log\frac{0.5}{0.1}+0.5\log\frac{0.5}{0.9}\Big] + 0.5\Big[0.5\log\frac{0.5}{0.9}+0.5\log\frac{0.5}{0.1}\Big]$ $=0.510826...$

But $\log p(x)=0$ , implying $\mathcal{L}_{\text{dvae}}>\log p(x)$ . In fact, taking 0.9 in the variational parameter to be arbitrarily close to 1 makes $\mathcal{L}_{\text{dvae}}$ explode.

Recurrent Gaussian Processes

Sun, 29 Nov 2015 21:55:43 -0500

César Lincoln C. Mattos, Zhenwen Dai, Andreas Damianou, Jeremy Forth, Guilherme A. Barreto, and Neil D. Lawrence. Recurrent Gaussian Processes. arXiv preprint arXiv:1511.06644, 2015.

Summary

This is another entry into the successful consolidation of Bayesian deep learning. In the Gaussian process community of recent literature, it’s been sort of a contentious issue to try to beat the practical success of parametric functions using neural networks, with their infinite-dimensional counterpart (Neal, 1996). Gaussian processes have nice Bayesian nonparametric properties over parametric functions, and they encode a distribution over predicted values (as well as latent representations); this provides a distribution modelling uncertainty and which implicitly regularizes through Bayesian inference. As such, there is no need for many of the heuristics proposed to avoid overfitting such as dropout (which is an approximate Bayesian technique anyways). Further, these deep architectures can be applied for learning salient representations of even the smallest data sets.

The paper is very much a spiritual successor to the feedforward architecture proposed in Damianou and Lawrence (2013) for composing deep Gaussian processes. Here, Mattos et al. propose a recurrent architecture for composing recurrent (deep) Gaussian processes. It’s a simple modelling change: replace the compositions of functions used in RNN’s with compositions of Gaussian processes. Not surprisingly, GPs for time series and state space models are related (Frigola et al., 2014; Damianou and Lawrence, 2015); the only difference is that the composition of Gaussian processes in the recurrent architecture leads to a latent autoregressive structure with far more expressive power. I view this difference similarly as the difference between GPLVMs (Lawrence, 2005) and deep GPs (Damianou and Lawrence, 2013).

Inference is non-trivial, and it requires somewhat involved calculations that apply tricks from variational inference on Bayesian GPLVMs (Titsias and Lawrence, 2010) over and over. They are able to set optimal variational distributions for certain latent variables such as the inducing inputs, and generally require a mean-field approximation. They also apply an inference network using an RNN, to avoid learning the massive number of local variational parameters: each layer-wise function of the RNN is the output of the corresponding layer in the composition of variational parameters.

Discussion

This is definitely one of my favorite ICLR submissions this year. I see the main contribution of this work as solving two issues: 1. maintaining internal memory representations of the data using a recurrent architecture for GPs, and 2. increasing the scalability of deep GPs.

Any foray into sequence modelling explains why point 1 is really cool. Point 2 is a problem in the original deep GP by Damianou and Lawrence (2013), as the number of variational parameters grows linearly with the size of the data. This makes deep GPs impractical on massive data sets in which neural networks are particularly advantageous. The authors solve this issue here using an RNN as the inference network; moreover, the particular layer-by-layer application of the RNN to output each set of local variational parameters is a pretty unique setup.

On modelling, I didn’t follow the intuition behind what the shared weights in RNNs correspond to in recurrent GPs (shared kernel hyperparameters?).

The model sizes studied in the experiments are rather small, using 1 or 2 hidden layers with at most 100 inducing points. Similar to the deep GP paper, it would be insightful to study varying layers of the recurrent GP and how this impacts performance. How well the recurrent GP maintains internal memory should also be explored. For instance, one can evaluate a standard benchmark on character-to-character sequences in text.

It would be very interesting to explore GRU and LSTM versions, as well as applications in which the recurrent GP employs attention. Perhaps the latter may also avoid the need for development on convolutional GPs.

Adversarial autoencoders

Sun, 29 Nov 2015 16:19:10 -0500

Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, and Ian Goodfellow. Adversarial Autoencoders. arXiv preprint arXiv:1511.05644, 2015.

Summary

Following the trend in auto-encoders for generative modelling, Makhzani et al. propose an adversarial version using the framework of generative adversarial networks (Goodfellow et al., 2014). I have a bit of trouble following the terminology in these lines of works, so in my summary I will use different notation from the papers, following statistical conventions which I personally find more elucidating. And I’ll motivate it differently, coming from various divergence measure arguments.

I’ll first review generative adversarial networks (GANs). Suppose we observe data $x$ . GANs employ a pair of models $(G,D)$ , where $G$ is the generative model and $D$ is a discriminative model to help perform inference on $G$ .

The discriminative model $D$ aims to output high probability for samples $x$ which come from the empirical data distribution $p_{\text{data}}(x)$ , and output low probability for samples $x$ which come from the generative model $p_{\text{g}}(x)$ .
The generative model $G$ aims to fool the discriminative model, i.e., it consists of a prior $p(z)$ and likelihood $p(x\mid z)$ , and it aims to generate data most closely matching the observed data.

The objective follows a minimax game

$\min_G \max_D\, \mathbb{E}_{p_{\text{data}}(x)}[\log D(x)] + \mathbb{E}_{p_{\text{g}}(x)}[\log (1- D(x))],$

which alternates between learning the discriminator $D$ and learning the generator $G$ . The intuition is that finding the optimal $p_{\text{g}}(x)$ among a class of generative models $G$ corresponds to minimizing the Jensen-Shannon divergence

$JSD(p_{\text{data}}(x) \| p_{\text{g}}(x)) = \frac{1}{2}\mathrm{KL}\Big(p_{\text{data}} \Big\| \frac{p_{\text{data}} + p_{\text{g}}}{2}\Big) + \frac{1}{2}\mathrm{KL}\Big(p_{\text{g}} \Big\| \frac{p_{\text{data}} + p_{\text{g}}}{2}\Big).$

Minimizing $JSD$ directly is not possible; this is where the objective above, using an additional discriminative model, comes in handy. Recall also that maximum likelihood estimation minimizes $\mathrm{KL}(p_{\text{data}} \| p_{\text{g}})$ (White, 1982). Using JSD leads to a more balanced approximation not underfitting or overfitting modes. It’s also worth noting that unlike approximate posterior inference, these are divergence measures directly on data distributions for $x$ , not on latent variable distributions for $z$ . For example, variational inference minimizes $\mathrm{KL}(q(z) \| p(z\mid x))$ .

In adversarial auto-encoders, we go back to minimizing divergence measures on the latent variables. Recall that the variational auto-encoder’s objective (Kingma and Welling, 2014) is equivalently

$\min_q\, \mathbb{E}_{q(z\mid x)}[-\log p(x\mid z)] + \mathrm{KL}(q(z\mid x)\|p(z)),$

where the first term is the reconstruction error—with the decoder $p(x\mid z)$ evaluating codes from the encoder $q(z\mid x)$ —and the second term is the regularizer. The adversarial auto-encoder’s objective simply changes the regularizer to $JSD(p(z)\|q(z))$ , where we define the aggregated posterior

$q(z) = \int q(z\mid x) p_{\text{data}}(x)\mathrm{d}x.$

$JSD(p(z)\|q(z))$ is just as intractable as minimizing $JSD$ in the data space, so it requires an adversarial network: $p(z)$ replaces the original $p_{\text{data}}(x)$ and $q(z\mid x)$ replaces the original $p_{\text{g}}(x)$ :

$\min_G \max_D\, \mathbb{E}_{p(z)}[\log D(z)] + \mathbb{E}_{q(z\mid x)}[\log (1- D(z))].$

To train adversarial autoencoders, one alternates between minimizing the reconstruction error of the auto-encoder, i.e., $\mathbb{E}_{q(z\mid x)}[-\log p(x\mid z)]$ , and training the adversarial network. The rest of the paper goes into standard experiments for semi-supervised learning on MNIST and SVHN. They also comment on an extension of generative moment matching networks (Li et al., 2015) to auto-encoders, also following these ideas.

Discussion

I see generative adversarial networks as fascinating inference algorithms because they’re a rare case where we can minimize a more general divergence measure between the generative model and the data distribution. It makes sense that one should avoid MLE’s KL objective, which can lead to overdispersed distributions, in favor of this approach. And it’s great to see attempts at trying to port these same ideas to the variational inference/auto-encoder setting.

The experiments seem to show that adversarial auto-encoders lead to higher visual fidelity than variational auto-encoders. It’s not clear to me why this is the case however. The fact that there’s even a KL term as a regularizer in the VAE objective is a simple byproduct of the variational lower bound. In the end, it is still minimizing $\mathrm{KL}(q(z\mid x)\|p(z\mid x))$ . However, replacing the KL regularizer with $JSD$ makes it unclear what the underlying divergence measure is any longer. I buy that $JSD$ ensures less “holes” as $q(z)$ approximates the prior $p(z)$ , but there’s a weird conflict as $q$ simultaneously gets trained when minimizing the reconstruction error. The underlying divergence measure, if it exists, is most likely not as simple as $JSD(p(z\mid x)\|q(z\mid x))$ —although if it were then that would be fantastic(?). Does this alternating procedure also necessarily converge?

Obligatory comment in relation to hierarchical variational models (Ranganath et al., 2015): Their “universal approximator posterior”, as a possible choice for the encoder $q(z\mid x)$ , is a hierarchical variational model. To wield the intractability of the density they naively apply Monte Carlo estimates. We’ve found this not to work well in practice due to very high variance, forcing a more complicated inference procedure which we outline in our paper.

edit (12/1/15): A previous version of this post wrote $\mathrm{JSD}(p(z)\|q(z\mid x))$ instead of $\mathrm{JSD}(p(z)\|q(z))$ . Thanks to Alireza Makhzani for the correction.

Infinite Dimensional Word Embeddings

Sat, 28 Nov 2015 17:30:15 -0500

Eric Nalisnick and Sachin Ravi. Infinite Dimensional Word Embeddings. arXiv preprint arXiv:1511.05392, 2015.

Summary

Word embeddings have been huge for the NLP community ever since Tomas Mikolov’s 2013 paper (it’s gotten over 1000 citations in 2 years!). The basic idea is to learn a $z$ -dimensional parameter vector $w_i\in\mathbb{R}^z$ associated to each word in a vocabulary. This is important as part of a machine learning pipeline which extracts features for language models, text classification, etc. Not unlike advertisements of discrete Bayesian nonparametric models, Nalisnick and Ravi propose the Infinite Skip-Gram model, which allows the dimension $z$ of the word vectors to grow arbitrarily.

Letting word vector $w_i\in\mathbb{R}^\infty$ , context vector $c_k\in\mathbb{R}^\infty$ , and random positive integer $z\in\mathbb{Z}^+$ , they consider multinomial logistic regression with (Eq.3)

$p(w_i,c_k,z) = \frac{1}{Z} e^{-E(w_i,c_k,z)},$

where $Z$ is the partition function summing over all $w_i,c_k,z$ . This is the same as the original model for word embeddings, except now there is an auxiliary variable $z$ and the vectors are infinite-dimensional. This seems intractable given some energy function $E(w_i,c_k,z)$ , as the partition function requires a summation over countably infinite values of $z$ . However, similar to Côté and Larochelle (2015), they show that the following penalized energy function enables finite computation (Eq.4):

$E(w_i,c_k,z) = -\sum_{j=1}^z w_{i,j} c_{k,j} + \Big[z\log a -\sum_{j=1}^z (w_{i,j}^2 + c_{k,j}^2)\Big].$

The first term is the original (unpenalized) energy function, which is the inner product of $z$ -dimensional vectors. The second term, the penalty, consists of two expressions: the $L_2$ penalty forces the word and context vectors to be zero after sufficiently large index, and the $z\log a$ term forces a convergent geometric series for the denominator of the conditional distribution $p(z\mid w, c)$ (Eq.5). Tractability of the conditional distribution is required during optimization.

For inference, they treat $z$ as a nuisance parameter and perform an approximate maximum likelihood estimation. Specifically, they minimize an upper bound to the negative log-likelihood $\mathcal{L}=-\log \int p(c,z\mid w)~\mathrm{d}z$ over the word vectors $w$ . Similar to how word embeddings require negative sampling for computational efficiency, Nalisnick and Ravi make additional Monte Carlo approximations, such as evaluating Monte Carlo estimates of the gradient of (an upper bound to) this objective.

Discussion

Very cool idea! I’m always a fan of infinite-dimensional models, not just because they’re mathematically interesting but because there’s no reason that word vectors should be restricted to a fixed dimensionality. In a streaming setting, for example, the word representations should grow increasingly more complex as more information in the data is available.

Although they don’t explicitly state this, what they’re really doing is applying a variational lower bound for inference, invoking Jensen’s inequality and setting the variational distribution as the prior $~q(z)= p(z\mid w_i)$ (Eq.7). If readers are familiar with the score function estimator used for variational inference (Paisley et al., 2012), the Monte Carlo gradient in Eq.10 applies the same trick. Following this interpretation, it surprises me then that the algorithm works so well. Using model priors for variational approximations make for quite a loose lower bound of the log-likelihood.

I highly recommmend reading this paper. It’s very well-written and easy to read even if you don’t know anything about word embeddings. The related work section places it very appropriately in context. They mention Vilnis and McCallum (2015) a lot, and I’d be very interested to see if they can extend their work to inferring infinite-dimensional distributions of words rather than infinite-dimensional point-estimates of words.

Neural Variational Inference for Text Processing

Sat, 28 Nov 2015 04:48:57 -0500

Yishu Miao, Lei Yu, Phil Blunsom. Neural Variational Inference for Text Processing. arXiv preprint arXiv:1511.06038, 2015.

Summary

Following recent work on variational auto-encoders and their advances in computer vision, the authors propose deep generative models and related inference algorithms for text. The Neural Variational Document Model (NVDM) is an instantiation of the variational auto-encoder for deep latent Gaussian models, where the data input is the bag-of-words representation of the document. There’s a cool interpretation of this, as the Gaussian latent variables can be thought of as the embedding of the discrete vocabulary in a real-valued latent space. In the scenario of multiple hidden layers, the deep latent Gaussian model can be seen as sequentially embedding each corresponding value within nested latent spaces.

As a more complex model, they also propose the Neural Answer Selection Model (NASM) for answer sentence selection, in which the task is to identify the correct answers to a question from a set of possible answers. The NASM employs a latent attention model similar to the deep recurrent attentive writer (Gregor et al., 2015) for modelling images. Given a question $q$ , it attends a set of answer sentences $~\{a_1,\ldots,a_n\}$ associated with $q$ ; this in turn determines the context vector, which are the words in the answer sentences that are prominent for predicting the answer matches to the current question. This enables the model to learn subtleties inherent in the questions.

Discussion

It’s a simple idea coming from the variational auto-encoder literature and their recent state-of-the-art advances for unsupervised learning in computer vision. This is well worth studying. Interestingly, there doesn’t seem to be any complications when transferring these same ideas to analyzing text.

Confusingly, it doesn’t apply neural variational inference from Mnih and Gregor (2014), as the generative model is continuous. Instead it uses the more common tricks standard in variational auto-encoders.

It would have been interesting to study a generative model of documents which also employs a latent attention architecture. Most immediate in my mind would be avoiding the bag-of-words representation and explicitly modelling the word-to-word sequence of the documents.