Distributing points on the sphere as to minimize the mean square error

of the quadrature formula , where is a centered Gaussian process with covariance function . Shown is .

Skip to content
# Math. Stat. Notes

## Expository notes and skywriting.

# Category: Bayesian statistics

# Bayesian numerical analysis

# Bayesian binary classification using partitions of unity

# 1. The problem

# Bayesian learning

# The discretization trick

# Posterior consistency under (possible) misspecification

# The choice of prior in bayesian nonparametrics – part 2

# The structure of

## Metrics on

# Remark on the asymptotics of the likelihood ratio and the K.-L. divergence

**The problem.**

**Better measures of divergence?**

# The choice of prior in bayesian nonparametrics – Introduction

# 1. Introduction

## 1.1 Learning and uncertainty

## 1.2 The choice of prior

# Constrained semiparametric modelling (for directional statistics)

# Explanations

### Trigonometric density basis

### Prior specification

### Approximation-theoric framework

# Comment on The Sample Size Required in Importance Sampling

# Example

Sunday afternoon project. I think it is possible to get topological garanties for the reconstruction of the classification boundary using the Nash-Tognoli theorem.

Points , are distributed on some space and associated a label . It is assumed that

where is some unknown integrable function. Estimating from the data allows us to predict given .Read More »

**Friday july 28 at 17:00**

**Rutherford Physics Building, Room 118, McGill**

Next week, I’ll be talking about *Bayesian learning* at the Mathematical congress of the americas and at the Canadian undergraduate mathematics conference. These are somewhat challenging talks: I need to sell the idea of Bayesian statistics to a general mathematical audience (which knows nothing about it), interest them in some though problems of Bayesian nonparametrics, and then present some of our research results. This must be done in under 20 minutes.

To make the presentation more intuitive and accessible, I borrowed some language from machine learning. I’m talking about learning rather than inference, uncertain knowledge rather than subjective belief, and “asymptotic correctness” rather than consistency. These are essentially synonymous, although some authors might use them in different ways. This should not cause problems for this introductory talk.Read More »

*I explain the discretization trick that I mentioned in my previous post (Posterior consistency under possible misspecification). I think it was introduced by Walker (New approaches to Bayesian consistency (2004)).*

Let be a set of densities and let be a prior on . If follows some distribution having density , then the posterior distribution of can be written as

The discretization trick is to find densities in the convex hull of (taken in the space of all densities) such that

We assume, without too much loss of generality, that our priors are discrete. When dealing with Hellinger separable density spaces, it is possible to discretize posterior distributions to study consistency (see this post about it).

Let be a prior on a countable space of probability density functions, with for all . Data follows (independently) some unknown distribution with density .

We denote by the Kullback-Leibler divergence and we let be half of the squared Hellinger distance.

The following theorem states that the posterior distribution of accumulates in Hellinger neighborhoods of , assuming the prior is root-summable (i.e. for some ) . In the well-specified case (i.e. ), the posterior accumulates in any neighborhood of . In the misspecified case, small neighborhoods of could be empty, but the posterior distribution still accumulates in sufficiently large neighborhoods (how large exactly is a function of and ).Read More »

See part 1. Most proofs are omitted; I’ll post them with the complete pdf later this week.

Recall that is is a Polish space (ie. a complete and separable metric space). It is endowed with its borel -algebra which is the smallest family of subsets of that contains its topology and that is closed under countable unions and intersections. All subsets of we consider in the following are supposed to be part of . A probability measure on is a function such that for any countable partition of we have that The set consists of all such probability measures.

Note that since is complete and separable, every probability measure is *regular* (and *tight*). It means that the measure of any can be well approximated from the measure of compact subsets of as well as from the measure of open super-sets of :

Let me review some facts. A natural metric used to compare the mass allocation of two measures is the *total variation distance* defined by

Let be three densities and suppose that, , , independently. What happens to the likelihood ratio

as ?

Clearly, it depends. If , then

almost surely at an exponential rate. More generally, if is closer to than to , in some sense, we’d expect that . Such a measure of “closeness” of “divergence” between probability distributions is given by the Kullback-Leibler divergence

It can be verified that with equality if and only if , and that

almost surely at an exponential rate. Thus the K.L.-divergence can be used to solve our problem.

There are other measures of divergence that can determine the asymptotic behavior of the likelihood ratio as in (e.g. the discrete distance). However, in this note, I give conditions under which the K.-L. divergence is, up to topological equivalence, the “best” measure of divergence.Read More »

*In preparation for the 11th Bayesian nonparametrics conference, I’m writing (and rewriting) notes on the background of our research (i.e. some of the general theory of bayesian nonparametrics). There are some good books on the subject (such as *Bayesian Nonparametrics* (Ghosh and Ramamoorthi, 2003)), but I wanted a more introductory focus and to present Choi and Ramamoorthi’s very clear point of view on posterior consistency (Remarks on the consistency of posterior distributions, 2008). *

Let be a complete and separable metric space and let be the space of all probability measures on . Some unknown distribution is generating observable data , where each is independently drawn from . The problem is to *learn about* using only and prior knowledge.

**Example** (Discovery probabilities).

A cryptographer observes words, following some distribution , in an unknown countable language . What are the -probabilities of the words observed thus far? What is the probability that the next word to be observed has never been observed before?

We need an employable definition of learning. As a first approximation, we can consider learning to be the reduction of uncertainty about what is . This requires a quantification of how uncertain we are to begin with. Then, hopefully, as data is gathered out uncertainty decreases and we are able to pinpoint .

This is the core of *Bayesian learning*, alghough our definition is not yet entirely satisfactory. There are some difficulties with this idea of quantifying uncertainty, at least when using information-theoric concepts. The solution we adopt here is the use of probabilities to quantify *uncertain knowledge* (bayesians would also talk of subjective probabilities quantifying rational belief). For example, you may know that a coin flip is likely to be fair, although it is not impossible the two sides of the coin are both the same. This is uncertain knowledge about the distribution of heads and tails in the coin flips, and you could assign probabilities to the different possibilities.

More formally, prior uncertain knowledge about what is is quantified by a probability measure on . For any , is the the prior probability that ““. Then, given data , prior probabilities are adjusted to posterior probabilities: becomes , the conditional distribution of given . The celebrated Bayes’ theorem provides a formula to calculate from and . Thus we have an operational definition of learning in our statistical framework.

* Learning is rationally adjusting uncertain knowledge in the light of new information.*

For explanations as to why probabilities are well suited to the representation of uncertain knowledge, I refer the reader to Pearl (*Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference*, 1988). We will also see that the operation of updating the prior to posterior posterior probabilities does work as intended.

Specifying prior probabilities, that is quantifying prior uncertain knowledge, is not a simple task. It is especially difficult when uncertainty is over the non-negligeable part of an infinite dimensional vector space. Fortunately, “probability is not about numbers, it is about the structure of reasoning”, as Glenn Shafer puts it (cited in Pearl, 1988, p.15). The exact numbers given to the events “” are not of foremost importance; what matters is how probabilities are more qualitatively put together, and how this relates to the learning process.

Properties of prior distributions, opening them to scrutiny, criticism and discussion, must be identified and related to what happens as more and more data is gathered.

Angular data arises in many scientific fields, such as in experimental biology for the study of animal orientation, and in bioinformatics in relation to the protein structure prediction problem.

The statistical analysis of this data requires adapted tools such as -periodic density models. Fernandez-Duran (*Biometrics, *60(2), 2004) proposed non-negative trigonometric sums (i.e. non-negative trigonometric polynomials) as a flexible family of circular distributions. However, the coefficients of trigonometric polynomials expressed in the standard basis are difficult to interpret and we do not see how an informative prior could be specified through this parametrization. Moreover, the use of this basis was criticized by Ferreira et al. (*Bayesian Analysis*, 3(2), 2008) as resulting in a “wigly approximation, unlikely to be useful in most real applications”.

Here, we suggest the use of a *density* basis of the trigonometric polynomials and argue it is well suited to statistical applications. In particular, coefficients of trigonometric densities expressed in this basis possess an intuitive geometric interpretation. Furthermore, we show how “wiggliness” can be precisely controlled using this basis and how another geometric constraint, periodic unimodality, can be enforced *[first proposition on the poster]*. To ensure that nothing is lost by using this basis, we also show that the whole model consists of precisely all positive trigonometric densities, together with the basis functions *[first theorem on the poster]*.

Priors can be specified on the coefficients of mixtures in our basis and on the degree of the trigonometric polynomials to be used. Through the interpretability of the coefficients and the *shape-preserving properties* of the basis, different types of prior knowledge may be incorporated. Together with an approximate understanding of mass allocation, these include:

- periodic unimodality;
- bounds on total variation; and
- knowledge of the marginal distributions (in the multivariate case).

The priors obtained this way are part of a well-studied family called *sieve priors, *including the well-known Bernstein-Dirichlet prior*,* and are *finite mixtures with an unknown number of components*. Most results and interpretations about the Bernstein-Dirichlet prior (see Petrone & Wasserman (*J. R. Stat. Soc. B.*, 64(1), 2002), Kruijer and Van der Vaart (*J. Stat. Plan. Inference*, 138(7), 2008), McVinish et al. (*Scand. J. Statist.*, 36(2), 2009) can carry over to the priors we consider, but we dot not discuss them further.

Our density models arise as the image of* “shape-perserving” linear approximation* operators. This approximation-theoric relationship is used to obtain a notably large prior Kullback-Leibler support and ensures strong posterior consistency at all bounded (not necessarily continuous) density. The result partly relies on known properties of sieve priors, as well as general consistency results (Walker (*Ann. Statist*., 32(5), 2004)), but extends known result by removing an usual continuity hypothesis on the densities at which consistency is achieved (see Wu & Ghosal (*Electron. J. Stat*., 2, 2008), Petrone & Veronese (*Statistica Sinica*, 20, 2010)). For contraction rates, higher order smoothness conditions are usually required (see Shen & Ghosal (*Scand. J. Statist*., 42(4), 2015)).

For example, consider the prior induced by the random density

where is a Dirichlet process, is distributed on and is a partition of the circle. It has the strong posterior consistency at all bounded density provided that the associated operator

is such that for all continuous .

More generally, let be a set of bounded densities on some compact metric space , let , , be a sequence of operators that are:

- shape preserving: maps densities to densities and ; and
- approximating: for all continuous ;

and finally let be priors on with full support. A sieve prior on is defined by

.

**Theorem.**

* If for some increasing sequence bounding the dimensions of , then the posterior distribution of is strongly consistent at each density of .*

The approximation theory literature is rich in such operators. The theorem shows that they provide strongly consistent priors on arbitrary density spaces simply given priors on .

**Basic density estimation:**

*A thousand samples (grey histogram) were drawn from the density in orange. The prior is defined by (1) with the Dirichlet process centered on the uniform density and with a precision parameter of 2. The degree is distributed as a . The blue line is the posterior mean, the dark blue shaded region is a 50% pointwise credible region around the median, and the light blue shaded region is a 90% credible region.*

*I summarize and comment part of The Sample Size Required in Importance Sampling (Chatterjee and Diaconis, 2015). One innovative idea is to bound the mean estimation error in terms of the tail behavior of , where and are the importance sampling target and proposal distributions, respectively.*

The problem is to evaluate

where is a probability measure on a space and where is measurable. The Monte-Carlo estimate of is

When it is too difficult to sample , for instance, other estimates can be obtained. Suppose that is absolutely continuous with respect to another probability measure , and that the density of with respect to is given by . Another unbiaised estimate of is then

This is the general framework of importance sampling, with the Monte-Carlo estimate recovered by taking . An important question is the following.

*How large should be for to be close to ?*

An answer is given, under certain conditions, by Chatterjee and Diaconis (2015). Their main result can be interpreted as follows. If and if is concentrated around its expected value , then a sample size of approximately is both necessary and sufficient for to be close to . The exact sample size needed depends on and on the tail behavior of . I state below their theorem with a small modification.

**Theorem 1.** (Chatterjee and Diaconis, 2015)

*As above, let . For any and ,*

*Conversely, for any and ,*

**Remark 1.**

Suppose and that is concentrated around , meaning that for some we have that and are both less than an arbitrary . Then, taking we find

However, if , we obtain

meaning that there can be a high probability that and are not close.

**Remark 2.**

Let , so that . In that case, only takes its expected value . The theorem yields

and no useful bound on .

**Comment.**

For the theorem to yield a sharp cutoff, it is necessary that be relatively large and that be highly concentrated around . The first condition is not aimed at in the practice of importance sampling. This difficulty contrasts with the broad claim that “a sample of size approximately is necessary and sufficient for accurate estimation by importance sampling”. The result in conceptually interesting, but I’m not convinced that a sharp cutoff is common.

I consider their example 1.4. Here is the exponential distribution of mean , is the exponential distribution of mean 2, and . Thus . We have , meaning that the theorem yields no useful cutoff. Furthermore, and . Optimizing the bound given by the theorem yields

The figure below shows trajectories of . The shaded area bounds the expected error.

This next figure shows trajectories for the Monte-Carlo estimate of , taking and . Here the theorem yields

**References.**

Chatterjee, S. and Diaconis, P. The Sample Size Required in Importance Sampling. https://arxiv.org/abs/1511.01437v2