1 Introduction

Given samples $(x,y)$ from a distribution with probability density function $p(x,y)$, the optimization goal of a classification problem or a regression problem is to find a good $p_{𝜃}\left(y\right|x)$ where $𝜃$ is the parameter of a chosen family of probability density functions. The objective can be derived in three different but related ways.

1.1 K-L divergence of conditional distribution

One criterion of a good $p_{𝜃}\left(y\right|x)$ is how close it is to $p\left(y\right|x)$. One such closeness measure is the K-L divergence between $p\left(y\right|x)$ and $p_{𝜃}\left(y\right|x)$ which is $\mathrm{\int \; <!--nolimits-->}<!--FUNCTION\; APPLICATION-->p\left(y\right|x)\log <!--FUNCTION\; APPLICATION-->\frac{p\left(y\right|x)}{p_{𝜃}\left(y\right|x)}dy$ . Of course, this should work across all $x$, therefore our objective should be

For our purpose, the first term is an unknown consant independent of $𝜃$. Removing this constant, our objective changes to

The left side of the above equation can also be written as:

So basically, the optimization objective is the following three equivalent functions:

1.2 K-L divergence of joint distribution

Since $p_{𝜃}(x,y)=p\left(x\right)p_{𝜃}(x,y)$, it is easy to arrive at the same conclusions by minimizing the K-L divergence between $p(x,y)$ and $p_{𝜃}(x,y)$:

Again, the first term is an unknown constant independent of $𝜃$ that can be removed. The objective changes to

1.3 Maximum likelihood

Given a set of samples $(x_i,y_i)$, assumed to be i.i.d, one objective could be to maximize the likelihood of observing these samples, which is

This is equivalent to minimizing the negative log likelihood

As before, the first term is an unknown constant independent of $𝜃$. Once the first term is removed, the ojective becomes

Divide it by the number of samples, and rewirte it in expectation form, the objective becomes

This is the same as what is derived in Section 1.1 and Section 1.2.

2 Classification

In a classification problem, $y$ takes on a fixed number of possible values usually encoded using numbers from 1 through $K$. A classifier usually outputs the entire probability vector $p_{𝜃}(y=1|x),p_{𝜃}(y=2\left|x\right),p_{𝜃}(y=3|x),\cdots ,p_{𝜃}(y=K\left|x\right)$. In the case of a binary classification problem, however, it is more customary to use $\{0,1\}$ to encode the two possible values that $y$ can take, and the classifier only outputs $f\left(x\right)=p_{𝜃}(y=1|x)$ with $p_{𝜃}(y=0|x)$ implied to be $1-f\left(x\right)$. In this case, the optimization objective can be rewritten as

This is usual called the binary cross entropy objective.

3 Regression

In a regression problem, a neural network’s output can be interpreted as the mean ${\mu }_{𝜃}\left(x\right)$ of a normal distribution $N\left({\mu }_{𝜃}\right(x),I)$. With this interpretation, the optimization objective can be rewritten as

where $d$ is the dimension of $y$. This objective is equivalent to

which is the well known mean squared error objective.