USAAIO
1
Part 20 (5 points, non-coding task)
In the loss function, we introduced a crutial learnable parameter \tau, called temperature.
Let us explore some properties of \tau.
Let z_0 > z_1 > \cdots > z_{N-1}.
Define
f_i = \frac{\exp \left( z_i / \tau \right)}
{\sum_{j = 0}^{N-1} \exp \left( z_j / \tau \right)} .
Do the following analysis. Reasoning is required.
- Compute
\lim_{\tau \rightarrow 0^+} f_i .
- Compute
\lim_{\tau \rightarrow \infty} f_i .
USAAIO
2
\color{green}{\text{### WRITE YOUR SOLUTION HERE ###}}
- We have
\begin{align*}
\lim_{\tau \rightarrow 0^+} f_i
& = \lim_{\tau \rightarrow 0^+} \frac{\exp \left( z_i / \tau \right)}
{\sum_{j = 0}^{N-1} \exp \left( z_j / \tau \right)} \\
& = \lim_{\tau \rightarrow 0^+} \frac{\exp \left( \left( z_i - z_0 \right) / \tau \right)}
{\sum_{j = 0}^{N-1} \exp \left( \left( z_j - z_0 \right) / \tau \right)} \\
& = \boxed{\begin{cases}
1 & \mbox{if } i = 0 \\
0 & \mbox{if } i \neq 0
\end{cases}} .
\end{align*}
- Compute
\begin{align*}
\lim_{\tau \rightarrow \infty} f_i
& = \lim_{\tau \rightarrow \infty} \frac{\exp \left( z_i / \tau \right)}
{\sum_{j = 0}^{N-1} \exp \left( z_j / \tau \right)} \\
& = \frac{1}{\sum_{j = 0}^{N-1} 1} \\
& = \boxed{\frac{1}{N}} .
\end{align*}
\color{red}{\text{""" END OF THIS PART """}}