2025 USA-NA-AIO Round 2, Problem 2, Part 11

AI Olympiads 2025 USA-NA-AIO Round 2
1

Part 11 (10 points, non-coding task)

So far, we have proved that GQA can always be represented by MLA.

In this part, you are asked to prove that GQA is not equivalent to MLA. What you need to do is to find one example that MLA cannot be represented as GQA.

To be specific, please do the following things:

  1. Construct \color{red}{\mathbf{W}^{\mathbf{DKV}, MLA}} \in \Bbb R^{1 \times 2}.

  2. Construct \color{blue}{\mathbf{W}^{\mathbf{UM}, MLA}} \in \Bbb R^{2 \times 1}.

  3. Do matrix multiplication \color{blue}{\mathbf{W}^{\mathbf{UM}, MLA}} \color{red}{\mathbf{W}^{\mathbf{DKV}, MLA}}.

  4. Show that this product matrix is not the concatenation of two copies of 1-by-2 matrices along axis 0.

2

\color{green}{\text{### WRITE YOUR SOLUTION HERE ###}}

Define

\color{red}{\mathbf{W}^{\mathbf{DKV}, MLA}} = \begin{bmatrix} 1 & 2 \end{bmatrix}

and

\color{blue}{\mathbf{W}^{\mathbf{UM}, MLA}} = \begin{bmatrix} 3 \\ 4 \end{bmatrix}

Hence,

\begin{align*} \color{blue}{\mathbf{W}^{\mathbf{UM}, MLA}} \color{red}{\mathbf{W}^{\mathbf{DKV}, MLA}} & = \begin{bmatrix} 3 & 6 \\ 4 & 8 \end{bmatrix} . \end{align*}

Two rows of this product matrix are not identical.

Therefore, this is an example that MLA cannot always be represented by GQA.

\color{red}{\text{""" END OF THIS PART """}}