Problem source

Verify that if 
\[
\mathbf{P}=\begin{pmatrix}1 & 2\\
2 & -1
\end{pmatrix}\qquad\mbox{ and }\qquad\mathbf{A}=\begin{pmatrix}-1 & 8\\
8 & 11
\end{pmatrix}
\]
then $\mathbf{PAP}$ is a diagonal matrix. 
Put $\mathbf{x}=\begin{pmatrix}x\\
y
\end{pmatrix}$ and $\mathbf{x}_{1}=\begin{pmatrix}x_{1}\\
y_{1}
\end{pmatrix}.$ By writing 
\[
\mathbf{x}=\mathbf{P}\mathbf{x}_{1}+\mathbf{a}
\]
for a suitable vector $\mathbf{a},$ show that the equation 
\[
\mathbf{x}^{\mathrm{T}}\mathbf{Ax}+\mathbf{b}^{\mathrm{T}}\mathbf{x}-11=0,
\]
where $\mathbf{b}=\begin{pmatrix}18\\
6
\end{pmatrix}$ and $ \mathbf{x}^{\mathrm{T}} $ is the transpose of $\mathbf{x},$
becomes 
\[
3x_{1}^{2}-y_{1}^{2}=c
\]
for some constant $c$ (which you should find).

Solution source

\begin{align*}
\mathbf{PAP} &= \begin{pmatrix}1 & 2\\
2 & -1
\end{pmatrix}\begin{pmatrix}-1 & 8\\
8 & 11
\end{pmatrix}\begin{pmatrix}1 & 2\\
2 & -1
\end{pmatrix} \\
&= \begin{pmatrix}1 & 2\\
2 & -1
\end{pmatrix}\begin{pmatrix}15 & -10\\
30 & 5
\end{pmatrix} \\
&= \begin{pmatrix}75 & 0\\
0 & -25
\end{pmatrix}
\end{align*}
Which is diagonal as required.

Letting $\mathbf{x}=\mathbf{P}\mathbf{x}_{1}+\mathbf{a}$

\begin{align*}
&& \mathbf{x}^{\mathrm{T}}\mathbf{Ax}+\mathbf{b}^{\mathrm{T}}\mathbf{x}-11&=0 \\
\Leftrightarrow && (\mathbf{P}\mathbf{x}_{1}+\mathbf{a})^T\mathbf{A}(\mathbf{P}\mathbf{x}_{1}+\mathbf{a}) + \mathbf{b}^T(\mathbf{P}\mathbf{x}_{1}+\mathbf{a}) - 11 &= 0 \\
\Leftrightarrow && \mathbf{x}_{1}^T\mathbf{PAP}\mathbf{x}_1 + \mathbf{x}_{1}^T\mathbf{PAa} + \mathbf{a}^T\mathbf{AP}\mathbf{x}_{1}+\mathbf{a}^T\mathbf{Aa} + \mathbf{b}^T(\mathbf{P}\mathbf{x}_{1}+\mathbf{a}) - 11 &= 0 \\
\Leftrightarrow && \mathbf{x}_{1}^T\mathbf{PAP}\mathbf{x}_1 +(2\mathbf{a}^T\mathbf{A}+\mathbf{b}^T)\mathbf{P}\mathbf{x}_{1}+\mathbf{a}^T\mathbf{Aa} + \mathbf{b}^T\mathbf{a} - 11 &= 0 \\
\end{align*}

It would be nice if we picked $\mathbf{a}$ such that $2\mathbf{a}^T\mathbf{A}+\mathbf{b}^T = 0$, if $\mathbf{a} = \begin{pmatrix} a_1 \\a_2 \end{pmatrix}$ then this equation becomes:

\begin{align*}
&& 2\begin{pmatrix}-a_1 + 8a_2 & 8a_1+11a_2 \end{pmatrix} + \begin{pmatrix}18 & 6 \end{pmatrix} &= 0 \\
\Rightarrow && a_1 = 1, a_2 = -1
\end{align*}

So our equation is now

\begin{align*}
&& \mathbf{x}_{1}^T\mathbf{PAP}\mathbf{x}_1 +(2\mathbf{a}^T\mathbf{A}+\mathbf{b}^T)\mathbf{P}\mathbf{x}_{1}+\mathbf{a}^T\mathbf{Aa} + \mathbf{b}^T\mathbf{a} - 11 &= 0 \\
\Leftrightarrow && \mathbf{x}_{1}^T\mathbf{PAP}\mathbf{x}_1-6 +12 - 11 &= 0 \\
\Leftrightarrow && 25(3x_1^2 - y_1^2) &= 5 \\
\Leftrightarrow && 3x_1^2 - y_1^2 &= \frac{1}{5}
\end{align*}