Problem source

Let $R_{\alpha}$ be the $2\times2$ matrix that represents a rotation through the
angle $\alpha$ and let
$$A=\begin{pmatrix}a&b\\b&c\end{pmatrix}.$$
\begin{questionparts}
\item Find in terms of $a$, $b$ and $c$ an angle $\alpha$ such that $R_{-\alpha}AR_{\alpha}$ is a diagonal matrix (i.e. has the value zero in top-right and bottom-left positions).
\item Find values of $a$, $b$ and $c$ such that the equation of the ellipse
\[x^2+(y+2x\cot2\theta)^2=1\qquad(0 < \theta < \tfrac{1}{4}\pi)\]
can be expressed in the form
\[\begin{pmatrix}x&y\end{pmatrix}A\begin{pmatrix}x\\y\end{pmatrix}=1.\]
Show that, for this $A$,  $R_{-\alpha}AR_{\alpha}$ is  diagonal if $\alpha=\theta$. Express the non--zero elements of this matrix in terms of $\theta$.
\item Deduce, or show otherwise,  that the minimum and maximum distances from the centre to the circumference of this ellipse are $\tan\theta$ and $\cot\theta$.
\end{questionparts}

Solution source

\begin{questionparts}
\item \begin{align*}
R_{-\alpha}AR_{\alpha} &= \begin{pmatrix} \cos \alpha & \sin\alpha \\ -\sin \alpha & \cos \alpha \end{pmatrix}\begin{pmatrix} a & b \\ b & c \end{pmatrix} \begin{pmatrix} \cos \alpha & -\sin\alpha \\ \sin \alpha & \cos \alpha \end{pmatrix} \\
&=  \begin{pmatrix} \cos \alpha & \sin\alpha \\ -\sin \alpha & \cos \alpha \end{pmatrix} \begin{pmatrix} a\cos \alpha + b \sin \alpha & -a\sin\alpha + b \cos\alpha \\
b\cos\alpha + c \sin\alpha  & c\cos\alpha-b\sin\alpha
\end{pmatrix} \\
&= \begin{pmatrix} a\cos^2\alpha+2b\sin\alpha\cos\alpha+c\sin^2\alpha & -a\sin\alpha\cos \alpha+b\cos^2\alpha +c\sin\alpha\cos\alpha-b\sin^2 \alpha\\ (c-a)\sin\alpha\cos \alpha +b(\cos^2\alpha-\sin^2 \alpha) & a\sin^2 \alpha -2b\sin\alpha\cos\alpha+c\cos^2\alpha \end{pmatrix} \\
&= \begin{pmatrix} * & \frac{c-a}{2}\sin2\alpha+b \cos 2\alpha\\\frac{c-a}{2}\sin2\alpha+b \cos 2\alpha & * \end{pmatrix} 
\end{align*}

Therefore this will be diagonal if $\tan 2\alpha = \frac{2b}{a-c} \Rightarrow \alpha = \frac12 \tan^{-1} \l \frac{2b}{a-c} \r$

\item 

\begin{align*}
x^2+(y+2x\cot2\theta)^2 &= x^2(1 + 4\cot^22\theta) + 4\cot2\theta xy + y^2 \\
&= \begin{pmatrix}x&y\end{pmatrix}\begin{pmatrix} 1 + 4\cot^22\theta & 2\cot 2\theta \\ 2\cot 2\theta & 1  \end{pmatrix}\begin{pmatrix}x\\y\end{pmatrix}
\end{align*}

Plugging this $\mathbf{A}$ in our result from before we discover
\begin{align*}
\frac12 \tan^{-1} \l \frac{2b}{a-c} \r &= \frac12 \tan^{-1} \l \frac{4\cot 2\theta}{1 + 4\cot^22\theta-1} \r \\
&= \frac12 \tan^{-1} \l  \tan 2 \theta \r \\
&= \theta
\end{align*}

Therefore, the matrix will be:

\begin{align*}
& \textrm{diag}\begin{pmatrix}
(1+4\cot^2 2\theta)\cos^2 \theta + 4\cot2\theta \sin\theta\cos\theta + \sin^2 \theta  \\
(1+4\cot^2 2\theta)\sin^2 \theta - 4\cot2\theta \sin\theta\cos\theta + \cos^2 \theta
\end{pmatrix} \\
=& \textrm{diag}\begin{pmatrix} \cos^2\theta + \frac{\cos^2 2\theta}{\sin^2 \theta} + 2\cos 2\theta + \sin^2 \theta \\
\sin^2\theta + \frac{\cos^2 2\theta}{\cos^2 \theta} - 2\cos 2\theta + \cos^2 \theta
\end{pmatrix} \\
=& \textrm{diag}\begin{pmatrix} 1 + \cos 2\theta \l \frac{\cos2\theta}{\sin^2 \theta} + 2\r \\
1 + \cos 2\theta \l \frac{\cos2\theta}{\cos^2 \theta} - 2\r \\ \end{pmatrix} \\
=& \textrm{diag}\begin{pmatrix} 1 + \cos 2\theta \l \frac{\cos^2 \theta + \sin^2 \theta}{\sin^2 \theta}\r \\
 1 -\cos 2\theta \l \frac{-\cos^2 \theta - \sin^2 \theta}{\cos^2 \theta}\r \\ \end{pmatrix} \\
=& \textrm{diag}\begin{pmatrix} 1 + (\cos^2\theta - \sin^2 \theta) \cosec^2 \theta \\
 1 - (\cos^2\theta - \sin^2 \theta)  \sec^2 \theta \\ \end{pmatrix} \\
=& \textrm{diag}\begin{pmatrix} \cot^2 \theta \\
\tan^2 \theta \\ \end{pmatrix} \\
\end{align*}

Therefore this is a rotation of an ellipse with equation:

$(\cot \theta x)^2 + (\tan \theta y)^2 = 1$, ie the shortest side and longest side are $\cot \theta$ and $\tan \theta$ respectively, but we know since $0  < \theta < \tfrac{1}{4}\pi$ the shortest will be $\tan \theta$ and the longest $\cot \theta$.