Problem source

The linear transformation $\mathrm{T}$ is a shear which transforms a point $P$ to the point $P'$ defined by 
\begin{enumerate}
\item $\overrightarrow{PP'}$ makes an acute angle $\alpha$ (anticlockwise)
with the $x$-axis, 
\item $\angle POP'$ is clockwise (i.e. the rotation from $OP$ to $OP'$
clockwise is less than $\pi),$ 
\item $PP'=k\times PN,$ where $PN$ is the perpendicular onto the line
$y=x\tan\alpha,$ where $k$ is a given non-zero constant. 
\end{enumerate}
If $\mathrm{T}$ is represented in matrix form by $\begin{pmatrix}x'\\
y'
\end{pmatrix}=\mathbf{M}\begin{pmatrix}x\\
y
\end{pmatrix},$ show that 
\[
\mathbf{M}=\begin{pmatrix}1-k\sin\alpha\cos\alpha & k\cos^{2}\alpha\\
-k\sin^{2}\alpha & 1+k\sin\alpha\cos\alpha
\end{pmatrix}.
\]
Show that the necessary and sufficient condition for $\begin{pmatrix}p & q\\
r & t
\end{pmatrix}$ to commute with $\mathbf{M}$ is 
\[
t-p=2q\tan\alpha=-2r\cot\alpha.
\]

Solution source

\begin{tikzpicture}[scale=1.5]

    \coordinate (O) at (0,0);
    \coordinate (X) at (3,0);
    \coordinate (Y) at (3,1);
    
    \draw[->] (-3, 0) -- (3,0);
    \draw[->] (0, -3) -- (0, 3);

    
    
    % \draw[line width=2.5pt] (0,0) -- (12, 2);
    % \draw[dashed] (0,0) -- (12,12);

    \node at (3,0) [right] {$x$};
    \node at (0,3) [above] {$y$};

    \draw[dashed] (-3,-1) -- (3,1);
    \draw[dashed] (-1,3) -- (1,-3);

    \pic [draw, angle radius=1.8cm, "$\alpha$"] {angle = X--O--Y};

    \draw (0,0) -- ({1},{1/3}) -- ({2/3},{4/3}) -- ({-1/3},{1}) -- cycle;
    \draw (0,0) -- ({1},{1/3}) -- ({3/2-1/3},{7/6+1/3}) -- ({-1/3+1/2},{1+1/6}) -- cycle;
\end{tikzpicture}

We can see that $\mathbf{M}$ sends $\begin{pmatrix} 1 \\ \tan \alpha \end{pmatrix}$ to itself, and $\begin{pmatrix} -\tan \alpha \\ 1 \end{pmatrix}$ to $\begin{pmatrix} -\tan \alpha \\ 1 \end{pmatrix} + k \begin{pmatrix} 1 \\ \tan \alpha \end{pmatrix}$

Therefore, we have:

\begin{align*}
&& \mathbf{M} \begin{pmatrix} 1 & -\tan \alpha \\ \tan \alpha & 1 \end{pmatrix} &= \begin{pmatrix} 1 & k - \tan \alpha  \\ \tan \alpha & 1 + k\tan \alpha \end{pmatrix} \\
&& \sec \alpha \mathbf{M} \begin{pmatrix} \cos \alpha  & -\sin\alpha \\ \sin \alpha & \cos \alpha \end{pmatrix} &= \begin{pmatrix} 1 & k - \tan \alpha  \\ \tan \alpha & 1 + k\tan \alpha \end{pmatrix} \\
\Rightarrow && \mathbf{M} &= \cos \alpha\begin{pmatrix} 1 & k - \tan \alpha  \\ \tan \alpha & 1 + k\tan \alpha \end{pmatrix}\begin{pmatrix} \cos \alpha  & \sin\alpha \\ -\sin \alpha & \cos \alpha \end{pmatrix} \\ 
&&&= \cos\alpha \begin{pmatrix}\cos \alpha -k\sin\alpha + \frac{\sin^2 \alpha}{\cos \alpha} & \sin \alpha + k \cos \alpha - \sin \alpha \\ \sin \alpha - \sin \alpha - k\frac{\sin^2 \alpha}{\cos \alpha} & \frac{\sin^2 \alpha}{\cos \alpha} + \cos\alpha + k \sin \alpha \end{pmatrix} \\
&&&= \begin{pmatrix}1-k\sin\alpha\cos\alpha & k\cos^{2}\alpha\\
-k\sin^{2}\alpha & 1+k\sin\alpha\cos\alpha
\end{pmatrix}
\end{align*}

Suppose $\begin{pmatrix}p & q\\
r & t
\end{pmatrix} \mathbf{M} = \mathbf{M} \begin{pmatrix}p & q\\
r & t
\end{pmatrix}$ then,

\begin{align*}
&& \begin{pmatrix}p & q\\
r & t
\end{pmatrix} \mathbf{M} &= \mathbf{M} \begin{pmatrix}p & q\\
r & t
\end{pmatrix} \\
\Leftrightarrow && \small \begin{pmatrix} p(1-k\sin\alpha\cos\alpha) + q(-k\sin^{2}\alpha) & pk\cos^{2}\alpha + q(1+k\sin\alpha\cos\alpha)\\
r(1-k\sin\alpha\cos\alpha) + t(-k\sin^{2}\alpha) & rk\cos^{2}\alpha + t(1+k\sin\alpha\cos\alpha)\end{pmatrix} &= \\
&& \qquad \small \begin{pmatrix} p(1-k\sin\alpha\cos\alpha) + rk\cos^{2}\alpha & q(1-k\sin\alpha\cos\alpha) + tk\cos^{2}\alpha \\
-pk\sin^{2}\alpha + r(1+k\sin\alpha\cos\alpha)  & -qk\sin^{2}\alpha+t (1+k\sin\alpha\cos\alpha)
 \end{pmatrix} \\
\Leftrightarrow && \begin{cases} p(1-k\sin\alpha\cos\alpha) + q(-k\sin^{2}\alpha) &= p(1-k\sin\alpha\cos\alpha) + rk\cos^{2}\alpha \\
pk\cos^{2}\alpha + q(1+k\sin\alpha\cos\alpha) &=q(1-k\sin\alpha\cos\alpha) + tk\cos^{2}\alpha \\
 r(1-k\sin\alpha\cos\alpha) + t(-k\sin^{2}\alpha) &=-pk\sin^{2}\alpha + r(1+k\sin\alpha\cos\alpha) \\
 rk\cos^{2}\alpha + t(1+k\sin\alpha\cos\alpha) &= -qk\sin^{2}\alpha+t (1+k\sin\alpha\cos\alpha)
\end{cases} \\
\Leftrightarrow && \begin{cases} -q\tan^{2}\alpha &= r \\
p\cos^{2}\alpha + q\sin\alpha\cos\alpha &=-q\sin\alpha\cos\alpha + t\cos^{2}\alpha \\
 -r\sin\alpha\cos\alpha + -t\sin^{2}\alpha &=-p\sin^{2}\alpha + r\sin\alpha\cos\alpha \\
 r &= -q\tan^{2}\alpha
\end{cases} \\
\Leftrightarrow && \begin{cases} -q\tan^{2}\alpha &= r \\
2q\sin\alpha\cos\alpha &=(t-p)\cos^{2}\alpha \\
 (p-t)\sin^{2}\alpha &=2r\sin\alpha\cos\alpha
\end{cases} \\
\Leftrightarrow && \begin{cases} -q\tan^{2}\alpha &= r \\
2q\tan \alpha &=(t-p)
\end{cases} \\
\end{align*}

as required