Problem source

Two square matrices $\mathbf{A}$ and $\mathbf{B}$ satisfies $\mathbf{AB=0}.$
Show that either $\det\mathbf{A}=0$ or $\det\mathbf{B}=0$ or $\det\mathbf{A}=\det\mathbf{B}=0$.
If $\det\mathbf{B}\neq0$, what must $\mathbf{A}$ be? Give an example to show that the condition $\det\mathbf{A}=\det\mathbf{B}=0$ is not sufficient for the equation $\mathbf{AB=0}$ to hold. 
Find real numbers $p,q$ and $r$ such that 
\[
\mathbf{M}^{3}+2\mathbf{M}^{2}-5\mathbf{M}-6\mathbf{I}=(\mathbf{M}+p\mathbf{I})(\mathbf{M}+q\mathbf{I})(\mathbf{M}+r\mathbf{I}),
\]
where $\mathbf{M}$ is any square matrix and $\mathbf{I}$ is the appropriate identity matrix. 
Hence, or otherwise, find all matrices $\mathbf{M}$ of the form $\begin{pmatrix}a & c\\
0 & b
\end{pmatrix}$ which satisfy the equation 
\[
\mathbf{M}^{3}+2\mathbf{M}^{2}-5\mathbf{M}-6\mathbf{I}=\mathbf{0}.
\]

Solution source

Since $0 = \det \mathbf{0} = \det \mathbf{AB} = \det \mathbf{A} \det\mathbf{B}$ at least one of $\det \mathbf{A}$ or $\det \mathbf{B}$ is zero. 

If $\det \mathbf{B} \neq 0$ then $\mathbf{B}$ is invertible, and multiplying on the right by $\mathbf{B}^{-1}$ gives us $\mathbf{A} = \mathbf{0}$.

If $\mathbf{A} = \begin{pmatrix} 1 & 1 \\ 0 & 0 \end{pmatrix}$ and $\mathbf{B} = \begin{pmatrix} 1 & 0 \\1 & 0 \end{pmatrix}$, then $\det \mathbf{A} = \det \mathbf{B} = 0$, but $\mathbf{AB} = \begin{pmatrix} 1 & 0 \\ 0 & 0 \end{pmatrix} \neq \mathbf{0}$

Since $\mathbf{M}$ commutes with itself and the identity matrix, this is equivalent to factorising the polynomial over the reals. Therefore

$$\mathbf{M}^{3}+2\mathbf{M}^{2}-5\mathbf{M}-6\mathbf{I}=(\mathbf{M}-2\mathbf{I})(\mathbf{M}+\mathbf{I})(\mathbf{M}+3\mathbf{I}),$$

Since we now know at least one of $\det (\mathbf{M}-2\mathbf{I})$, $\det (\mathbf{M}+\mathbf{I})$, $\det (\mathbf{M}+3\mathbf{I})$, we should look at cases:

Since at least one of those must be non-zero, we must have the following cases: $(a,b) = (2,-1), (-1,2), (-1,-3), (-3,-1), (2,-3), (-3,2)$

In each of those cases, we will have:

$\begin{pmatrix} 0 & c \\ 0 & b+k  \end{pmatrix}\begin{pmatrix} a+l & c \\ 0 & 0 \end{pmatrix} = \begin{pmatrix} 0 & 0 \\ 0 & 0\end{pmatrix}$ and so all of those solutions are valid. So $c$ can be anything as long as $(a,b)$ are in that set of solutions