Problem source

Let $G$ be the set of all matrices of the form 
			\[
			\begin{pmatrix}a & b\\
			0 & c
			\end{pmatrix},
			\]
			where $a,b$ and $c$ are integers modulo 5, and $a\neq0\neq c$.
			Show that $G$ forms a group under matrix multiplication (which may
			be assumed to be associative). What is the order of $G$? Determine
			whether or not $G$ is commutative. 
			Determine whether or not the set consisting of all elements in $G$
			of order $1$ or $2$ is a subgroup of $G$.

Solution source

Claim $G$ is a group under matrix multiplication

\begin{itemize}

\item (Closure) Suppose $\mathbf{A}$ and $\mathbf{B}$ are matrices of that form, then $\begin{pmatrix} a_1 & b_1 \\ 0 & c_1 \end{pmatrix} \begin{pmatrix} a_2 & b_2 \\ 0 & c_2 \end{pmatrix} = \begin{pmatrix} a_1a_2 & a_1b_2 + b_1c_2 \\ 0 & c_1c_2 \end{pmatrix}$, this is clearly of the required form since if $a_1, a_2, c_1, c_2 \neq 0$ then $a_1a_2, c_1c_2 \neq 0$

\item (Associative) By inheritance from matrix multiplication

\item (Identity) Consider $\begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix}$ also clearly of the required form.

\item (Inverse) Consider $(ac)^{-1}\begin{pmatrix} c & -b \\ 0 & a \end{pmatrix}$, since $ac \neq 0$ we can assume it has an inverse mod $5$. therefore we have another matrix of the required form.

\end{itemize}

There are $4$ possible values for $a$ and $c$ and $5$ possible values for $b$, so $4 \times 4 \times 5 = 80$ elements, so the group is order $80$.

$G$ is not commutative, consider

$\begin{pmatrix} 1 & 1 \\ 0 & 2 \end{pmatrix} \begin{pmatrix} 1 & 1\\ 0 & 1 \end{pmatrix} = \begin{pmatrix} 1 & 2 \\ 0 & 2 \end{pmatrix}$

$\begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 1 & 1 \\ 0 & 2 \end{pmatrix}  = \begin{pmatrix} 1 & 3 \\ 0 & 2 \end{pmatrix}$


The elements of order $1$ or $2$ satisfy $\begin{pmatrix} a & b \\ 0 & c \end{pmatrix} = \begin{pmatrix} a^{-1} & -ba^{-1}c^{-1} \\ 0 & c^{-1} \end{pmatrix}$

Therefore $a^2 = 1, c^2 = 1 \Rightarrow a, c = 1, 4$ and $b = -ba^{-1}c^{-1} \Rightarrow b = 0$ or , $ac = -1$, so we have $(a,b,c) = (1,0,1), (4,0,4), (1, *, 4), (4, *, 1)$

So there are $12$ elements of order $1$ or $2$. But this can't be a subgroup since $12 \not \mid 80$