Solution: Claim \(G\) is a group under matrix multiplication
- (Closure) Suppose \(\mathbf{A}\) and \(\mathbf{B}\) are matrices of that form, then \(\begin{pmatrix} a_1 & b_1 \\ 0 & c_1 \end{pmatrix} \begin{pmatrix} a_2 & b_2 \\ 0 & c_2 \end{pmatrix} = \begin{pmatrix} a_1a_2 & a_1b_2 + b_1c_2 \\ 0 & c_1c_2 \end{pmatrix}\), this is clearly of the required form since if \(a_1, a_2, c_1, c_2 \neq 0\) then \(a_1a_2, c_1c_2 \neq 0\)
- (Associative) By inheritance from matrix multiplication
- (Identity) Consider \(\begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix}\) also clearly of the required form.
- (Inverse) Consider \((ac)^{-1}\begin{pmatrix} c & -b \\ 0 & a \end{pmatrix}\), since \(ac \neq 0\) we can assume it has an inverse mod \(5\). therefore we have another matrix of the required form.
There are \(4\) possible values for \(a\) and \(c\) and \(5\) possible values for \(b\), so \(4 \times 4 \times 5 = 80\) elements, so the group is order \(80\).
\(G\) is not commutative, consider
\(\begin{pmatrix} 1 & 1 \\ 0 & 2 \end{pmatrix} \begin{pmatrix} 1 & 1\\ 0 & 1 \end{pmatrix} = \begin{pmatrix} 1 & 2 \\ 0 & 2 \end{pmatrix}\)
\(\begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 1 & 1 \\ 0 & 2 \end{pmatrix} = \begin{pmatrix} 1 & 3 \\ 0 & 2 \end{pmatrix}\)
The elements of order \(1\) or \(2\) satisfy \(\begin{pmatrix} a & b \\ 0 & c \end{pmatrix} = \begin{pmatrix} a^{-1} & -ba^{-1}c^{-1} \\ 0 & c^{-1} \end{pmatrix}\)
Therefore \(a^2 = 1, c^2 = 1 \Rightarrow a, c = 1, 4\) and \(b = -ba^{-1}c^{-1} \Rightarrow b = 0\) or , \(ac = -1\), so we have \((a,b,c) = (1,0,1), (4,0,4), (1, *, 4), (4, *, 1)\)
So there are \(12\) elements of order \(1\) or \(2\). But this can't be a subgroup since \(12 \not \mid 80\)