Problem source

Show that the coefficient of $x^{-12}$ in the expansion of 
\[
\left(x^{4}-\frac{1}{x^{2}}\right)^{5} 
\left(x-\frac{1}{x}\right)^{6} 
\]
is $-15$, and calculate the coefficient of $x^2$.
Hence, or otherwise,
calculate the coefficients of $x^4$ and $x^{38}$ in the expansion of 
\[
(x^2-1)^{11}(x^4+x^2+1)^5.
\]

Solution source

The powers of $x$ in the first bracket will be $x^{20}, x^{14}, \cdots, x^{-10}$. The powers of $x$ in the second bracket will be $x^6, x^4, \cdots, x^{-6}$.

Therefore we can achieve $x^{-12}$ in only one way:

\begin{array}{c|c|c|c|c}
1\text{st bracket} & 2\text{nd bracket} & 1\text{st coef} & 2\text{nd coef} & \text{prod} \\ \hline 
x^{-10} & x^{-2} & \binom{5}{5}(-1)^5 = -1 & \binom{6}{4}(-1)^4 = 15& -15 \\ 
\end{array}

We can achieve $x^2$ as follows:

\begin{array}{c|c|c|c|c}
1\text{st bracket} & 2\text{nd bracket} & 1\text{st coef} & 2\text{nd coef} & \text{prod} \\ \hline 
x^{-4} & x^{6} & \binom{5}{4}(-1)^4 = 5 & \binom{6}{0}(-1)^0 = 1& 5 \\ 
x^{2} & x^{0} & \binom{5}{3}(-1)^3 = -10 & \binom{6}{3}(-1)^3 = -20 & 200 \\
x^{8} & x^{-6} & \binom{5}{2}(-1)^2 = 10 & \binom{6}{6}(-1)^6 = 1 & 10
\end{array}

Therefore the coefficient is $215$ 

$(x^2-1)(x^4+x^2+1) = x^6-1$, therefore 

\begin{align*}
(x^2-1)^{11}(x^4+x^2+1)^5 &= (x^2-1)^6(x^6-1)^5 \\
&= x^6\left(x-\frac1x\right)^6(x^6-1)^6 \\
&= x^6\left(x-\frac1x\right)^6\left(x^2\left(x^4-\frac{1}{x^2}\right)\right)^5 \\
&= x^6\left(x-\frac1x\right)^6x^{10}\left(x^4-\frac{1}{x^2}\right)^5 \\
&= x^{16}\left(x-\frac1x\right)^6\left(x^4-\frac{1}{x^2}\right)^6 \\
\end{align*}

Therefore the coefficient of $x^4$ is the coefficient of $x^{4-16} = x^{-12}$ in our original expression, ie $-15$.

Similarly, the coefficient of $x^{38}$ is the coefficient of $x^{38-16} = x^{22}$, which can only be achieved in one way:

\begin{array}{c|c|c|c|c}
1\text{st bracket} & 2\text{nd bracket} & 1\text{st coef} & 2\text{nd coef} & \text{prod} \\ \hline 
x^{20} & x^{2} & \binom{5}{0}(-1)^0 = 1 & \binom{6}{2}(-1)^2 = 15& 15 \\ 
\end{array}

Therefore the coefficient is $15$