Problem source

Let $\mathbf{a},\mathbf{b}$ and $\mathbf{c}$ be the position vectors of points $A,B$ and $C$ in three-dimensional space. Suppose that $A,B,C$ and the origin $O$ are not all in the same plane. Describe
the locus of the point whose position vector $\mathbf{r}$ is given by 
\[
\mathbf{r}=(1-\lambda-\mu)\mathbf{a}+\lambda\mathbf{b}+\mu\mathbf{c},
\]
where $\lambda$ and $\mu$ are scalar parameters. By writing this equation in the form $\mathbf{r}\cdot\mathbf{n}=p$ for a suitable
vector $\mathbf{n}$ and scalar $p$, show that 
\[
-(\lambda+\mu)\mathbf{a}\cdot(\mathbf{b}\times\mathbf{c})+\lambda\mathbf{b}\cdot(\mathbf{c}\times\mathbf{a})+\mu\mathbf{c}\cdot(\mathbf{a}\times\mathbf{b})=0
\]
for all scalars $\lambda,\mu.$ 
Deduce that 
\[
\mathbf{a}\cdot(\mathbf{b}\times\mathbf{c})=\mathbf{b}\cdot(\mathbf{c}\times\mathbf{a})=\mathbf{c}\cdot(\mathbf{a}\times\mathbf{b}).
\]
Say briefly what happens if $A,B,C$ and $O$ are all in the same plane.

Solution source

$\mathbf{r}=(1-\lambda-\mu)\mathbf{a}+\lambda\mathbf{b}+\mu\mathbf{c} = \mathbf{a} + \lambda(\mathbf{b}-\mathbf{a})+\mu(\mathbf{c}-\mathbf{a})$

Therefore it is the plane through $\mathbf{a}$ with direction vectors $\mathbf{b}-\mathbf{a}$ and $\mathbf{c}-\mathbf{a}$, ie it is the plane through $\mathbf{a},\mathbf{b},\mathbf{c}$.

The normal to this plane will be $(\mathbf{b}-\mathbf{a} ) \times (\mathbf{c}-\mathbf{a}) = \mathbf{b}\times \mathbf{c}-\mathbf{a} \times \mathbf{c}-\mathbf{b}\times \mathbf{a}$, so we must have:

\begin{align*}
&& \mathbf{r} \cdot \left (\mathbf{b}\times \mathbf{c}-\mathbf{a} \times \mathbf{c}-\mathbf{b}\times \mathbf{a} \right) &= \mathbf{a} \cdot  \left (\mathbf{b}\times \mathbf{c}-\mathbf{a} \times \mathbf{c}-\mathbf{b}\times \mathbf{a} \right) \\
&&&=  \mathbf{a} \cdot  (\mathbf{b}\times \mathbf{c})
\end{align*}

Therefore,

\begin{align*}
 && \mathbf{a} \cdot  (\mathbf{b}\times \mathbf{c}) &=  \mathbf{r} \cdot \left (\mathbf{b}\times \mathbf{c}-\mathbf{a} \times \mathbf{c}-\mathbf{b}\times \mathbf{a} \right) \\
&&&= \left ( (1-\lambda-\mu)\mathbf{a}+\lambda\mathbf{b}+\mu\mathbf{c} \right)\cdot \left (\mathbf{b}\times \mathbf{c}-\mathbf{a} \times \mathbf{c}-\mathbf{b}\times \mathbf{a} \right) \\
&&&= (1-\lambda- \mu) \mathbf{a}\cdot (\mathbf{b} \times \mathbf{c})-\lambda \mathbf{b}\cdot(\mathbf{a} \times \mathbf{c})-\mu \mathbf{c}\cdot(\mathbf{b} \times \mathbf{a}) \\
\Rightarrow && 0 &= (-\lambda- \mu) \mathbf{a}\cdot (\mathbf{b} \times \mathbf{c})-\lambda \mathbf{b}\cdot(\mathbf{a} \times \mathbf{c})-\mu \mathbf{c}\cdot(\mathbf{b} \times \mathbf{a}) \\
&&&= -(\lambda+ \mu) \mathbf{a}\cdot (\mathbf{b} \times \mathbf{c})+\lambda \mathbf{b}\cdot(\mathbf{c} \times \mathbf{a})+\mu \mathbf{c}\cdot(\mathbf{a} \times \mathbf{b}) \\
\end{align*}

The result follows from setting $\mu = 0, \lambda = 1$ and $\mu = 1, \lambda = 0$.

If they all lie in the same plane then the plane described is through the origin, and those values are all the same, but equal to $0$.