Using properties of determinants prove that: $\left|\begin{array}{ccc} 1+a^{2}-b^{2} & 2 a b & -2 b \\ 2 a b & 1-a^{2}+b^{2} & 2 a \\ 2 b & -2 a & 1-a^{2}-b^{2} \end{array}\right|=\left(1+a^{2}+b^{2}\right)^{3}$

Solution:

Operating $R_{1} \rightarrow R_{1}+b R_{3}, R_{2} \rightarrow R_{2^{-}} a R_{3}$
$\begin{array}{l} \left|\begin{array}{ccc} 1+a^{2}-b^{2}+2 b^{2} & 2 a b-2 a b & -2 b+b-a^{2} b-b^{3} \\ 2 a b-2 a b & 1-a^{2}+b^{2}+2 a^{2} & 2 a-a+a^{3}+a b^{2} \\ 2 b & -2 a & 1-a^{2}-b^{2} \end{array}\right| \\ =\left|\begin{array}{ccc} 1+a^{2}+b^{2} & 0 & -b-a^{2} b-b^{3} \\ 0 & 1+a^{2}+b^{2} & a+a^{3}+a b^{2} \\ 2 b & -2 a & 1-a^{2}+b^{2} \end{array}\right| \\ =\left|\begin{array}{ccc} 1+a^{2}+b^{2} & 0 & -b\left(1+a^{2}+b^{2}\right) \\ 0 & 1+a^{2}+b^{2} & a\left(1+a^{2}+b^{2}\right) \\ 2 b & -2 a & 1-a^{2}-b^{2} \end{array}\right| \end{array}$
Taking $\left(1+a^{2}+b^{2}\right)$ from $R_{1}$ and $R_{2}$
$=\left(1+a^{2}+b^{2}\right)^{2}\left|\begin{array}{ccc} 1 & 0 & -b \\ 0 & 1 & a \\ 2 b & -2 a & 1-a^{2}-b^{2} \end{array}\right|$
Operating $R_{3} \rightarrow R_{3}-2 b R_{1}+2 a R_{2}$
$=\left(1+a^{2}+b^{2}\right)^{2}\left|\begin{array}{ccc} 1 & 0 & -b \\ 0 & 1 & a \\ 0 & 0 & 1+a^{2}+b^{2} \end{array}\right|$
Taking $\left(1+a^{2}+b^{2}\right)$ from $R_{3}$
$\left(1+a^{2}+b^{2}\right)^{3}\left|\begin{array}{ccc} 1 & 0 & -b \\ 0 & 1 & a \\ 0 & 0 & 1 \end{array}\right|$
Expanding with respect to $C_{1}$
$\begin{array}{l} =\left(1+a^{2}+b^{2}\right)^{3} 1 \times[1-0] \\ =\left(1+a^{2}+b^{2}\right)^{3} \end{array}$
Hence proved.