Solution:

Operating $R_{1} \rightarrow R_{1}+b R_{3}, R_{2} \rightarrow R_{2^{-}} a R_{3}$
$\begin{array}{l}
\left|\begin{array}{ccc}
1+a^{2}-b^{2}+2 b^{2} & 2 a b-2 a b & -2 b+b-a^{2} b-b^{3} \\
2 a b-2 a b & 1-a^{2}+b^{2}+2 a^{2} & 2 a-a+a^{3}+a b^{2} \\
2 b & -2 a & 1-a^{2}-b^{2}
\end{array}\right| \\
=\left|\begin{array}{ccc}
1+a^{2}+b^{2} & 0 & -b-a^{2} b-b^{3} \\
0 & 1+a^{2}+b^{2} & a+a^{3}+a b^{2} \\
2 b & -2 a & 1-a^{2}+b^{2}
\end{array}\right| \\
=\left|\begin{array}{ccc}
1+a^{2}+b^{2} & 0 & -b\left(1+a^{2}+b^{2}\right) \\
0 & 1+a^{2}+b^{2} & a\left(1+a^{2}+b^{2}\right) \\
2 b & -2 a & 1-a^{2}-b^{2}
\end{array}\right|
\end{array}$
Taking $\left(1+a^{2}+b^{2}\right)$ from $R_{1}$ and $R_{2}$
$=\left(1+a^{2}+b^{2}\right)^{2}\left|\begin{array}{ccc}
1 & 0 & -b \\
0 & 1 & a \\
2 b & -2 a & 1-a^{2}-b^{2}
\end{array}\right|$
Operating $R_{3} \rightarrow R_{3}-2 b R_{1}+2 a R_{2}$
$=\left(1+a^{2}+b^{2}\right)^{2}\left|\begin{array}{ccc}
1 & 0 & -b \\
0 & 1 & a \\
0 & 0 & 1+a^{2}+b^{2}
\end{array}\right|$
Taking $\left(1+a^{2}+b^{2}\right)$ from $R_{3}$
$\left(1+a^{2}+b^{2}\right)^{3}\left|\begin{array}{ccc}
1 & 0 & -b \\
0 & 1 & a \\
0 & 0 & 1
\end{array}\right|$
Expanding with respect to $C_{1}$
$\begin{array}{l}
=\left(1+a^{2}+b^{2}\right)^{3} 1 \times[1-0] \\
=\left(1+a^{2}+b^{2}\right)^{3}
\end{array}$
Hence proved.