Solution:

Option(A) is correct.
To Find: The value of $\cot ^{-1}\left(\cot \left(\frac{5 \pi}{4}\right)\right)$
Now, let $x=\cot ^{-1}\left(\cot \left(\frac{5 \pi}{4}\right)\right)$
$\Rightarrow \cot x=\cot \left(\frac{5 \pi}{4}\right)$
Here range of principle value of $\cot$ is $\left[-\frac{\pi}{2}, \frac{\pi}{2}\right]$
$\Rightarrow x=\frac{5 \pi}{4} \notin\left[-\frac{\pi}{2}, \frac{\pi}{2}\right]$
Hence for all values of $x$ in range $\left[-\frac{\pi}{2}, \frac{\pi}{2}\right]$, the value of $\cot ^{-1}\left(\cot \left(\frac{5 \pi}{4}\right)\right)$ is
$\begin{array}{l}
\Rightarrow \cot x=\cot \left(\pi+\frac{\pi}{4}\right)\left(\because \cot \left(\frac{5 \pi}{4}\right)=\cot \left(\pi+\frac{\pi}{4}\right)\right) \\
\Rightarrow \cot x=\cot \left(\frac{\pi}{4}\right)(\because \cot (\pi+\theta)=\cot \theta) \\
\Rightarrow x=\frac{\pi}{4}
\end{array}$