[tex]\cot^4\theta-4\cot^2\theta-5=(\cot^2\theta-5)(\cot^2\theta+1)=0[/tex]
[tex]\cot^2\theta-5=0\implies\cot^2\theta=5\implies\cot\theta=\pm\sqrt5[/tex]
Recall that [tex]\cot\theta=\cot(\theta+\pi)[/tex], which is to say [tex]\cot x[/tex] has period [tex]\pi[/tex]. This in turn means that [tex]\cot\theta=0[/tex] will have the same solutions as [tex]\cot(\theta+n\pi)=0[/tex] for any integer [tex]n[/tex]. So the general solution to the first case is
[tex]\cot\theta=\pm\sqrt5\implies\theta=\cot^{-1}(\pm\sqrt5)+n\pi[/tex]
where [tex]n[/tex] is any integer.
On the other hand,
[tex]\cot^2\theta+1=0\implies\cot^2\theta=-1[/tex]
but [tex]x^2\ge0[/tex] for any value of [tex]x[/tex], so this equation has no (real) solutions for [tex]\theta[/tex].
