Let
[tex]I(a) = \displaystyle \int_0^\infty \frac{\sqrt x \arctan(ax)}{1+x^2} \, dx[/tex]
Differentiate with respect to a :
[tex]I'(a) = \displaystyle a \int_0^\infty \frac{x^{\frac32}}{(1+x^2)(1+a^2x^2)} \, dx[/tex]
Substitute y = √x :
[tex]I'(a) = \displaystyle 2a \int_0^\infty \frac{y^4}{(1+y)(1+a^2y)} \, dy[/tex]
Polynomial division yields
[tex]\dfrac{y^4}{(1+y)(1+a^2y)} \\\\ = \dfrac1{a^2}y^2 - \left(\dfrac1{a^2} + \dfrac1{a^4}\right)y + \dfrac1{a^2} + \dfrac1{a^4} + \dfrac1{a^6} - \dfrac{\left(1+\frac1{a^2} + \frac1{a^4} + \frac1{a^6}\right)y + \frac1{a^2} + \frac1{a^4} + \frac1{a^6}}{(1+y)(1+a^2y)}[/tex]
Computing I'(a) isn't so difficult from here. You'd find (assuming a ≥ 0)
[tex]I'(a) = \displaystyle \frac\pi{\sqrt2\left(\sqrt a + a + a^{\frac32} + a^2\right)}[/tex]
Integrate both sides with respect to a. On the right side, substituting b = √a yields
[tex]\displaystyle \int \frac{da}{\sqrt a + a + a^{\frac32} + a^2} = \int \frac{2b}{b + b^2 + b^3 + b^4} \, db \\\\ = 2 \int \frac{db}{1 + b + b^2 + b^3} \\\\ = 2 \int \frac{db}{(1+b)(1+b^2)} \\\\ = \int \left(\frac1{1+b} + \frac{1-b}{(1+b^2)}\right) \, db \\\\ = \frac14 \left(2 \arctan(b) + 2 \ln(1 + b) - \ln(1 + b^2)\right) + C \\\\ = \frac14 \left(2 \arctan(\sqrt a) + 2 \ln(1 + \sqrt a) - \ln(1 + a)\right) + C[/tex]
Noting that a = 0 makes the integral I(a) vanish, we have
[tex]0 = \dfrac14 \left(2 \arctan(\sqrt0) + 2\ln(1 + \sqrt0) - \ln(1 + 0)\right) + C \implies C = 0[/tex]
and so
[tex]\displaystyle I(a) = \frac\pi{4\sqrt2} \left(2 \arctan(\sqrt a) + 2 \ln(1 + \sqrt a) - \ln(1 + a)\right)[/tex]
We recover the integral we want with a = 1, which gives a value of
[tex]\displaystyle \int_0^\infty \frac{\sqrt x \arctan(x)}{1 + x^2} \, dx = \boxed{\frac{\pi^2 + 2\pi\ln(2)}{4\sqrt2}}[/tex]
