The first thing you need to do is say that x = 3sec(u) ==> dx = 3sec(u)tan(u) du
If we say this we can proceed in this manner
∫ 1/[x^2√(x^2 - 9)] dx
= ∫ 3sec(u)tan(u)/{9sec^2(u)√[9sec^2(u) - 9]} du, by applying substitutions
= ∫ 3sec(u)tan(u)/{27sec^2(u)tan(u)] du, since tan^2(u) = sec^2(u) - 1
= 1/9 ∫ 1/sec(u) du, by canceling sec(u)tan(u) du
= 1/9 ∫ cos(u) du, since 1/sec(u) = cos(u)
= (1/9)sin(u) + C.
With x = 3sec(u) ==> sec(u) = x/3 and cos(u) = 3/x, we have:
sin(u) = √[1 - cos^2(u)] = √[1 - (3/x)^2] = √(x^2 - 9)/x.
Therefore, back-substituting yields:
∫ 1/[x^2√(x^2 - 9)] dx = (1/9)sin(u) + C
= (1/9)[√(x^2 - 9)/x] + C
= √(x^2 - 9)/(9x) + C.
The answer will be: sqrt(x^9-9)/9x
I hope this helps a lot