First of all, notice how this equation has this relationship:
sqrt(x^2 - 1) prop sqrt(a^2x^2 - a^2) prop sqrt(a^2sec^2theta - a^2)
where prop means "proportional to", and a = 1. Thus, let:
a = 1
x = asectheta = sectheta
dx = secthetatanthetad theta
sqrt(x^2 - 1) = sqrt(sec^2theta - 1) = sqrt(tan^2theta) = tantheta
Now you can write this as:
int sqrt(x^2 - 1)dx
= int tanthetasecthetatanthetad theta
= intsecthetatan^2thetad theta
Something you can do to simplify this (and it may seem odd at first) is to rewrite this as:
= int sectheta(sec^2theta - 1)d theta
since tan^2theta = sec^2theta - 1. Now we only have sectheta to deal with.
= intsec^3theta - secthetad theta
These two integrals require special tricks, but they do have real answers. Let's take them separately for simplicity.
int secthetad theta
= int (sectheta)((sectheta + tantheta)/(sectheta + tantheta))d theta
= int (sec^2theta + secthetatantheta)/(sectheta + tantheta)d theta
Now notice that if you take the derivative of the denominator, you get the numerator times d theta. i.e.:
d[sectheta + tantheta] = (sec^2theta + secthetatantheta)d theta
Thus, let:
u = sectheta + tantheta
du = secthetatantheta + sec^2thetad theta
=> int 1/u du
= ln|u| = ln|sectheta + tantheta|
Now let's take the other integral.
int sec^3thetad theta
With this, the best trick one can try is Integration by Parts. Typically, you let u = sectheta, so let:
u = sectheta
du = secthetatanthetad theta
dv = sec^2thetad theta
v = tantheta
= secthetatantheta - int secthetatan^2thetad theta
Look at that, we got back the original integral (didn't even need an identity for this step)!
The whole thing now is:
int secthetatan^2thetad theta
= secthetatantheta - color(darkgreen)(int secthetatan^2thetad theta) - int secthetad theta
2int secthetatan^2thetad theta = secthetatantheta - int secthetad theta
Technically, we're done. Now we just have:
int secthetatan^2thetad theta = 1/2[secthetatantheta - int secthetad theta]
= 1/2[secthetatantheta - ln|sectheta + tantheta|]
Now if you look all the way at the top:
x = sectheta
sqrt(x^2 - 1) = tantheta
Thus:
= color(blue)(1/2[xsqrt(x^2 - 1) - ln|x + sqrt(x^2 - 1)|] + C
You can also see it here.