The amount of material removed is the volume of the region within the sphere bounded by the cylinder. Consider a sphere of radius [tex]2a[/tex] centered at the origin; this sphere has equation
[tex]x^2+y^2+z^2=4a^2\iff z^2=4a^2-x^2-y^2[/tex]
The given cylinder has equation
[tex]x^2+y^2=a^2[/tex]
The volume of the region of interest [tex]\mathcal D[/tex] is given by
[tex]\displaystyle\iiint_{\mathcal D}\mathrm dV[/tex]
Converting to cylindrical coordinates, setting
[tex]x=r\cos\theta[/tex]
[tex]y=r\sin\theta[/tex]
[tex]z=\zeta[/tex]
we have
[tex]z^2=4a^2-r^2[/tex]
and
[tex]\mathrm dV=\mathrm dx\,\mathrm dy\,\mathrm dz=\left|\dfrac{\partial(x,y,z)}{\partial(r,\theta,\zeta)}\right|\,\mathrm dr\,\mathrm d\theta\,\mathrm d\zeta[/tex]
[tex]\implies\mathrm dV=r\,\mathrm dr\,\mathrm d\theta\,\mathrm d\zeta[/tex]
where [tex]\dfrac{\partial(x,y,z)}{\partial(r,\theta,\zeta)}[/tex] is the Jacobian of the transformation from [tex](x,y,z)[/tex] to [tex](r,\theta,\zeta)[/tex]. The region [tex]\mathcal D[/tex] is described by the set
[tex]\left\{(r,\theta,\zeta)\,:\,0\le r\le a\land0\le\theta\le2\pi\land-\sqrt{4a^2-r^2}\le\zeta\le\sqrt{4a^2-r^2}\right\}[/tex]
The integral is then
[tex]\displaystyle\iiint_{\mathcal D}\mathrm dV=\int_{\theta=0}^{\theta=2\pi}\int_{r=0}^{r=a}\int_{\zeta=-\sqrt{4a^2-r^2}}^{\zeta=\sqrt{4a^2-r^2}}r\,\mathrm d\zeta\,\mathrm dr\,\mathrm d\theta[/tex]
The integral with respect to [tex]\zeta[/tex] is symmetric about [tex]\zeta=0[/tex], so we instead compute twice the integral from [tex]\zeta=0[/tex] to [tex]\zeta=\sqrt{4a^2-r^2}[/tex], and we can immediately compute the integral with respect to [tex]\theta[/tex]:
[tex]=\displaystyle4\pi\int_{r=0}^{r=a}r\sqrt{4a^2-r^2}\,\mathrm dr[/tex]
Now, let [tex]s=4a^2-r^2[/tex], so that [tex]\mathrm ds=-2r\,\mathrm dr[/tex]:
[tex]=\displaystyle-2\pi\int_{r=0}^{r=a}-2r\sqrt{4a^2-r^2}\,\mathrm dr=-2\pi\int_{s=4a^2}^{s=3a^2}\sqrt s\,\mathrm ds[/tex]
[tex]=-2\pi\cdot\dfrac23s^{3/2}\bigg|_{s=4a^2}^{s=3a^2}[/tex]
[tex]=\dfrac{4\pi}3\left((4a^2)^{3/2}-(3a^2)^{3/2}\right)[/tex]
[tex]=\dfrac{4\pi(8-3^{3/2})a^3}3[/tex]
