Rather than carrying out IBP several times, let's establish a more general result. Let
[tex]I(n)=\displaystyle\int x^ne^x\,\mathrm dx[/tex]
One round of IBP, setting
[tex]u=x^n\implies\mathrm du=nx^{n-1}\,\mathrm dx[/tex]
[tex]\mathrm dv=e^x\,\mathrm dx\implies v=e^x[/tex]
gives
[tex]\displaystyle I(n)=x^ne^x-n\int x^{n-1}e^x\,\mathrm dx[/tex]
[tex]I(n)=x^ne^x-nI(n-1)[/tex]
This is called a power-reduction formula. We could try solving for [tex]I(n)[/tex] explicitly, but no need. [tex]n=5[/tex] is small enough to just expand [tex]I(5)[/tex] as much as we need to.
[tex]I(5)=x^5e^x-5I(4)[/tex]
[tex]I(5)=x^5e^x-5(x^4e^x-4I(3))=(x-5)x^4e^x+20I(3)[/tex]
[tex]I(5)=(x-5)x^4e^x+20(x^3e^x-3I(2))=(x^2-5x+20)x^3e^x-60I(2)[/tex]
[tex]I(5)=(x^2-5x+20)x^3e^x-60(x^2e^x-2I(1))=(x^3-5x^2+20x-60)x^2e^x+120I(1)[/tex]
[tex]I(5)=(x^3-5x^2+20x-60)x^2e^x+120(xe^x-I(0))[/tex]
Finally,
[tex]I(0)=\displaystyle\int e^x\,\mathrm dx=e^x+C[/tex]
so we end up with
[tex]I(5)=(x^4-5x^3+20x^2-60x+120)xe^x-120e^x+C[/tex]
[tex]I(5)=(x^5-5x^4+20x^3-60x^2+120x-120)e^x+C[/tex]
and the antiderivative is
[tex]\displaystyle\int2x^5e^x\,\mathrm dx=(2x^5-10x^4+40x^3-120x^2+240x-240)e^x+C[/tex]
