And how can I show the work?

so I substituted the x with 3 and 5, got two decimals (27.16 and 16.10) but my teacher says it's not correct, but all she wrote down on my paper was ' subtract now ' and I've subtracted all that I could think of and I still didn't get the answer. Help is much appreciated!

And how can I show the work so I substituted the x with 3 and 5 got two decimals 2716 and 1610 but my teacher says its not correct but all she wrote down on my class=