the anwer is: Yes you need more energy ( 5,2eV > 4,4eV)
Please see the definition of the "work function" at the attached link.
I would not say that removing an electron from Gold surface is more 'difficult' than removing from Silver surface. You just need a little bit more energy in the Gold case.