I had a similar problem on my HDi a couple of months back......check the braided flexi hose between the vac pump and the metal lines on the bulkhead. This was split in two places on mine and looked compressed instead of round. It wasn't easy to see that it was split due to the braid covering it. There's also a one way valve that plugs in on the top left of the servo at the other end of the vac line from the pump. This was sticking and had to be replaced too. The valve is easy to test when it's off the car, simply blow through it and if there's any resistance and it pauses before it opens then change it as it should operate instantly.
I changed both and problem solved.
I changed both and problem solved.