If it's one direction only, i'd think it is a relief valve. The swash plate just alters the angle of the rotating group that pumps the oil. If the rotating group or wear plate were worn, the issue would be both directions.
I can't remember totally..... But look at the top of the pump, there will be i believe 4 allen headed plugs. Hopefully they are all on the top and not have two hidden underneath. Either way, if there are on the top, remove them one by one and inspect them. I am not sure if they are all the same, but of the springs look the same thickness, try swapping them around and see if the issue moves.
They have an Oring around them to seal as well, so they may be a little stiff to get out.
I will have a look through my images to see if i have a record of where these relief valves are located.