It will still turn, but the speed will be almst half of what they state as your flow is 9gpm, so 400-450 rpm at a guess. The other thing is, your hydraulic pressure is lower, so it won't get as much torque as is stated.
With that said, to gain torque you need more displacement in the motor, this one has essentially twice the displacement of the motor that would ideally match your machine, so it will have more power.
Being an auger, you don't really need the speed, you just need the torque to pull through the ground. I believe the motor would do it, but it will not do it as fast. With your lower flow rate, you will have this limiting factor anyway.
You do know they also sell splined PTO adaptors right? you can get a motor with a standard keyed shaft and get an adaptor to suit, i have bought one off them in the past.