If I were you, I'd stick to the exact same load you are shooting right now and get closer if you have to. Heavy charges don't do much for accuracy most of the time, and the difference in velocity is probably going to be minimal with a 320 grain bullet... meaning probably less than 150 fps.
What you are comfortable shooting is more important than velocity. Your most accurate load you can shoot, is your best choice of loads for any game animal.
Good luck on your elk hunt.
Aloha...
