Animating with 3D Rotation Tool

I always have issues when I try to animate an object using the 3D rotation tool. Every time I use it on the second keyframe I rotate the object using the 3d rotation handles on the object and then when I play it back it always over rotates. Why does it do this? This tool is so frustrating. I hope they fix it.