If you are asking about the challenges that you face in real world then there are too many! Deformation, occlusion, object size are some of them that you may find in different benchmark datasets. In my experience, illumination change and motion blur are two of the most difficult problems to handle. The difficulty will increase if your object is not completely opaque.
Of course, it is not just the accuracy rate. The tracking should be robust to many factors such as varieties of poses, 3D spatial deformations, illuminations and many more. Further the ability of recovering from lost tracking, assigning tracking models for stray objects and the efficiency of tracking should be concerned.