Drifting of energy means you are not in the stable phase of the system. Secondary reason is that, it may be an artifact. May i know 1) which package you are using for simulations? 2) Are your temperature and potential energy stablized?
I'm using NAMD (NVE ensemble), yes temperature and potential energy are stabilized (potential energy is stable and fluctuating in range of average ± 200 Kcal/mol). it seems in NVE ensemble that due to rounding and truncation errors during the integration process, there is always a slight drift in energy as stated in this link:
yes, you are right that there are rounding and truncation errors. in normal case, 5 kCal/mol.ns is considerable change in NVE ensemble as per my knowledge. but if it is continuosly falling than you have to check where is the problem. Have you tried longer runs?
Another aspect is that your potential energy is stablized and temperature is stablized then kinetic energy should also be stablized ( K.E = 0.5 kBT). Now, total energy = PE+KE, so your total energy should also be stablized. Check the algorithm of NAMD or writing file section, if it is giving something else. You can do one more check, from nve you have idea about the average temperature of the system, setup one NVT run followed by NVE run and see if your drift is decreasing.