For differentiation of dolomite and calcite you use staining with alizarin red (and potassium ferricyanide for differentiation of dolomite and ankerite). You should find many papers on this topic.
If you cannot use this method it is more difficult. Dolomite often forms euhedral rhombs and can be a little more cloudy or beige than calcite.
Following my textbook magnesite should lack twin lamellae. But for recognition of this your minerals probably should have a minimum size.
in mudlogging services there is calcimetry which u can by degassing CO2 identify the magnesium and calcium and from graph and behave of trend you will know exactly which type of carbonate is.
use staining with red alizarin to differentiate calcite from dolomite but before you can evaluate your sample using HCL (30%): calcite reacts easily at cold with Hcl. Dolomite reacts with hcl after heating. For others minerals it could be interesting to use DRX
Along with all above techniques mentioned by other researchers, the PXRD would be more beneficial because both of the minerals have separate XRD patterns and can easily be distinguished. The major peak of calcite is at 29.5 two theta while the major peak of dolomite is at 30.5 two-theta.
It can be distinguished in the field between calcite and dolomite, that calcite is more reactive with dilute hydrochloric acid. Under a polarized microscope, Calcite commonly shows lamellar twinning, said usually to be absent from dolomite and dolomite grains can be clearly distinguished dolomite rhombs.