If we considered your nanoparticles as any other drug,The encapsulation effeciency (EE) of any drug carrier can be determined using two routes depending on the drug and carrier properties and amounts. Either by measuring the amount of drug attached to the carrier relative to the amount of drug that was available for encapsulation and hence EE can be determined as:

EE% = ([drug attached to carrier]/[original amount of drug] )x100%,

Or by measuring the amount of free drug that was left from the original available amount of the drug for encapsulation and hence EE can be determined as:

EE% =((original drug –drug outside)/original amount of drug))x100%

Using High performance liquid chromatography (HPLC) with UV detector depends on the nature of your nanoparticles

Lets say initial concentration of magnetite nanoparticle was 5mg. So if we separate the nanoparticles from the supernatant, measure the dry weight and subtract it from the initial then we will get the encapsulated product?

% EE= Initial total NP concentration (mg) - Concentration in supernantent (mg) /total concentration (mg) x 100

Yes, that's right, and you can also Destruction of liposome particles after purification from free drug and find out the amount of drug encapsulated inside the liposome compared to the amount of the original drug

EE% = ([drug attached to carrier]/[original amount of drug] )x100%,