In my opinion and experience, As you said, inert fines are intrinsic part of binder (Not added apart, like somehow would be treated as aggregate) , you must consider w/cement as = w/binder ratio for the hole cementitious material . Ex. W/(C+p) >"Cement + puzolan).

it should be added if they have contribution to stregnth if not dont add it