I'm not sure if I understand you correctly, but if you can form a reaction equation like A + B -> k1 C + k2 D, you can estimate the theoretical yield (assuming that no other reactions ocur). With that you can calculate your own yield in reference to this value.
Otherwise you will need to identify the other byproducts and adapt your reaction equation.
Refer them to one of your reactants (or two reactants with an explanation if one of them comes from reagent A that reacts with B and the other only from B in excess, for example
If reaction goes live mentioned above: A + B -> C + D or A -> C + D I would first calculate the maximum amount of C and D that theoretically is produced.
I assume you know the mechanism of the reaction and you know how much A (and B) you mixed so you should know the max of C and D.
The question is how you quantify those peaks and your products. If you have a method then just compare obtained results with the theoretical ones.
To quantify both products, you will need pure samples of both your products to act as standards. Run them together on HPLC to develop a method which gives you separate peaks. Then run each standard* at five different concentrations (preferably in triplicate) and generate a standard curve for each one from mean peak area vs known concentration. Then you can calculate the concentration of your product from its peak area in the HPLC run. This strategy also works in GC, CZE and all forms of quantitative chromatography.
Then calculate the yield as a percentage of your maximum theoretical concentration.
*To save time, you can continue running the standards together while generating the concentration standard curve.
First you have to run a standard solution in column chromatography and generate a curve , with the help of this curve you may easily find the yield of products.