I hope I am understanding your question right. You are sub-cloning a gene into pRSET-A, and then transforming BL21(DE3) cells with the ligation product. Does X-Gal and IPTG affect the transformation numbers?
First X-Gal shouldn't do anything to the numbers. In fact nothing will happen with it at all as there is no LacZ or LacZa genes to express, either on the plasmid or in the host cell strain.
IPTG however could affect the transformation numbers in BL21(DE3) as it will cause the gene to be expressed. If the gene has any toxic or detrimental effects on the cell then you might not get colonies, or only ones that are empty vector (and perhaps wrong orientation inserts) growing. Expression of the protein will also place an extra burden on the cells which could result in slower growth.
For this reason (and others) it is best to transform cloning strains such as DH5α with the ligation before transferring a correct plasmid construct to the expression strain. X-Gal will still do nothing here with pRSET-A, (but other plasmids that contain LacZa would).
For BL21(DE3) strains IPTG should only be added for induction of expression, not when generating colonies.