If you are making minimal medium plates with xylose as the only carbon source, then you shouldn't need TTC. Any colonies that grow should be using xylose as sole carbon source. You can make some identical plates without xylose as a control to show the colonies growing on the xylose plates do not grow on the non-xylose plates.
I have seen with other substrate this is not true and further Congo red is used. I would further printout that its harder to find papers with proper explanation in them.
It should be true, what sort of medium are you using? Generally the reason yiu have an indicator dye like TTC is for use on rich complex medium like nutrient agar or LB. But on real minimal medium it should not be needed
So you are using a rich medium where everything will grow. The yeast extract and peptone can both serve as carbon (energy) sources.
You need to try a synthetic minimal salts medium where the added xylose is the only carbon source if you want to directly select for xylose utilizing microbes. For example M9 salts is a common one used for E. coli but there are many other there.