please be more specific. Are you asking about differentiating between bremsstrahlung and isotopic nuclear sources? . Speaking generaly, the answer is: try to measure the x-ray output spectrum.
if you have unknown sealed source with unknown shield composition and you indicate that its is a beta emitter how you can identify the radioactive material type (Sr- Kr .. etc ) is there a away without opening the source because its a powerful beta emitter .
I would second on Cesar's approach. Different material would have a different diffraction spectrum. The frequency of the peaks of the diffraction spectrum of your measured X-ray output can be correlated with a published spectrum of atomic diffractions spectrum.
What makes things tricky is your have an unknown shield composition, which would absorb and diffract photons from the source material. You can find the rough geometry of the system by some non-destructive imaging system, like a X-ray CT or something similar. Once you know the geometry, you could apply Cesar's approach.