Is it possible for elliptically polarized excitons to emit photons of similar angular momentum which isn't an integer multiple of the Plank's constant?
No, the spin of a photon is always equal to 1 in units of Planck's constant.
Elliptic polarization simply means that the intensities of the two polarization states aren't equal, it doesn't mean that there aren't exactly two states.
There aren't any caveats and it's not necessary to have a beam. A single photon can have unequal probabilities for being in one of the two states of polarization.
It is easy to show that a circularly polarized photon carries ħ of angular momentum with a specific rotational direction (right or left-handed). However, what about linearly or elliptically polarized light? I will attempt to show that even linearly polarized photons must carry ħ angular momentum with a specific rotational direction (either right or left-handed). We will do a thought experiment. Suppose we imagine a carbon monoxide molecule (CO) isolated in the vacuum with no external forces acting on it. The CO molecule is a simple diatomic molecule with charge separation. One atom has positive charge and the other atom has negative charge. This gives the CO molecule an infrared spectrum which is easy to interpret. The molecule can vibrate and rotate. When it rotates, its rotation is quantized. The first allowed rotational frequency is 115 GHz and this corresponds exactly to ħ angular momentum. The second allowed rotational frequency is 230 GHz, and this corresponds to 2ħ quantized angular momentum. Much higher quantized harmonics of this fundamental rotational frequency are also observed corresponding to larger multiples of ħ.
Suppose a CO molecule is rotating with 2ħ angular momentum. Its rotational axis is designated the Z axis and the rotational direction is right-handed in the positive Z direction. If it emits a photon in the positive Z direction, that photon will be right-handed circularly polarized. The CO molecule loses ħ of right-handed polarized angular momentum and the rotational rate drops from 230 GHz to 115 GHz. There is conservation of angular momentum.
Now we will repeat the experiment but the CO molecule with 2ħ angular momentum emits a photon perpendicular to the rotational axis (in the X-Y plane). The CO molecule again loses ħ of right-handed angular momentum, but this time the emitted photon must be linearly polarized in the X-Y plane. A rotating electrical dipole has a specific emission pattern. A rotating dipole must emit circularly polarized photons in the polar direction and linearly polarized photons in the equatorial plane. Elliptical polarized photons are emitted in other directions.
Now we have a problem because the CO molecule must lose ħ of right-handed angular momentum, but the linearly polarized photon does not appear to be carrying any specific rotational direction of angular momentum. However, the linearly polarized photon is carrying ħ of right-handed orbital angular momentum. Orbital angular momentum is present when the wavefront is not perpendicular to the propagation direction.
We can check this by continuing the thought experiment. Suppose there is a mirror which reflects the linearly polarized photon back to the same CO molecule. If the molecule absorbs the photon again, the rotation must increase from 115 GHz to 230 GHz, with the same rotational direction. This is a requirement to conserve angular momentum. Therefore, the linearly polarized photon must impart angular momentum with the correct rotation direction.
A previous answer to this question said, “a single (elliptically polarized) photon can have unequal probabilities for being in one of the two states of polarization.” This is wrong. There must be conservation of angular momentum. Orbital angular momentum solves this problem.