I don't think is this straight-forward unless the sets of ratings can be considered independent. In that case it would be fairly easy to bootstrap and a normal approximation may also be OK in largeish samples.
For the normal approximation you'd get the SE for each kappa coefficient (e.g., see https://www-users.york.ac.uk/~mb55/msc/clinimet/week4/kappa_text.pdf ) and use that to obtain the SE for the difference in kappa:
SEdiff = (SE1^2 + SE2^2)^0.5
Then you can compute, say, the 95% CI for the difference in the usual way: