Inverting and non-inverting are the two modes available in a general op-amp. Thus, it affects only the polarity of the output voltage in reference to the differential voltage 'V-' subtracted from 'V+'. The speed is determined by the interfacing circuit between the input of the op-amp and the output. That means the modes could be equal in speeds if the said interface is same for both but will differ in the output polarity.
I suppose that in both cases - inverting and non-inverting - the same gain values shall be used. In this case, the "speed" depends on the bandwidth of the closed loop. For equal gain magnitudes the bandwidth of the inverting configuration always is somewhat smaller than for the non-inverting configuration. For large gains the difference can be neglected - but for small gains (5 and below) it can be recognized.