There is several limiting factors. First of all, those are the configuration and the plate material of the acousto-optic modulator. If the co-propagating (or counter-propagating) SAW to waveguide configuration is used then the frequency is determined by the SAW velocity for selected material and the difference between wave vectors of the converted modes. Also those factors determine the IDT period. Effective modulation is possible in small frequency range and usually the modulation frecuency is about several hundred MHz. If perpendicular propagating SAW to waveguide configuration is used then frequency is determined by the technological capabilities of creating IDT (however higher SAW velocity provides creating not too small period of IDT for the high modulation frequency). This configuration provides modulation frequency up to several GHz but it's harder to create.
For a more specific answer you should select plate material and configuration.
Also see in the Tsai, Chen S., ed. Guided-wave acousto-optics: interactions, devices, and applications. Vol. 23. Springer Science & Business Media, 2013.
In more practical terms, I think one has to distinguish between the carrier frequency and the modulation frequency. The former is typically on the order of tens of MHz, and in my personal experience, you can obtain AM modulation frequencies up to about 10% of the carrier frequency. In order to obtain the maximum modulation frequency, you have to focus your optical beam to a spot size close to the wavelength of the acoustic wave inside the device. With a 40 MHz carrier frequency, I was able to obtain modulation frequencies in the single-MHz regime. For my application, it was sufficient to drive the AOM by a continuous sine wave signal. If you want to use the AOM in some kind of servo loop, you have to account for the acoustic delay between actuator and the interaction zone with the laser beam. For this kind of application, the effective bandwidth reduces by another factor 10 or so, i.e., you can only reach an effective bandwidth of a few hundred kHz.
The maximum modulation frequency depends on the center frequency of an acoustic optic modulator e.g. if center frequency is 80 MHz modulation can be 70 (negative)-80 (center)-90 (positive) so minimum is 0 on negative side hence the limitation. However its better to avoid extreme minimum and scan near the center for clear results. If scanning exceeds minima as 0 mentioned above beat notes are generated as below 0 cannot be achieved. Hence repetition of signal is observed.