Why increasing frequency?

Shannon's equation provides the theoretical best channel capacity that one can achieve, based on the width of the channel and the signal to noise ratio (for a single propagation path).

Capacity (b/s) = Bandwidth (Hz) * LogBase2(1 + S/N)

where the signal to noise ratio, S/N, is expressed as a power ratio, not in dB.

As you can see, channel capacity, in bits per second, for any given signal to noise ratio, is directly proportional to channel bandwidth. This is the reason that higher frequencies are beneficial.

Said another way, it is certainly feasible, in the 5 GHz band, to create channels that are, say, 80 MHz wide. Can you create 80 MHz wide channels in the Medium Wave band, say between 500 kHz and 1600 kHz? Impossible. If you could, those channels would have the same capacity, whether MW or SHF. But such wide channels don't fit in the lower frequency bands.

As to attenuation, it's true that objects or bad weather attenuate the higher frequencies more than the lower frequencies. The higher the frequency, the less signals diffract. So RF begins to behave like light, as frequency rises.

But in free space, the problem of attenuation is quite different. The reason higher frequencies appear to attenuate more, in free space, is artificial.

Think of antennas as being devices that collect photons. If you normalize your propagation equations based on unity antenna gain, sure enough, higher frequencies don't appear to travel as far. But that's just because the unity gain antennas are physically smaller at higher frequencies than at lower frequencies. Meaning, they collect fewer photons. Instead, if you consider equal size antennas, let's say with dish reflectors, you will notice that when designed for the higher frequencies, these antennas will provide substantial gain, which compensates for the supposedly greater attenuation of higher frequencies.

This is the equation that determines the gain of a parabolic antenna:

Gain = 10 * LogBase10(efficiency * (pi*diam)^2 / lambda^2)

Notice how diameter and wavelength work in opposite directions, when you're calculating the gain. This is the issue, in free space, where you don't have obstacles or weather-related effects to cause increased attenuation. Antenna gain is needed, all else equal, to compensate for the fact that small unity gain antennas do not collect as many photons as large unity gain antennas.