What is the experiment that showed that there are only three generations of particles in the SM? Can we expect one more generation? Are there constraints, this generation of particles expected to satisfy?
It was done by observing decay channels of neutral vector boson Z_0. The idea, in brief, is to count how many decay channels (Z -> particle +antiparticle) we have for quarks, charged leptons and comparing them with the decay channels for neutrinos. The experiment basically tells us the number of neutrino species.
The number of neutrino species quoted by LEP experiements is 3.00 +/- 0.08. If LEP results are combined with previous low energy data then the result is 2.92 +/- 0.05 The scattering process involved in this measurement is e+ e- --> neutrino + neutrino_bar + photon. This process has a large cross section at CM energies near the Z pole.
In ppbar colliders total Z width can be found from ratio of events for W+/- -> l+/- + neutrino and Z --> l+ l- . Using total Z width one can determine number of light neutrino species.
Also see:
"CP violation in the standard model with four families" A. Dutta and E.A. Paschos, Page 292, in the book CP violation edited by C. Jarlskog, World Scientific, 1989.
Oblique electroweak parameters constrain parameter spaces of a 4 generation standard model. See page 23 of the following thesis;
If you are interested in answers outside SM, I could advice you the following paper that describes the behaviour of elementary particles in the Non-Standard Model