Our group is working on a mutant which heterozygote shows much more severe mutated phenotype while the homozygous mutant shows less difference to the wild type. What theory can we use to explain this, If anybody can give us some advice?
You are probably dealing with a dominant-negative mutation. It usually works by that the mutated gene-product dimerizes with the wild-type and causes an adverse effect.
For example, if a protein is supposed to form a heterodimer, the mutated allele can be incorporated because it can still "hitchhike" on the wildtype. To gain a protein with a new/adverse effect is often more deleterious than a total lack of the protein, explaining why the homozygous mutant shows a less severe phenotype.
You are probably dealing with a dominant-negative mutation. It usually works by that the mutated gene-product dimerizes with the wild-type and causes an adverse effect.
For example, if a protein is supposed to form a heterodimer, the mutated allele can be incorporated because it can still "hitchhike" on the wildtype. To gain a protein with a new/adverse effect is often more deleterious than a total lack of the protein, explaining why the homozygous mutant shows a less severe phenotype.
Yes I agree with the explanation given by Elika. In simple words it is better not to have the protein (the homozygous mutation) than having defective protein (heterozygous mutation)
For a dominant negative effect, as described above, your mutant must make some protein, such as a truncation or point mutation. If you cannot detect the mutant form of the protein, the dominant negative theory is hard to justify.
Elika and Gregory made some good point: Especially, you need to have a function as higher order complexes with at least two different proteins that are involved in executing the function.
To give only one explanation: I guess you made some backcrosses and you probably introgressed accidentally a slightly different allel of the second (unknown) partner protein - or even of the very same protein. Hence, the two allels you are working with full-fill an incpompatible function that e.g. if you are dealing with DNA-binding protein complexes (e.g. MADS-box proteins) will target the wrong binding motifs. A loss-of-function might be compensated by physiological plasticity..
There are reports on stuff like this - although I didn't find any good quote by a rapid search - especially from bacteria, yeast and plants. I can't remember about anything like that from animals.
Ellika and Dierk explained the dominant-negative mutation clearly, and Gregory gave us a hint to analysis the protein function.
But the trick thing is the homozygous mutant is more stable, maybe because the protein is dimerized or polymerized, we would carry out experiment to solve that.
This condition seems to be explained by overdominance theory. Overdominance is where the hetrozyotes show extreme phenotype than either of the homozygotes. Your case seems the mutant is dominant and the heterozygote manifests this dominant character to one extreme.
This is a more general model at the phenotype level. Overdominance could have various molecular mechanisms which may depend on the type of mutant product or the pathway in which the gene product is involved or interaction among different genes.
The simple dominant-negative effect as outlined by Ellika explains well situations where a missense mutation is more severe than a null-mutation, both in heterozygous states. This is well described in several human diseases like osteogenesis imperfecta.
It doesn't really explain why the same mutation (!) in heterozygous state causes a more severe phenotype than in homozygous state, which is what I think Wilson described. This is rare at least in human genetics, don't know about plant genetics. One example that came to my mind in human genetics is craniofrontonasal syndrome, but this is an X-linked disease (heterozygous females more severely affected than hemizygous males) and the explanation is more on a cellular level (due to X-inactivation, the females are mosaic for wt and mut cells, and the cellular interaction between the two populations causes the severe phenotype).
One theory here could be that the gene product homodimerizes. The wt/wt-Dimers are stable, the mut/mut-Dimers are (at least somewhat) stable and functional, but the wt/mut-Dimers are not stable/functional. With heterodimers, this doesn't really work out. Why should the heterodimers work better if you add more molecules of your mut-partner?
Any explanation will need to look closely at the function and mode of action of the wt and mutated product. I do agree that a simple null mutation seems unlikely, however.
I think in this case the model of overdominance can not be justified. Usually, gene mutation, indicating overdominance caused by the presence of recessive lethal alleles, which remain in the population due to benefits of heterozygotes. For example, known system of lethal alleles of lethal giant larvae. Heterozygotes with normal and mutated variant of gene are often characterized higher viability.
So explanation by dominant-negative effect is the most acceptable.