To estimate heritability, you must test genotypes with known relationships (such as parent-offspring, half-sib, etc.). Are the lines that you tested related in such a way?

Unfortunately, all of lines that I use are F5 population. However, two of seven elders of this population, I planted as a check. Is such genetic material can be used to estimate the heritability?

First, you need estimate the variance componets based on the appropriate mean square terms from the experimental desing. Please see: 

http://articles.extension.org/pages/60430/introduction-to-the-augmented-experimental-design-webinar

Good luck.

Aurelio, that is a great webinar by a fantastic professor! Heritability can certainly be estimated from variance components which can themselves be estimated using an augmented design.

Apri, When you say that your checks are "elders of this population", what do you mean specifically? What generation are they? How are they related to the F5 lines in your experiment?

With data from only the F5s and checks, I believe that the best you can do in this situation is to estimate heritability in the broad sense (genetic variance / phenotypic variance).

In order to estimate narrow-sense heritability, you would need an estimate of the additive genetic variance. I am unaware of a method for such an estimation using F5 pedigree lines. Fortunately, in your highly homozygous F5 population, the effects of dominance will be diminished; however, dominance will still be present.

I would caution that your estimate of heritability may not be very accurate, and thus may not be very useful. As Jennifer Kling states in her webinar, "Unreplicated experiments are inherently imprecise, no matter how sophisticated the design." There is no substitution for replication.

Best of luck with your research!