The reason is due to the different types of bonding between cellulose and starch. Cellulose has beta-1,4 bonds that are not digested by our enzymes (which can digest alfa-1,4 and alfa-1,6 bonds that are present in starch and glycogen). However, some bacteria from the gastrointestinal tract of human and other animal species can use cellulose as an energy source since they are able to break the beta 1,4 bonds. Actually, that is one of the concepts of fiber (carbohydrates that are not digested by the enzymes produced by the animal, being fermented by bacteria).
Cellulose is insoluble and forms largely crystalline, insoluble fibers, whose degradation requires a sophisticated set of enzymes (endo- and exocellulases, beta-glucosidases). All animals able to use cellulose efficiently do so by hosting ellulolytic microorganisms (bacteria and fungi) in their GI tract. The most efficient herbivores are the ruminants, who have a specialized stomach, the rumen, hosting such cellulolytic microorganisms that digest cellulose and associated hemicellulose (xylan mannan etc.). The microbial biomass and the metabolite thus generated low Mw fatty acids, lactic acid) are then digested and used by the animal.
When it comes to biological systems, stereochemistry is very crucial. Enzymes are highly selective and often discriminate a molecule when there is a minute change in the spatial arrangement of substituents. The central difference between starch and cellulose is the stereochemical configuration around the 1-4 glycosidic bond as earlier pointed out by Erico. While the monomeric units in starch are connected by alpha linkages, those in cellulose are connected by beta. This variation in orientation makes cellulose stronger and less soluble than starch as stated earlier by Pierre. Amylase for example which is an enzyme in humans responsible for breaking down polysaccharides (starch) into dimeric units is highly stereospecific toward alpha oriented polysaccharides and would not hydrolise the beta oriented cellulose.
All the above answers supply a detailed description. To be simple, we can digest starch (and glycogen) using alpha-amylases, as they hydrolyze alpha-1,4 and alpha-1,6 bonds. But we can not hydrolyze beta-1,4 linkages of cellulose. I.e. we do not have cellulase. Among vertebrates, cattle and other ruminants can use cellulose as food, as their rumen harbors bacteria and protists with the ability to secrete cellulase. Therefore, as potatoes have a lot of starch, we can use them as a source of glucose. But in grass, we find a lot of cellulose, means we can not consume grass, like potatoes!