I sometimes believe that olfaction actually represents a basic form of affective processing. Has olfaction maybe always been the earliest form of an emotion system (evolutionary start of an emotion system)? Smelling things is so obviously and directly related to a "feeling" of pleasant or unpleasant, much more than it is related to "knowledge" about its origin (or semantic content).
The sense of olfaction has the most direct connections with the limbic system (e.g. amygdala) among all senses. Thus, neuroanatomy would strongly support this idea too.