It must also be a random average as different fields have very different ideas about what an IF should look like: in Biology it might be 22 and in Education Studies 1.8 but that doesn't mean much because the two fields are not (often) interdependent. Also, the number of publications per year might impact on IFs and 2 or 5 year IFs make a difference. So when we broadly talk about IF we probably do average the concept in our heads, just so that we can share a common language. I feel, that most of us assume it IF is a political tool - it matters on your CV but not in your day-to-day research.
Nice answer Erik, I was thinking more of the IF calculation itself, the numerator and denominator. Is that an average? I liked your response all the same.