This is a great question! An email server detects spam by using spam filter software which evaluates incoming emails on a number of criteria. (Yes, you can run an email server without having spam filter software enabled - you'd just see any and all spam email.)

Now how do anti-spam software/services detect spam? They primarily utilize different methods including content-based message encoding and keyword filtering rules, email authentication rules (if the sending server has SPF/DomainKeys/DKIM/Sender ID enabled), mail server IP blacklists, and domain blacklists for sending domain and email content.

When a new email arrives on your mail server, it is initially evaluated against theIP and domain blacklists and for email server authentication status, and then for the email content rules. Usually, if the email scores higher than a preset criteria (for most services, an administrator can set scoring criteria) the email is marked as spam and dealt with appropriately.

Note: To reduce server load, spam filtering services will outright reject email that arrives from IPs or domains on blacklists (this is why ensuring the sending IP and all domains are not on blacklists is so important). They also might reject or delay email if you are an unknown email sender violating email sending limits (this is called greylisting). These tactics are designed to relieve email servers from abusive email spammers who try to send as many spam emails as they can in a short amount of time.

The most common spam filtering software is SpamAssassin, and many other popular spam filtering software/services use SA as a primary source. Other popular spam filtering services include Barracuda (hardware device based), Cloudmark, and a variety of MS Exchange based products.