if two documents differing by one word only are considered two unique documents, hashing techniques will provide an easy way
for instance, hash any word to a 0/1 sparse string, OR-accumulate all the hashes of the words in the document to get the "signature" of the document and compare to the signature of the documents of the database ; this scales linearly with the size of the database and it is possible to speed-up by structuring the signature space
.
now if unique means "thematically unique", the above will obviously fail !
with this shallow requirement you could just use some MD5 check sum to figure out whether a document is unique.
More interesting is of course the question of "how to identify dublicates within large amounts of documents?". For that I would use minHashing & Locality Sensitivity Hashing, described in "Mining Massive Datasets".