.

unsure what you mean by "unique"

.

if two documents differing by one word only are considered two unique documents, hashing techniques will provide an easy way

for instance, hash any word to a 0/1 sparse string, OR-accumulate all the hashes of the words in the document to get the "signature" of the document and compare to the signature of the documents of the database ; this scales linearly with the size of the database and it is possible to speed-up by structuring the signature space

.

now if unique means "thematically unique", the above will obviously fail !

.

Fabrice Clerot thanks for response,

unique means " if two documents differing by one word only are considered two unique documents, " satisfactory ...

Hi Sharma,

with this shallow requirement you could just use some MD5 check sum to figure out whether a document is unique.

More interesting is of course the question of "how to identify dublicates within large amounts of documents?". For that I would use minHashing & Locality Sensitivity Hashing, described in "Mining Massive Datasets".