It is not clear what you are after. It is easy to write a script to extract the terms for yourself (or I am sure you can find one in the NLTK kit or something similar). Do you want the root form of the terms (e.g. "sing" for "sang") or will stemming do? The first is a little hard, though you can use a part-of-speech tagger and Wordnet to help you. The second is easy using Porter stemmer code (though without part-of-speech it can be wrong sometimes, e.g. "building").

If you want to exclude the stopwords from the vocab file, why not just sort them both alphabetically and run a difference script to compare them?

Another thought might be to use tfidf (term frequency, inverse document frequency) to score each term according to each newsgroup. That should minimise the influence of stopwords and common terms and help identify terms that are more common/important to a particular group than other newsgroups.

Thanks Michael very much.

I have use Prorter steemer code and stopword to filter the vocabulary corpus. But what annoy me most is many words like 'pb, se, ns, vm, ...', and so on, are still contained in the vocabulary corpus. I don't quite understand the meanings of those words, maybe because my native language is not English.

I don't know them from the 20 Newsgroups corpus, which I have used. They don't mean anything to me as abbreviations for anything in particular (unlike 'cf', 'etc', 'ie' which relate to Latin terms used in English). I had a quick look in the corpus for 'pb', 'se', 'ns' and 'vm' but couldn't find them, except for "per se" which is Latin for "by itself". Can you give examples? Has your tokenising divided terms like ns1 or web addresses  like ericsson.se?

I download the 20 newsgroups of Matlab/Octave dataset(http://qwone.com/~jason/20Newsgroups/20news-bydate-matlab.tgz) and the vocabulary for the indexed data(http://qwone.com/~jason/20Newsgroups/vocabulary.txt) from the official website.

Then I shrink the dimension of words for input data from 53975 to 5000, by filtering out the words with low frequency. After that, there still some abbreviations of words like 'pb', 'se', ...(Seeing in Ps), in that corpus. So I think there are somethings still need to be done to make  this corpus more meaningful.

Any other methods will help me out? Or if it is appropriate for you to share your relevant works or methods with me?

Ps:

the abbreviation  words are list as blow(the top 200 words(length==2), order by its frequcency),

ve ca ll cs mr uk mb st ac pc de cc hp db ms au os al cd ed la dr tv ii ma se gm dc pa pp ny fi ps dx cb id nl xt ad nj nt pm ns em hd ch um il bu ra md si du oz mi po bc ai ti pt br ab lc bh tm uu sw di gt tx wa va aa rs hr km ei uh mm sp nz jr ee el tu op io bd ld sf ha wd ux ph sg sc mc sx kb es rc ah hz rm dg gl ne ip mu xv ql cx ap vt ss mg eh sb ga er mx ds iv ta rz gb tb fr sh ya wc mt pd ic jp gs tt sj ye bm rf cl bs da nc bb le fm wb xx ot sl lw je en su xm lq mo wg ba mp cf dl cr bi ft te qu rt sr gc za ge eu kg ls ut zx dd im vl ct cg dk vm rr nh dy oo nm kk ff sq pl mk hc bx jb pb

I think that the problem is that urls have been divided up into their individual elements. Therefore "se" is actually from web addresses from Sweden (.se). Similarly "jp" is for Japanese websites (.jp) and "fr" is from French ones (.fr). This is the problem with not doing the tokenisation yourself - you don't control what is considered a term.  While you could search for a list of all these url terms and use it to filter your list, there will be some abbreviations that are actually valid, e.g. "tv" for "television" and "uk" for "U.K.", or the United Kingdom of Great Britain and Northern Ireland and "mr" for "Mr." or Mister. Personally, I think you would be best finding software that enables you to do the tokensiation yourself and modify it to match what you consider to be a word. Have a look at sites like http://www.nltk.org/ (Python scripts).

I am afraid that while my work on the 20 newsgroups did include tokenisation, my output was specialised for my research. I can tell you though that my tokeniser was based on the tokeniser.sed that was used for the Penn Treebank - http://www.cis.upenn.edu/~treebank/tokenizer.sed .

Dear Michael, thanks for your explanation.

I have checked the NLTK and the script from Penn Treebank provided by you, and done some research about tokensiation online.

I think it is not necessary to do the tokensiation by myself. I just need shrink the dataset(http://qwone.com/~jason/20Newsgroups/vocabulary.txt) by matching it with a-priori dictionary, which will filter those abbreviation words automatically. Is that appropriate?

But I think the best way to handle this problem is replacing those abbreviation words by its real meaning, for example, replacing 'se' by sweden, 'tv' by television etc, rather than just deleting those words without any consideration.

That certainly is one way. Watch out for terms like "us". Unless you know its original usage, you won't know whether it was "U.S." or "US" or "us" (i.e. first person plural pronoun which will be a stopword).

OK, thanks for your reminder. It's very nice of you.