Perplexity

.

if you have a bilingual english-"your studied language" available (in electronic form ...) you might adapt the techniques developped in

http://www.aclweb.org/anthology/D15-1036

.

The basic idea is that semantic vectors (such as the ones provided by Word2Vec) should preserve most of the relevant information about a text while having relatively low dimensionality which allows better machine learning treatment than straight one-hot encoding of words. Another advantage of topic models is that they are unsupervised so they can help when labaled data is scarce. Say you only have one thousand manually classified blog posts but a million unlabeled ones. A high quality topic model can be trained on the full set of one million. If you can use topic modeling-derived features in your classification, you will be benefitting from your entire collection of texts, not just the labeled ones.

Getting the embedding

Ok, word embeddings are awesome, how do we use them? Before we do anything we need to get the vectors. We can download one of the great pre-trained models from GloVe:

The (python) meat

We got ourselves a dictionary mapping word -> 100-dimensional vector. Now we can use it to build features. The simplest way to do that is by averaging word vectors for all words in a text. We will build a sklearn-compatible transformer that is initialised with a word -> vector dictionary.

These vectorizers can now be used almost the same way as CountVectorizer or TfidfVectorizer from sklearn.feature_extraction.text. Almost - because sklearn vectorizers can also do their own tokenization - a feature which we won’t be using anyway because the benchmarks we will be using come already tokenized. In a real application I wouldn’t trust sklearn with tokenization anyway - rather let spaCy do it.

Now we are ready to define the actual models that will take tokenised text, vectorize and learn to classify the vectors with something fancy like Extra Trees. sklearn’s Pipeline is perfect for this:

Sharing link:

http://nadbordrozd.github.io/blog/2016/05/20/text-classification-with-word2vec/

You have to pass a weight matrix to the Embedding layer. Here is an example:

Let's say index_dict is a dictionary that maps all the words in your dictionary to indices from 1 to n_symbols (0 is reserved for the masking).

So, an example index_dict is the following:

{ 'yellow': 1, 'four': 2, 'woods': 3, 'ornate': 31, 'woody': 5, 'cyprus': 6, 'marching': 7, 'canes': 8, 'caned': 9, 'hermann': 10, 'lord': 11, 'meadows': 12, 'shaving': 13, 'swivel': 14 ... }

And you also have a dictionary called word_vectors that maps words to vectors like so:

{ 'yellow': array([0.1,0.5,...,0.7]), 'four': array([0.2,1.2,...,0.9]), ... }

The following code should do what you want

# assemble the embedding_weights in one numpy array vocab_dim = 300 # dimensionality of your word vectors n_symbols = len(index_dict) + 1 # adding 1 to account for 0th index (for masking) embedding_weights = np.zeros((n_symbols, vocab_dim)) for word,index in index_dict.items(): embedding_weights[index, :] = word_vectors[word] # define inputs here embedding_layer = Embedding(output_dim=vocab_dim, input_dim=n_symbols, trainable=True) embedding_layer.build((None,)) # if you don't do this, the next step won't work embedding_layer.set_weights([embedding_weights]) embedded = embedding_layer(input_layer) # ... continue model definition here

Note that this kind of setup will result in your embeddings being trained from their initial point! If you want them fixed, then you have to set trainable=False.

You can use on of the performance evaluation methods, such as the Word error rate (WER) or Preplexity (PPR).

Intrinsic evaluation using semantic similarity or by using extrinsic evaluation which evaluates the model using NLP applications

Because you are testing low recourse language, the intrinsic evaluation will not be easy. Assuming you don’t have a analogy set to test nor semantically classified words, then the best option is to use the word embedding in an NLP application, that is an extrinsic model evaluation. But the question will be “what would be the cheapest NLP application that can be applied easily with low resource language!” . I suggest class based language model perplexity, where word embedding classes are used. This is not a complete NLP application but rather developing and testing the class based model to be used later for NLP application. Also I would suggest trying to enlarge your data by combining it with other dominant language that have vocabulary intersections if this is applicable.