I have worked with Google's Word2Vec model. It works in a way that is similar to deep approaches such as recurrent neural nets or deep neural nets, but it implements certain algorithms, such as hierarchical softmax, that make it computationally more efficient. I have found many articles online saying that it is deep learning and some others with contrary view. Looking at its features and as its presented by Google and Kaggle.com, I would have arguably agree that its a super deep learning but considering how it works, I will reason otherwise.
Please what is your view about Word2Vec model?
Your comments will be appreciated
Dear Chinedu Nwoye ,
The term "deep learning" was coined in 2006, and refers to machine learning algorithms that have multiple non-linear layers and can learn feature hierarchies [1].
Most modern machine learning relies on feature engineering or some level of domain knowledge to obtain good results. In deep learning systems, this is not the case -- instead, algorithms can automatically learn feature hierarchies, which represent objects in increasing levels of abstraction. Although the basic ingredients of many deep learning algorithms have been around for many years, they are currently increasing in popularity for many reasons, including advances in compute power, the falling cost of computing hardware, and advances in machine learning research.
https://www.kaggle.com/c/word2vec-nlp-tutorial/details/what-is-deep-learning
Regards, Shafagat
No, Word2Vec is not a deep learning model, it can use continuous bag-of-words or continuous skip-gram as distributed representations, but in any case, the number of parameters, layers and non-linearlities will be too small to be considered a deep learning model.
See for example the diagram in:
Rong, Xin. "word2vec parameter learning explained." arXiv preprint arXiv:1411.2738 (2014).
Where you can see that just one encoder-decoder matrix W is learned.
Also, you can check Christopher Manning's lecture on Word2Vec here:
https://www.youtube.com/watch?v=ERibwqs9p38&list=PL3FW7Lu3i5Jsnh1rnUwq_TcylNr7EkRe6&index=2
Cheers,
Sebastian
"Depth of architecture refers to the number of levels of composition of non-linear operations in the function learned. Whereas most current learning algorithms cor
respond to shallow architectures(1, 2 or 3 levels), the mammal brain is organized in a deep architecture with a given input percept represented at multiple levels of abstraction" [1]
I could also argue based on my own research that an additional factor is based on the initial research carried out in NN's where using two hidden layers was computationally expensive (especially using second order methods). This led the scientific community to focus on single hidden layer architectures and naming multiple hidden layers as deep. While Hiinton's discovery has propelled the deep architecture paradigm, new research on GPU's have also facilitated the deployment of deep architectures.
[1] Learning Deep Architectures for AI by Yoshua Bengio
word2vec itself is a simple bi-layered neural network architecture, it turns text into meaningful vectors form that deeper networks can understand. In other words the out put of simple neural word2vec model is used as input for Deep Networks.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Artificial Intelligence