How to concatenate the visual feature vector and the textual feature vector to get the final feature vector?
Suppose I have a dataset of different objects (Image, and its label).
1. I read both the things separately.... (image+label),
2. Extracted visual and textual features using CNN
3. Now I want to concatenate both the features vectors into single vector for classification??
Convert the two different vectors into linear or scalar format. Then just concatenate them.
You just need to concatenate them and then if you want you can apply PCA or other feature selection algorithm
Dear Azhar Imran
Before you do the concatenate both feature vectors, you should to check two feature vector. The two feature vector could be concatenated directly if they were from the same numerical range (homogeneous). However, if they were not from the same numerical range (heterogeneous), they should be normalized by using normalization techniques to transform the feature vector into common domain prior to concatenate them.
For achieved the good results my suggestion to do feature selection to select the optimal feature from both feature vectors and then do concatenation of those optimal features.
There are different ways to merge two features. The one that the majority goes with is the horizontal concatenation. You compute all the textural and the visual features ( shape or color ), concatenate them horizontally, and then normalize the vector.
The other way to goes with is bag-of-visual-words, you just need to stack the features to build a dictionary of words.
To merge two feature vectors, you first need to check if they are from the same scale. If not, you need to normalize them before to concatenate. Features selection also can be useful to select some optimal features. In other hand, you can build two different models and make a fusion from the score of each model. It depends on the task you are doing.
Best regards,
This paper discuss various mechanisms to combine (fuse) features of two modalities. https://research.fb.com/publications/efficient-large-scale-multi-modal-classification/
In addition, please take a look at the following works.
Article Learning Fused Representations for Large Scale Multi-modal C...
Preprint Image and Encoded Text Fusion for Multi-Modal Classification
Conference Paper Multimodal Classification Fusion in Real-World Scenarios
Here is a paper that fuses image features extracted from a CNN and textual features:
https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8538003
Article Multi-Modal Multi-Scale Deep Learning for Large-Scale Image Annotation
- Badges
- Science topic
- Similar topics
- Computer Science
- Images