I am trying to build a joint classifier for Bimodal Sentiment Analysis which takes two modalities(audio and video files) as inputs. Any suggestions, how can I concatenate the below audio and video features to train a CNN based deep learning models?
Audio features:
X_aud = np.asarray(aud_data)
y_aud = np.asarray(aud_labels)
X_aud.shape, y_aud.shape
((1440, 40), (1440,))
Video features:
X_img = np.asarray(image_data)
y_img = np.asarray(img_labels)
X_img.shape, y_img.shape
((11275, 256, 512, 3), (11275,))
Any help would be highly appreciated. Thanks In Advance!
An initial step might be to scale the values to lie in the range (0,1), normalize the variance of the distribution by dividing by the covariance matrix , concatenate the resulting feature vectors and feed as input to the neural network
I would suggest starting with LSTM encoder for the temporal dimension of both inputs. For the fusion within a convolutional neural network I recommend a similar architecture used in this paper:
Conference Paper A Deep Learning-based Radar and Camera Sensor Fusion Archite...
This architecture considers multiple fusion levels for inputs that are different in their level of abstraction and information.
- Badges
- Science topic
- Similar topics
- Computer Science
- Images