Swin-Transformer transform the image to tokens to input to transformer.
Is each token (before-embedding) value an integer?
In practice, where is this done? https://github.com/microsoft/Swin-Transformer
The code `self.head = nn.Linear(self.num_features, num_classes)` seems not output integers?
In this context, the token embeddings are typically initialized as real-valued vectors. These embeddings are learned during the training process of the model and can take on real number values, not necessarily integers.
A patch of a non-overlapping image is first divided into and then each piece is flatten and linearly created to form a sequence of visual tokens. These token values are not integers but are continuous stochastic representations and the linear embedding in the patchembed module in the Swin Transformer codebase and here is the implementation of the linear embedding.
self. head = nn. Once the linear(self. num_features, num_classes), this is a fully connected layer that is used for the final classification head. The output of this layer is not always binary values. The values it generates depend on the input it is given and the weights of the layer it has. In the case of classification, the output of the layer is usually passed through the softmax function, which in turn results in a probability distribution over the classes.
- Badges
- Science topic