See: Laurer M, van Atteveldt W, Casas A, Welbers K. Less Annotating, More Classifying: Addressing the Data Scarcity Issue of Supervised Machine Learning with Deep Transfer Learning and BERT-NLI. Political Analysis. 2024;32(1):84-100. doi:10.1017/pan.2023.20

Training deep learning models to do well with limited data (few-shot learning) or to make accurate predictions in situations they’ve never encountered before (zero-shot learning) is a tough challenge. However, there are several strategies that researchers and engineers use to make this possible. Here’s a more approachable breakdown:

1. Start with Something Pretrained

- Pretrained Models: Imagine you’re learning a new skill—if you already know something similar, you’ll learn faster. The same goes for models. Starting with a model that’s been trained on a large, general dataset and then fine-tuning it with your specific, smaller dataset can help it perform better with fewer examples.

- Feature Extraction: You can also use the pretrained model to pull out useful features from the data and then train a simpler model on top of those features, making it easier to work with less data.

2. Teach the Model How to Learn

- Meta-Learning: This approach is like teaching the model how to learn new tasks quickly. You train the model on a bunch of different tasks so that when it encounters a new one, it can adapt quickly with just a few examples.

- Simpler Approaches: There are also simpler methods that focus on learning good representations of data, so the model can generalize better across different tasks with minimal data.

3. Create More Data or Make the Most of What You Have

- Data Augmentation: This is like taking what you know and imagining variations of it. For example, flipping or rotating images to create new training examples. This helps the model see a wider variety of examples without actually needing more real data.

- Synthetic Data: Sometimes, we generate completely new data using models that are designed to create data similar to what we already have, effectively expanding our training set.

4. Let the Model Learn on Its Own

- Self-Supervised Learning: Here, the model learns by solving puzzles that it creates from the data itself, like predicting missing parts of an image or sentence. This helps it build useful knowledge without needing labeled data.

- Contrastive Learning: The model learns by comparing things that are similar or different, which helps it understand the structure of the data and generalize better.

5. Prevent the Model from Overfitting

- Regularization Techniques: When working with limited data, it’s easy for a model to just memorize the examples instead of learning to generalize. Techniques like dropout (randomly turning off parts of the model during training) or mixing data points help the model stay flexible and avoid overfitting.

6. Use Special Architectures for Few-Shot and Zero-Shot Learning

- Siamese Networks: These networks are designed to compare pairs of data points and are great for recognizing whether new examples are similar to what the model has seen before.

- Transformers and Attention: These models, especially in natural language processing, can be prompted with specific instructions or examples, making them powerful for zero-shot tasks where they haven’t been trained directly.

7. Incorporate External Knowledge

- Knowledge Graphs: Imagine giving the model a cheat sheet about how different concepts are related. This extra context can help it make better predictions in unfamiliar situations.

- Language Models: Large models like GPT have been trained on vast amounts of text, so they "know" a lot about the world. You can use this knowledge by fine-tuning them for specific tasks, even with little or no new data.

8. Adapt to New Domains

- Adversarial Training: This approach helps the model learn features that work across different domains, so it’s better equipped to handle new environments.

- Domain-Invariant Features: The idea is to teach the model to focus on features that don’t change much across different settings, making it more robust.

9. Handle Zero-Shot Learning

- Embedding Methods: By mapping both the data and labels to a common space, the model can make predictions for new classes based on how close they are in this space to what it has already seen.

- Attributes: Instead of predicting a class directly, the model predicts a set of attributes like colours or shapes), which can then be used to identify new classes.

By using these techniques, you can train models to perform better with limited data and even tackle completely new situations. Often, the best results come from combining several of these strategies to make the most of the data and knowledge we have.

Nisha Bhadauriya Agarwal

Both few-shot and zero-shot learning are active research areas with many exciting developments. Training deep learning models to generalize from limited data or make predictions in novel situations involves following techniques:

Few-shot Learning

• Transfer Learning
• Meta-Learning
• Data Augmentation

Zero-shot Learning

• Generative Models
• Self-Supervised Learning
• Semantic Embeddings

Few-Shot Learning (FSL)

Objective: Learn to perform new tasks with only a few labeled examples.

1. Meta-Learning ("Learning to Learn")

• Idea: Train the model on a variety of tasks so it learns a shared learning strategy.
• Popular Algorithms: MAML (Model-Agnostic Meta-Learning): Learns a good initialization that can be quickly adapted to new tasks with few gradient steps. Prototypical Networks: Learns a metric space where classification is performed by computing distances to prototype representations of each class. Siamese Networks: Compares input pairs and learns a similarity function, useful for verification tasks.

2. Data Augmentation

• Use domain-specific transformations, GANs, or self-supervised learning to create more training examples.
• Example: Rotating, cropping, or color-jittering images in computer vision tasks.

3. Transfer Learning / Fine-Tuning

• Approach: Pretrain a model on a large dataset, then fine-tune it with the limited target task data.
• Example: Using a BERT model pretrained on a general corpus, and fine-tuning it on a small dataset for text classification.

Zero-Shot Learning (ZSL)

Objective: Make accurate predictions for tasks or classes not seen during training.

1. Semantic Embedding / Attribute-Based Models

• Map both inputs and class labels to a shared semantic space (e.g., word embeddings or attribute vectors).
• Use distance/similarity in this space for inference.
• Example: For classifying animals, use attributes like "has wings", "can fly", etc., to represent unseen classes.

2. Pretrained Language-Vision Models

• Leverage large-scale models like: CLIP (OpenAI): Aligns text and image embeddings using contrastive learning. Can recognize classes by comparing image features with text prompts. GPT / T5 / BERT: Can perform ZSL through prompt-based learning or instruction tuning.

3. Prompt Engineering / Instruction Tuning

• Use natural language prompts to describe new tasks or classes.
• Example: Ask GPT-4 "Classify this email as spam or not spam" without fine-tuning—just prompt engineering.

Cross-Strategy Techniques

1. Self-Supervised Learning: Train models to predict parts of the data from other parts (e.g., masked tokens, image patches). Enables learning powerful representations without labeled data.

2. Contrastive Learning: Learn to distinguish between similar and dissimilar pairs. Improves generalization by clustering semantically similar data in latent space.

3. Knowledge Distillation: Transfer knowledge from a large "teacher" model to a smaller "student" model, which can then perform well on limited data.