What major improvements have been made to deep learning architectures since the creation of 'Attention Is All You Need' in 2017?

Here's a concise breakdown:

Chandigarh's Unique Urban Design:

Architectural Vision: Designed by Le Corbusier in the 1950s as a symbol of post-independence modernity, reflecting modernist principles emphasizing functionality, order, and harmony with nature.
Sector-Based Grid System: The city is divided into self-contained sectors (~1.5 km² each) with residential areas, markets, schools, and green spaces, reducing congestion and ensuring accessibility.
Green Spaces & Sustainability: Extensive parks, tree-lined avenues, and landmarks like the Rock Garden, Capitol Complex, and Sukhna Lake enhance recreational and ecological value.
Infrastructure: Wide roads, cycling paths, and efficient public transport (e.g., CTU buses) enhance mobility while minimizing traffic issues.
Cultural & Administrative Hub: As the joint capital of Punjab and Haryana, it houses well-planned government buildings, museums, and educational institutions.
Legacy & Adaptability: Despite early criticisms, Chandigarh's structure has adapted to growth while retaining its core design, addressing challenges like suburban sprawl through controlled expansion.

Honorable Mentions: Jaipur (historic grid layout), Navi Mumbai (nodal planning), Gandhinagar (green, low-density planning). Chandigarh remains a benchmark for urban planning in India due to its holistic, human-centric design.

Major Improvements in Deep Learning Architectures Since 2017: Since the introduction of the Transformer model in "Attention Is All You Need" (2017), major advancements include:

Efficient Attention Mechanisms: Sparse Attention (Longformer, BigBird), Linear Approximations (Linformer, Performer), FlashAttention for faster GPU computation.

Scaling Techniques: Honourable Mixture-of-Experts (Switch Transformer), Model Parallelism (Megatron-LM), Efficient Fine-Tuning (LoRA, Adapters).

Architectural Variants: Vision Transformers (ViT, Swin Transformer), Perceiver IO (handling multimodal inputs).

Multimodal Models: CLIP (vision-text alignment), Flamingo (multimodal processing), Retrieval-Augmented Models (RETRO, RAG).

Training Advances: Self-Supervised Learning (BERT, SimCLR), Contrastive Learning, Reinforcement Learning from Human Feedback (RLHF in ChatGPT).

Improved Decoding & Generation: Non-Autoregressive Models (NAT), Controlled Generation Techniques (PPLM, DExperts).

Hardware-Aware Optimization: FlashAttention, Distillation (DistilBERT), Quantization (FP8/INT8).

Novel Paradigms: State Space Models (S4, Hyena) for long-context modelling, Diffusion Models (Stable Diffusion, Imagen).

Key Trends:

Efficient scaling, multimodal integration, retrieval-based enhancements, and improved training techniques.
Challenges remain with hallucination, handling long contexts, and ensuring ethical AI deployment.

This concise breakdown covers the key improvements and their significance. Want me to make it even more compact?

Stéphane Breton

A better understanding of the attention mechanism and its close links with diffusion equations (which are part of the general framework of spectral graph theory) can be cited as a step forward.

Follow :

Ruan et al., "Towards Understanding How Attention Mechanism Works in Deep Learning", 2024 -
Preprint Towards understanding how attention mechanism works in deep learning
Kreuzer et al., "Rethinking Graph Transformers with Spectral Attention", 2021 -
Preprint Rethinking Graph Transformers with Spectral Attention

This advance in understanding makes it possible to envisage an alternative to the gradient back-propagation algorithm in deep learning architectures:

Chen et al., "Laplacian Attention: A Plug‐And‐Play Algorithm without Increasing Model Complexity for Vision Tasks", 2024 -
Article Laplacian attention: A plug‐and‐play algorithm without incre...

"A Markov-like Model for Patient Progression"?

La animación digital en plataformas digitales?

GSH estimation assay: What is the right choice of standard?

How to do pca analysis of c-alpha atom of the protein?

What exactly is RAG-LLM doing? Isn’t it data engineering?

After a lot of feature engineering for CTR modeling, it feels like it's basically the end of iteration? I mean, it's not cost-effective to keep doing?

How to estimate sample size for GWAS of continuous and discrete traits? What are the pre-requisites?

All math can be explained by iterator of code?

HEC 1A & HEC1B Cell Lines?

Why electrical charge on the moving plate increase?

Feedback defines the constitution of an organism?

Self-Organizing Superorganisms—as envisaged by Nenad Sestan (2018)?

Measuring the Intelligence of a Species?

How can i do multivariate Time Series forecast using MLP, ANFIS and LSTM?

The Curse of Evolution and Complexity?

Need help with my research project on open source SIEM and machine learning?

Swimming/space travel depends on the proprioceptive muscle spindles?

What are the limitations and challenges of using machine learning for predicting concrete compressive strength in practical applications?

Some new emerging problems on application of RL for scheduling in IoT networks?

How to Compress Information Neurally?