Parameter Efficient Pre-Training: Comparing ReLoRA and GaLore
Can we use parameter-efficient training methods and achieve PEFT-like efficiency gains during the pre-training stage too?
Posts by Year
2021
2024
BERT Busters: Outlier Dimensions that Disrupt Transformers
Models of BERT family are overall robust to pruning, but they have an Achilles heel: the outlier dimensions, without which the quality of the model drops sig...
2020
What Types of Generative Models Are There?
The world is filled with data. Can we learn from this data to generate something new?
Question Answering for Artificial Intelligence (QuAIL)
QuAIL is a new challenging NLP benchmark that combines reading comprehension and commonsense reasoning.
The Dark Secrets of BERT
BERT and its Transformer-based cousins are still ahead on all NLP leaderboards. But how much do they actually understand about language?