糖心TV

Overview

Modern neural networks operate at unprecedented scales across model size, data and compute. A central research problem is to understand how their performance scales with these factors, which guides how networks can be trained optimally at scale. In recent years, empirical heuristics for scaling have arguably driven much of the success of Large Language Models (LLMs). Theoretical work on scaling laws has also seen much fruitful progress, shedding light on empirical phenomena such as model collapse, emergence and training stability, while providing concrete practical insights on techniques such as hyperparameter tuning.

This three-day workshop will bring together researchers working at the frontiers of theoretical scaling laws to share their insights about the field. The workshop will be the first of its kind in the UK, inspired by successes of similar workshops in the and .

• The first half of the workshop consists of introductory tutorials, with the aim of equipping attendees with basic tools for framing and understanding problems in this field;
• The second half will feature talks on latest research advances.

The aim is for researchers across academia and industry to learn about and participate in this active field of research, which has seen many fruitful empirical outcomes.