Lilith
The main line: language models we pretrain from scratch, with training kept simple enough to repeat.
Lilith · Models
Small language models, trained from scratch and released with every step of their training, so that how they learn can be studied rather than guessed.
In preparationNo Lilith model has been trained or released yet. This page describes what we are building.
She uttered the Ineffable Name and flew away into the air.
כשברא הקב״ה אדם הראשון יחיד, אמר לא טוב היות האדם לבדו, ברא לו אשה מן האדמה כמוהו וקראה לילית, מיד התחילו מתגרין זה בזה … אמרה לו שנינו שוין לפי ששנינו מאדמה, ולא היו שומעין זה לזה, כיון שראתה לילית אמרה שם המפורש ופרחה באויר העולם, עמד אדם בתפלה לפני קונו ואמר, רבש״ע אשה שנתת לי ברחה ממני, מיד שגר הקב״ה שלשה מלאכים הללו אחריה להחזירה … עזבו אותה והלכו אחריה והשיגוה בתוך הים במים עזים שעתידין המצריים לטבוע בו וספרוה דבר ה' ולא רצתה לחזור …We borrow the image, not the belief. A model learns from its data something nobody wrote down, and it acts on what it learned. Lilith is the model that knows a hidden name. Eisoptric’s work is to learn that name, and to keep what the model does within rules people set.
One main line, and smaller siblings built for controlled comparison.
The main line: language models we pretrain from scratch, with training kept simple enough to repeat.
Named after Lilith’s children in later tradition: smaller siblings of each Lilith model, varying one thing at a time (size, seed or data order) so that one change can be compared against another.
Saved densely through training, and most densely early on, where many capabilities first appear.
Which tokens the model saw at each step, so a change in behaviour can be traced back to what it was learning from.
The training code, hyperparameters and environment: enough to train the same model again.
What we measured along the way: evaluations across checkpoints, the probes we ran, and what did not work.
Lilith will not be a frontier model, and is not meant to be. A small model can be trained many times, inspected at every step, and compared under one controlled change at a time.
That is what studying how models learn requires, and it is what frontier-scale training cannot offer.
Training runs, releases and notes will be announced on GitHub.