Verified, then logged.
Sun 22 Nov 2026 · 00:17 KST / 15:17 UTC한국어 | ENSearch /

Breaking01 / AI

Mixing English in pre-training matches 2–13× the Arabic data

Seen from behind in a university computer lab lit through an Arabic lattice window, a researcher faces a monitor showing two blurred side-by-side columns of text in different scripts. AI-generated image

Mixing English data into the pre-training of Arabic language models produced a gain equivalent to 2 to 13 times the unique Arabic data on task accuracy, according to company-published figures in a paper Apple released on 13 Oct 2026.

On validation loss, the error measured on data not used in training, the gain was equivalent to 2 to 3 times the unique Arabic data, also a company-published figure. The experiments covered about 1,000 pre-training runs, with Arabic as the target language and English as the auxiliary language. Models came in four sizes, from 150 million to 1.43 billion parameters, and the authors include researchers affiliated with the Technical University of Denmark (DTU).

The paper said mixing delivered larger improvements than hyperparameter tuning, the adjustment of settings fixed before training, on both validation loss and task accuracy. That gap grew with model size. The authors concluded that target-language validation loss systematically underestimates the value of mixing.

The researchers recommended mixing in a language with abundant data and transferring hyperparameters from a small model using μP, a technique for carrying settings across model sizes. Their recommendations also tell practitioners to "prioritize the mixing ratio over hyperparameter tuning".

According to the paper, pre-training for most of the world's languages repeats the same data many times, which degrades generalisation. High weight decay, one of the available remedies, shrinks weights to restrict network capacity. Mixing diversifies the training signal and adds knowledge the repeated data cannot supply, and validation loss captures only the first of those effects.

The published abstract does not say whether the same multiples apply to language pairs other than Arabic and English.