Pretraining on 14.8T tokens of the multilingual corpus, primarily English and Chinese. It contained a greater ratio of math and programming compared to the pretraining dataset of V2.Liang, who had Beforehand focused on applying AI to investing, experienced bought a "stockpile of Nvidia A100 chips," a style of tech which is now banned from export to