•1 min read•from Machine Learning
Released a free 9.8M doc Indic multilingual corpus — Hindi, Bengali, Tamil, Telugu + 7 more (CC0, HuggingFace) [P]
Built this over the past few weeks as part of a multilingual research project. Figured I'd share it here. Check it out!
~9.8M web documents across 11 languages — hi, bn, ta, te, mr, gu, kn, ml, pa, ur, en. ~8.4B tokens. CC0 license.
[link] [comments]
Want to read more?
Check out the full article on the original site
Tagged with
#rows.com
#multilingual corpus
#Indic
#web documents
#tokens
#CC0 license
#Hindi
#Bengali
#Tamil
#Telugu
#multilingual research project
#HuggingFace
#language codes
#language diversity
#data sharing
#dataset
#8.4B tokens
#research contribution
#open-source
#Indian languages