Dataset Viewer (First 5GB)
Auto-converted to Parquet Duplicate
id
stringclasses
3 values
country
stringclasses
3 values
category
stringclasses
1 value
dialect
stringclasses
3 values
text
stringclasses
3 values
meta
dict
مصر-لهجة-000001
مصر
لهجة
اوي يا ماشي، طعميه؟
مثال مصري قصير يوضح اللهجة اليومية.
{ "k1": "اوي", "k2": "ماشي", "k3": "طعميه" }
السعودية-لهجة-000001
السعودية
لهجة
هلا والله، وش واججك؟
مثال خليجي قصير يوضح اللهجة اليومية.
{ "k1": "هلا", "k2": "وش", "k3": "واججك" }
المغرب-لهجة-000001
المغرب
لهجة
واش كاين؟
مثال مغربي قصير يوضح اللهجة اليومية.
{ "k1": "واش", "k2": "كاين", "k3": "وقت" }

Dataset evaluation: See EVALUATION.md for schema checks, indexing status, and quality limitations. Viewer note: default is a lightweight preview; select full to load the complete corpus.

Current Hub Validation Status

  • Repository claim: 3,000,000 records
  • Dataset Server indexed rows: 1,183,361
  • Dataset Server estimate: 2,064,964

The 3M target figure is a raw-repository claim and is not yet fully verified by the Hub index. Validate the JSONL files before publishing a definitive record count; do not cite 3M as a verified count until the audit is complete.

Arab Dialects Dataset - 20 Countries

A large-scale Arabic dialects dataset covering 20 Arab countries and 7 content types per country, with a raw target of 3,000,000 records across 140 JSONL files (12.07 GB). The current Hub index exposes 1,183,361 rows and estimates 2,064,964 rows; the raw target still requires a full JSONL audit.

1. Contents

2. Dataset Summary

Attribute Value
Dataset name arab-dialects-20-countries-3m
Countries 20
Files 140 *.jsonl files (20 x 7)
Total records 3,000,000 raw target; 1,183,361 currently indexed; 2,064,964 estimated
Total size 12.07 GB (12,363.8 MB)
File format JSONL, UTF-8 (one record per line)
Languages Arabic dialects: Egyptian, Gulf, Levantine, Maghrebi, Yemeni, Iraqi, Sudanese, Somali + MSA explanations
Suggested tasks Dialect text generation, dialect understanding, dialect/country classification, Arabic chatbot, dialect-to-MSA translation
License CC-BY-4.0
Loading method load_dataset with streaming=True (recommended for 12 GB)

3. Repository Map

3.1 Folder Tree

arab-dialects-20-countries-3m/
├── README.md                  # This file - Hugging Face Dataset Card
├── LICENSE                    # CC-BY-4.0 license
├── CITATION.cff               # Citation metadata
├── .gitattributes             # Git LFS settings for *.jsonl
│
├── 01_مصر/  (Egypt)
│   ├── 01_اللهجة.jsonl               # 21,432 records (dialect)
│   ├── 02_المصطلحات.jsonl            # 21,428 records (terms)
│   ├── 03_النكت.jsonl                # 21,428 records (jokes)
│   ├── 04_المواقف.jsonl              # 21,428 records (situations)
│   ├── 05_الثقافة_والعادات.jsonl     # 21,428 records (culture)
│   ├── 06_طريقة_الكلام.jsonl         # 21,428 records (speaking style)
│   └── 07_المحادثات.jsonl            # 21,428 records (dialogues)
│
├── 02_السعودية/ (Saudi Arabia)   # same 7 files
├── 03_الامارات/ (UAE)            # same 7 files
├── 04_الكويت/ (Kuwait)
├── 05_قطر/ (Qatar)
├── 06_البحرين/ (Bahrain)
├── 07_عمان/ (Oman)
├── 08_اليمن/ (Yemen)
├── 09_العراق/ (Iraq)
├── 10_سوريا/ (Syria)
├── 11_لبنان/ (Lebanon)
├── 12_الاردن/ (Jordan)
├── 13_فلسطين/ (Palestine)
├── 14_السودان/ (Sudan)
├── 15_ليبيا/ (Libya)
├── 16_تونس/ (Tunisia)
├── 17_الجزائر/ (Algeria)
├── 18_المغرب/ (Morocco)
├── 19_موريتانيا/ (Mauritania)
└── 20_الصومال/ (Somalia)         # same 7 files per country

Rule: each country = exactly 150,000 records = 21,432 + 6 x 21,428.

3.2 Visual Map (Mermaid)

graph TD
    ROOT[arab-dialects-20-countries-3m/<br/>3M target records - 12.07GB]
    ROOT --> DOCS[README + LICENSE + CITATION + .gitattributes]
    ROOT --> EG[01_Egypt<br/>150K - 605MB]
    ROOT --> GULF[Saudi Arabia ... Oman<br/>Gulf countries]
    ROOT --> LEV[Iraq ... Palestine<br/>Levant + Iraq]
    ROOT --> AFR[Sudan ... Somalia<br/>North Africa + Horn]
    EG --> T1[01_dialect.jsonl]
    EG --> T2[02_terms.jsonl]
    EG --> T3[03_jokes.jsonl]
    EG --> T4[04_situations.jsonl]
    EG --> T5[05_culture.jsonl]
    EG --> T6[06_speaking_style.jsonl]
    EG --> T7[07_dialogues.jsonl]
    T7 --> REC[JSON record<br/>id - country - category<br/>dialect - text - meta]
pie title Records distribution by type (per country, 150K)
    "Dialect" : 21432
    "Terms" : 21428
    "Jokes" : 21428
    "Situations" : 21428
    "Culture" : 21428
    "Speaking style" : 21428
    "Dialogues" : 21428

4. Countries Table (20 folders)

# Folder Country Records Size Distinctive words (examples)
1 01_مصر Egypt 150,000 605.3 MB ezzayyak, aamel eh, ya ma'allem, keda, awi
2 02_السعودية Saudi Arabia 150,000 624.7 MB wesh halek, absher, kafu, ya hala
3 03_الامارات UAE 150,000 623.1 MB shhalek, zein, rayoog, majboos
4 04_الكويت Kuwait 150,000 621.4 MB shkhbarek, wayed, dewaniya
5 05_قطر Qatar 150,000 605.0 MB shhalek, tayyeb, dalla, kafu alaik
6 06_البحرين Bahrain 150,000 625.6 MB shkhbarek, wayed zein, jabati
7 07_عمان Oman 150,000 606.1 MB mooh, Salalah, luban, shuwaa
8 08_اليمن Yemen 150,000 616.3 MB mandi, fahsa, salta, Adani tea
9 09_العراق Iraq 150,000 614.3 MB shlonak, shaku maku, yaba, hassa
10 10_سوريا Syria 150,000 618.3 MB keefak, tkram, shawarma, kibbeh
11 11_لبنان Lebanon 150,000 623.5 MB meshe lhal, tkram aynak, manoushe
12 12_الاردن Jordan 150,000 619.2 MB sho malak, mansaf, zalameh
13 13_فلسطين Palestine 150,000 628.2 MB musakhan, knafeh, ya ammi
14 14_السودان Sudan 150,000 617.0 MB zol, ya zol, jabana, aseeda
15 15_ليبيا Libya 150,000 613.4 MB shen halek, bahi, bazeen, halba
16 16_تونس Tunisia 150,000 619.8 MB shnahwalek, labes, barsha
17 17_الجزائر Algeria 150,000 616.5 MB wash rak, labes, bezaf, ya khou
18 18_المغرب Morocco 150,000 614.8 MB labas, kidayer, bezaf, atay
19 19_موريتانيا Mauritania 150,000 628.6 MB labas, yasser, ya wkheiret
20 20_الصومال Somalia 150,000 622.4 MB Mogadishu, baasto, shaah
Total 20 folders — 3,000,000 12,363.8 MB (12.07 GB) —

5. Data Types Table (7 files)

Each country contains the same 7 files. This table is per country - multiply by 20 for the global total.

# File name category Records/country Total (x20) Description
1 01_اللهجة.jsonl لهجة (dialect) 21,432 428,640 Authentic greeting expressions and phrases in the local dialect + MSA meaning
2 02_المصطلحات.jsonl مصطلحات (terms) 21,428 428,560 Work/market/home glossary: word + meaning + usage context
3 03_النكت.jsonl نكت (jokes) 21,428 428,560 Jokes and funny situations in the country dialect
4 04_المواقف.jsonl مواقف (situations) 21,428 428,560 Daily scenes: taxi, wedding, cafe, market
5 05_الثقافة_والعادات.jsonl ثقافة (culture) 21,428 428,560 Hospitality, dishes (kabsa/mansaf/tagine/bazeen...), weddings, majlis
6 06_طريقة_الكلام.jsonl طريقة_كلام (speaking style) 21,428 428,560 Tone and style: how to speak like locals, sentence openers/closers
7 07_المحادثات.jsonl محادثات (dialogues) 21,428 428,560 Full dialogues: friend-friend, seller-customer, WhatsApp chat

6. Record Schema

6.1 Fields Table

Field Type Example Description
id string مصر-نكت-000001 Unique ID: country-category-number
country string مصر (Egypt) Country name in Arabic
category string نكت (jokes) One of: لهجة, مصطلحات, نكت, مواقف, ثقافة, طريقة_كلام, محادثات
dialect string واحد مصري اتصل بصاحبه... Short original dialect text (50-300 chars)
text string واحد مصري اتصل... + long explanation Full expanded text (~3.5-4 KB) - best for training
meta.k1 string بقى Distinctive word 1 from the country dialect
meta.k2 string عامل ايه Distinctive word 2
meta.k3 string يا جدع Distinctive word 3

6.2 Real Record Example

{
  "id": "مصر-نكت-000001",
  "country": "مصر",
  "category": "نكت",
  "dialect": "واحد مصري اتصل بصاحبه قاله 'بقى فينك؟' قاله 'في البيت' قاله 'طب عامل ايه وافتح الباب ما انا قدام البيت' 😂",
  "text": "واحد مصري اتصل بصاحبه قاله 'بقى فينك؟' ... وهذا يعكس روح أهل مصر في كلامهم اليومي حيث يستخدمون 'بقى' و'عامل ايه' و'يا جدع' بكثرة ... [record 1 - Egypt - jokes]",
  "meta": {"k1": "بقى", "k2": "عامل ايه", "k3": "يا جدع"}
}

Note: dialect and text are in Arabic (the dataset content language). All documentation around them is in English.

7. Loading and Usage

7.1 Load with Hugging Face datasets (recommended: streaming for 12 GB)

from datasets import load_dataset

# Load everything with streaming (no 12 GB download needed)
ds = load_dataset("ISLAM-PO/arab-dialects-20-countries-3m", "full", split="train", streaming=True)
print(next(iter(ds)))

# Filter one country / category
egy_jokes = ds.filter(lambda x: x["country"] == "مصر" and x["category"] == "نكت")
for row in egy_jokes.take(3):
    print(row["dialect"])

7.2 Load a single country (faster)

from datasets import load_dataset

# Egypt only
ds_eg = load_dataset("json", data_files={"train": "hf://datasets/ISLAM-PO/arab-dialects-20-countries-3m/01_مصر/*.jsonl"}, split="train", streaming=True)

# One file only
ds_one = load_dataset("json", data_files={"train": "hf://datasets/ISLAM-PO/arab-dialects-20-countries-3m/18_المغرب/07_المحادثات.jsonl"}, split="train")

7.3 Local read with Python / Pandas

import json, glob
import pandas as pd

files = glob.glob("arab-dialects-20-countries-3m/01_مصر/*.jsonl")
rows = []
for f in files[:1]:
    with open(f, encoding="utf-8") as fh:
        for line in fh:
            rows.append(json.loads(line))
df = pd.DataFrame(rows)
print(df[["id", "category", "dialect"]].head())
print(df["category"].value_counts())

7.4 Language model training (short example)

# Use the 'text' field for causal LM or 'dialect' for instruction tuning.
# Example prompt:
# instruction: "Write a joke in the Moroccan dialect"
# input: row["dialect"]

8. Generation and Reproduction

The dataset was generated with generate_dataset.py (per-country templates + distinctive vocabulary + contextual padding to reach the target size).

# Small demo (~16 MB, for validation)
python generate_dataset.py --demo

# Full generation (150K per country / ~10 GB+)
python generate_dataset.py --full --per-country 150000 --total-gb 10
Parameter Description Default
--demo 700 records/country (100 per type) for testing —
--full Full generation —
--per-country Records per country 150000
--total-gb Target size in GB (automatic padding if below target) 10

Note: per-record size is computed as total_gb x 1024^3 / (20 x per_country) ~= 3579 bytes. Actual size is ~3.8-4.1 KB with JSON overhead, so the final output is 12.07 GB (above target by design, to guarantee the 10 GB requirement).

9. Considerations and Limitations

Topic Details
Data nature Synthetic, template-generated data, not recorded from native speakers. Good for pre-training and experiments, but review quality before production use.
Repetition Templates repeat with varied words and IDs. Deduplicate before final evaluation.
Bias Balanced representation (150K per country) - does not reflect real population sizes. Large dialects (Egyptian/Levantine) have the same weight as smaller ones.
Cultural sensitivity Jokes and situations are for language research/entertainment only. If you find offensive content for a country, please open an Issue.
Language quality Distinctive words are real per dialect, but long compositions may contain non-100% native phrasing. Human review is recommended for critical samples.
Size 12 GB - use streaming=True or load a single country if your machine is limited.

10. Contributors

Role Name / Handle Contribution
Dataset owner & concept ISLAM-PO Idea, 20 countries + 7 types definition, 10 GB requirement
Generation & engineering Muse Spark (AI Assistant) generate_dataset.py, dialect templates, documentation
Dialect review (open) Open for contribution Native speakers per country review words/jokes and add authentic expressions
Culture review (open) Open for contribution Fix dishes and customs per country

Want to contribute? Send a Pull Request adding new words/templates in generate_dataset.py under COUNTRIES or TEMPLATES, or fix any incorrect expression.

11. License

CC-BY-4.0 - Creative Commons Attribution 4.0 International
  • Allowed: commercial use, modification, distribution, training.
  • Single requirement: give attribution - dataset name + link + license.
  • See the LICENSE file for full details.

12. Citation

@dataset{arab_dialects_20_2026,
  title   = {Arab Dialects Dataset: 20 Countries, 7 Content Types, 3M target Records},
  author  = {ISLAM-PO and Contributors},
  year    = {2026},
  publisher = {Hugging Face},
  version = {1.0.0},
  url     = {https://huggingface.co/datasets/ISLAM-PO/arab-dialects-20-countries-3m},
  note    = {3,000,000 records, 140 JSONL files, 12.07 GB, CC-BY-4.0}
}

The CITATION.cff file contains the same data in Citation File Format.

13. Push to Hugging Face Hub

# 1. Install tools
pip install huggingface_hub datasets

# 2. Login
huggingface-cli login

# 3. Init repo (Git LFS is required for large JSONL files)
cd arab-dialects-20-countries-3m
git init
git lfs install
git lfs track "*.jsonl"
# .gitattributes is already included - make sure it is committed

# 4. Push (upload takes a while for 12 GB - push in batches on slow connections)
huggingface-cli repo create arab-dialects-20-countries-3m --type dataset --yes
git remote add origin https://huggingface.co/datasets/ISLAM-PO/arab-dialects-20-countries-3m
git add README.md LICENSE CITATION.cff .gitattributes
git commit -m "docs: HF dataset card"
git push origin main
# Then push countries in batches:
git add 01_مصر 02_السعودية 03_الامارات 04_الكويت
git commit -m "data: gulf+egypt batch"
git push origin main
# ... repeat for remaining countries

Tip: Use the repository URL shown above when creating a new repository. Add --private to repo create if you want it private first.


Last updated: 2026-09-03 | Version: 1.0.0 | Status: complete generation target; verify indexed count before publication / 12.07 GB

Downloads last month
3,191