translate_enaz_10m: an open English-Azerbaijani dataset of 10M pairs

Good Azerbaijani machine translation starts with data, and open parallel data for Azerbaijani is far less abundant than for the large European languages. In April 2023 Learning Machine published translate_enaz_10m on Hugging Face: an open Azerbaijani dataset of about 10 million English-Azerbaijani sentence pairs. This article explains where the texts come from, how the pairs were made, what the files look like and what to keep in mind before you train a model on them.

What is inside

• Size: about 10 million sentence pairs; 975 MB zipped and 2.8 GB unzipped.
• Format: TSV, one English-Azerbaijani pair per line, the two sides separated by a tab.
• Tasks: translation and text generation.
• Licence: OpenRAIL.
• Publisher: Learning Machine LLC, April 2023 (the citation entry on the dataset card carries the year 2022).

Where the texts come from

The Azerbaijani sentences were extracted from various text sources of the National Library of Azerbaijan: books, news, poems and articles, as the library's own tags describe them. Only clean texts went into the corpus. Wikipedia articles were deliberately left out: for this purpose they contain a lot of irrelevant data, and a large, noisy source can easily outweigh a smaller clean one during training.

The result is a corpus with a broad mix of genres: literary language, journalism and general-interest articles. That is a good base for a general-purpose model.

How the pairs were made

The Azerbaijani side is the original text after cleaning. The English side was produced with Google Translate. This has practical consequences that are worth understanding before you choose how to use the data:

• English to Azerbaijani is the natural direction. The text the model learns to produce is original, human-written Azerbaijani. This is exactly how we used the corpus for our own translation model.
• Azerbaijani to English is possible, but with a caveat. Here the target is machine-translated English, so a model will learn to imitate that translation style, including its mistakes.
• The English source can contain errors. If precision in your domain matters, sample-check pairs and filter the ones that look wrong.
• Specialised domains need their own data. For legal, medical or customer-support text, add in-domain pairs on top of this general corpus.

How to download and read it

The dataset page on Hugging Face describes the corpus, and its loader takes the data from one archive, translate_enaz_10m.zip. Download it directly:

wget https://learningmachine.az/datasets/translate_enaz_10m.zip
unzip translate_enaz_10m.zip   # dataset_enaz_10m.tsv

The archive holds one file, dataset_enaz_10m.tsv: one pair per line, without a header row. The first column is the English sentence, the second is the original Azerbaijani sentence. At 2.8 GB it is too large to open in a spreadsheet, so read it as a stream:

import csv

path = 'dataset_enaz_10m.tsv'

with open(path, encoding='utf-8', newline='') as f:
    reader = csv.reader(f, delimiter='	', quoting=csv.QUOTE_NONE)
    for i, (english, azerbaijani) in enumerate(reader):
        print(english, '|', azerbaijani)
        if i == 4:
            break

For filtering and statistics, pandas can process the file in chunks without loading all of it into memory:

import pandas as pd

for chunk in pd.read_csv(path, sep='	', header=None, names=['en', 'az'],
                         quoting=3, chunksize=500_000, on_bad_lines='skip'):
    chunk = chunk.dropna().drop_duplicates()
    # filter, sample or save the chunk here

Practical tips before training

1. Hold out a validation set before any experiments, and keep it fixed so results stay comparable.
2. Remove duplicates and extreme pairs: empty sides, very long sentences, and pairs where one side is several times longer than the other.
3. Check the Azerbaijani letters. Make sure ə, ı, ğ, ö, ü, ç and ş are encoded consistently across the data and in your inference input.
4. Decide on a maximum length in tokens for your model and drop or split longer pairs.
5. Read the licence. OpenRAIL allows broad use but includes use restrictions; check that your project fits them.

What it was used for

We trained our open English-to-Azerbaijani model mt5-enaz-10m on this corpus: mT5-base, 10 epochs, 620k steps, with an evaluation loss of 0.53. The model is available on Hugging Face together with example code. More about the corpus is on its project page.

Need data for your own task?

A model is only as good as the data behind it. If you need a corpus collected, cleaned, aligned or labelled for your task, in Azerbaijani or another language, see our data collection and processing service or describe your task and we will suggest a solution.