Introduction
This is the data directory of the Hunglish CDROM, containing the parallel
corpus as sentence pair files and, where copyright permits, the original
monolingual files as well. The corpus is sorted by genre: the main categories
are as follows.
film
Movie subtitles. The raw files were provided to MOKK for
research only and can not be republished on this CD. The sentence aligned
files are given in "shuffled" format: one sentence per line (Hungarian
sentence TAB English sentence) and the lines alphabetically sorted so as
to make republishing of the original subtitle files impossible. This
data segment has many speling errors owing to OCR text extraction, and
as a special aid for subselecting a higher quality dataset, sentence by
sentence figures of alignment merit are provided in the file
quality.
law
Legal texts. The raw files came from CELEX and are reproduced in full under
the law/en (English) and law/hu
(Hungarian) subdirectories. The aligned files are in the law/bi directory.
lit
Literature. For "classical" material no longer under copyright,
the raw files came from Project
Gutenberg and the Hungarian Electronic
Library. For these, both raw (en, hu) and aligned (bi) files are
available. For "modern" material still under copyright and made avaliable to
MOKK for research purposes, the sentence pair (bi) files are shuffled together
in one bi/Shuffle.
mag
Magazines and news. This material is still largely in preparation
at the time the CD goes to press, please visit the
Hunglish website for more.
swdoc
Software documentation. The raw files come from OpenOffice.org,
Mozilla, Gnome, KDE, and other major FOSS (Free Open Source Software) projects.
mono
Monolingual Hungarian files taken from the Hungarian National
Corpus: parliament and city council minutes, laws, regulations, and other
"official" material as well as the archives of the chat rooms of a major
Hungarian internet portal, index.hu. Please note that the tokenization of these
files differs from the rest of the corpus inasmuch as both sentence-final
punctuation and trailing periods after abbreviations are given as
whitespace-separated tokens. This explains the discrepancy between the
numbver of words and sentences quoted in the summary below and the number
coming from wc.
Summary
| directory |
size (MB) |
words |
contents |
raw text |
film |
18 |
3.27 m |
move subtitles |
never |
| law |
233 |
31.53 m |
EU law (CELEX) |
full |
| lit |
85 |
17.24 m |
literature |
when (C) lapsed |
| mag |
5 |
0.36 m |
magazines, news |
yes, research only |
| swdoc |
8 |
1.27 m |
software documentation |
full |
| mono |
8 |
46.4 m |
monolingual Hungarian |
full |
A more detailed inventory is available in the form of a catalog
and wc files in each directory, the former providing author and title
information (in both languages) and the latter containing the output of wc for
each file. A file conf which summarizes our confidence in the
alignment is also provided for each directory so that users can select a
high-confidence subset of the corpus if they wish.
Copyright
The raw files are either public domain or provided for research purposes
with the consent of the copyright holder, Diplomacy and Trade Magazine
(see mag/*/DTM*).
Documentation
Further details on corpus preparation and alignment
can be found in this draft and this
paper.
Acknowledgements
We gratefully acknowledge CELEX, Project Gutenberg, the Hungarian
Electronic Library, Typotex, and Diplomacy and Trade Magazine.