Authors:
(1) Martyna Wiącek, Institute of Computer Science, Polish Academy of Sciences;
(2) Piotr Rybak, Institute of Computer Science, Polish Academy of Sciences;
(3) Łukasz Pszenny, Institute of Computer Science, Polish Academy of Sciences;
(4) Alina Wróblewska, Institute of Computer Science, Polish Academy of Sciences. Editor's note: This is Part 6 of 10 of a study on improving the evaluation and comparison of tools used in natural language preprocessing. Read the rest below. Table of Links Abstract and 1. Introduction and related works NLPre benchmarking 2.1. Research concept 2.2. Online benchmarking system 2.3. Configuration NLPre-PL benchmark 3.1. Datasets 3.2. Tasks Evaluation 4.1. Evaluation methodology 4.2. Evaluated systems 4.3. Results Conclusions

Appendices
Acknowledgements
Bibliographical References
Language Resource References 3.2. Tasks The complete set of NLPre tasks was originally curated for evaluating language systems in the CoNLL shared task 2018 (Zeman et al., 2018). These tasks mainly focus on preliminary text processing, such as tokenisation or divulging morphosyntactic features. We follow the CoNLL task choice and include all these tasks in NLPre-PL. Segmentation A segmentation task consists in splitting texts into sentences (Sentences), orthographic tokens (Tokens), and syntactic words (Words), the latter being the basic units of morphosyntactic analysis. Segmentation is not a trivial task. In some languages, an orthographic token may be recognised as a multi-word token (multiword for short) combining multiple syntactic words, e.g. in Polish, the token spalibyśmy (Eng. we would sleep) consists of the past participle spali (Eng. slept), the conditional marker by (Eng. would) and the mobile inflection śmy. Since the consistent model of segmentation into words and sentences was used in NKJP1M and PDB-UD, we maintain this data segmentation in NLPre-PL. It is also worth mentioning that the CoNLL format (but not TEI and DAG) allows for annotating orthographic tokens; thus, they are included in the NLPre-PL benchmark. Tagging A tagging task is the process of identifying parts of speech (i.e. POS tagging) and possibly morphological features (i.e. morphological analysis) of words. It follows a predefined POS tagset. As mentioned in Section 3.1, two tagsets are used in the NLPre-PL datasets: Morfeusz and UD. Lemmatisation Lemmatisation involves predicting canonical forms of syntactic words. Canonical forms are conventionally established identifiers of lexemes (i.e. sets of inflectionally related syntactic words). Since Polish is a fusional language with a large number of inflected words, lemmatisation is an important task, albeit not trivial, e.g. the lemma of kluczy can be either the infinitive kluczyć (Eng. to weave) or the noun klucz (Eng. a key). Dependency parsing Dependency parsing is the process of automatically predicting the syntactic structure of an input sentence. A dependency structure is a labelled directed tree with nodes corresponding to syntactic words and edges between these words specifying dependency relations. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. Authors: (1) Martyna Wiącek, Institute of Computer Science, Polish Academy of Sciences; (2) Piotr Rybak, Institute of Computer Science, Polish Academy of Sciences; (3) Łukasz Pszenny, Institute of Computer Science, Polish Academy of Sciences; (4) Alina Wróblewska, Institute of Computer Science, Polish Academy of Sciences. Authors: Authors: (1) Martyna Wiącek, Institute of Computer Science, Polish Academy of Sciences; (2) Piotr Rybak, Institute of Computer Science, Polish Academy of Sciences; (3) Łukasz Pszenny, Institute of Computer Science, Polish Academy of Sciences; (4) Alina Wróblewska, Institute of Computer Science, Polish Academy of Sciences. Editor's note: This is Part 6 of 10 of a study on improving the evaluation and comparison of tools used in natural language preprocessing. Read the rest below. Editor's note: This is Part 6 of 10 of a study on improving the evaluation and comparison of tools used in natural language preprocessing. Read the rest below. Editor's note: This is Part 6 of 10 of a study on improving the evaluation and comparison of tools used in natural language preprocessing. Read the rest below. Editor's note: This is Part 6 of 10 of a study on improving the evaluation and comparison of tools used in natural language preprocessing. Read the rest below. Table of Links Abstract and 1. Introduction and related works Abstract and 1. Introduction and related works NLPre benchmarking NLPre benchmarking 2.1. Research concept 2.1. Research concept 2.2. Online benchmarking system 2.2. Online benchmarking system 2.3. Configuration 2.3. Configuration NLPre-PL benchmark NLPre-PL benchmark 3.1. Datasets 3.1. Datasets 3.2. Tasks 3.2. Tasks Evaluation Evaluation 4.1. Evaluation methodology 4.1. Evaluation methodology 4.2. Evaluated systems 4.2. Evaluated systems 4.3. Results 4.3. Results Conclusions

Appendices
Acknowledgements
Bibliographical References
Language Resource References Conclusions Appendices
Acknowledgements
Bibliographical References
Language Resource References Conclusions Appendices Acknowledgements Bibliographical References Language Resource References Appendices Acknowledgements Bibliographical References Language Resource References 3.2. Tasks The complete set of NLPre tasks was originally curated for evaluating language systems in the CoNLL shared task 2018 (Zeman et al., 2018). These tasks mainly focus on preliminary text processing, such as tokenisation or divulging morphosyntactic features. We follow the CoNLL task choice and include all these tasks in NLPre-PL. Segmentation A segmentation task consists in splitting texts into sentences (Sentences), orthographic tokens (Tokens), and syntactic words (Words), the latter being the basic units of morphosyntactic analysis. Segmentation is not a trivial task. In some languages, an orthographic token may be recognised as a multi-word token (multiword for short) combining multiple syntactic words, e.g. in Polish, the token spalibyśmy (Eng. we would sleep) consists of the past participle spali (Eng. slept), the conditional marker by (Eng. would) and the mobile inflection śmy. Since the consistent model of segmentation into words and sentences was used in NKJP1M and PDB-UD, we maintain this data segmentation in NLPre-PL. It is also worth mentioning that the CoNLL format (but not TEI and DAG) allows for annotating orthographic tokens; thus, they are included in the NLPre-PL benchmark. Segmentation Tagging A tagging task is the process of identifying parts of speech (i.e. POS tagging) and possibly morphological features (i.e. morphological analysis) of words. It follows a predefined POS tagset. As mentioned in Section 3.1, two tagsets are used in the NLPre-PL datasets: Morfeusz and UD. Tagging Lemmatisation Lemmatisation involves predicting canonical forms of syntactic words. Canonical forms are conventionally established identifiers of lexemes (i.e. sets of inflectionally related syntactic words). Since Polish is a fusional language with a large number of inflected words, lemmatisation is an important task, albeit not trivial, e.g. the lemma of kluczy can be either the infinitive kluczyć (Eng. to weave) or the noun klucz (Eng. a key). Lemmatisation Dependency parsing Dependency parsing is the process of automatically predicting the syntactic structure of an input sentence. A dependency structure is a labelled directed tree with nodes corresponding to syntactic words and edges between these words specifying dependency relations. Dependency This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Researchers Learn to Measure AI’s Language Skills 

New Framework Promises to Train AI to Better Understand Hard-to-Grasp Languages Like Polish

Read My Stories

Too Long; Didn't Read

Make resilience your competitive advantage

Researchers Challenge AI to Tackle the Toughest Parts of Language Processing

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

AI Breakthrough Sharpens Telescope Images-Astronomy’s Next Big Leap

New Framework Simplifies Comparison of Language Processing Tools Across Multiple Languages

Researchers Build Public Leaderboard for Language Processing Tools

New Web App Lets Researchers Test and Rank Language AI Tools in Real Time

Researchers Create Plug-and-Play System to Test Language AI Across the Globe

AI Breakthrough Sharpens Telescope Images-Astronomy’s Next Big Leap

New Framework Simplifies Comparison of Language Processing Tools Across Multiple Languages

Researchers Build Public Leaderboard for Language Processing Tools

New Web App Lets Researchers Test and Rank Language AI Tools in Real Time

Researchers Create Plug-and-Play System to Test Language AI Across the Globe

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps