103 reads New Story
Researchers Learn to Measure AI’s Language Skills
by MorphologyDecember 30th, 2024
Too Long; Didn't Read
Researchers in Poland have developed an open-source tool that improves the evaluation and comparison of AI used in natural language preprocessing.
Authors:
(1) Martyna Wiącek, Institute of Computer Science, Polish Academy of Sciences;
(2) Piotr Rybak, Institute of Computer Science, Polish Academy of Sciences;
(3) Łukasz Pszenny, Institute of Computer Science, Polish Academy of Sciences;
(4) Alina Wróblewska, Institute of Computer Science, Polish Academy of Sciences.
Editor's note: This is Part 7 of 10 of a study on improving the evaluation and comparison of tools used in natural language preprocessing. Read the rest below.
Table of Links
Abstract and 1. Introduction and related works
- NLPre benchmarking
2.2. Online benchmarking system
- NLPre-PL benchmark
- Evaluation
- Conclusions
- Appendices
- Acknowledgements
- Bibliographical References
- Language Resource References
4. Evaluation
4.1. Evaluation methodology
To maintain the de facto standard to NLPre evaluation, we apply the evaluation measures defined for the CoNLL 2018 shared task and implemented in the official evaluation script.[11] In particular, we focus on F1 and AlignedAccuracy, which is similar to F1 but does not consider possible misalignments in tokens, words, or sentences.
In our evaluation process, we follow default training procedures suggested by the authors of the evaluated systems, i.e. we do not conduct any optimal hyperparameter search in favour of leaving the recommended model configuration as-is. We also do not further fine-tune selected models.
This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license.
[11] https://universaldependencies.org/conll18/conll18_ud_eval.py
L O A D I N G
. . . comments & more!
. . . comments & more!
