A Critical Look at Benchmarking Datasets: Problem of finding relationship between sentences

Abstract

Competitions on many data sets attract the attention of the researchers and thanks to the competition, better systems are developed. Recently, however, it has been found that algorithms use annotation artifacts on some datasets. These findings have led to questioning of the results-oriented assessment. It was previously shown that the use of the second sentence is sufficient to prediction of the relationship between two sentences in a popular benchmark dataset. In this study, the Turkish version of the dataset was created and similar findings were obtained for both languages. In addition to the attack methods in the literature, word-based and sentence-based new attacks were applied, and when the sentences were replaced, the estimates were shown to contain logical inconsistencies. For example, if the relationship between two sentences is contradiction, then the relationship should not change when the sentences change. However, it was seen that this is not the case in a significant part of the samples. As a result, it is revealed that the result oriented achievements obtained in this dataset are not real by examining the internal dynamics of the system. This critical approach should not be left on all popular datasets.