publications

2022

1. SIGIR

   Knowledge Graph Question Answering Datasets and Their Generalizability: Are They Enough for Future Research?

   Jiang, L., and Usbeck, R.

   In The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 2022

   Abs

   Existing approaches on Question Answering over Knowledge Graphs (KGQA) have weak generalizability. That is often due to the standard i.i.d. assumption on the underlying dataset. Recently, three levels of generalization for KGQA were defined, namely i.i.d., compositional, zero-shot. We analyze 25 well-known KGQA datasets for 5 different Knowledge Graphs (KGs). We show that according to this definition many existing and online available KGQA datasets are either not suited to train a generalizable KGQA system or that the datasets are based on discontinued and out-dated KGs. Generating new datasets is a costly process and, thus, is not an alternative to smaller research groups and companies. In this work, we propose a mitigation method for re-splitting available KGQA datasets to enable their applicability to evaluate generalization, without any cost and manual effort. We test our hypothesis on three KGQA datasets, i.e., LC-QuAD, LC-QuAD 2.0 and QALD-9). Experiments on re-splitted KGQA datasets demonstrate its effectiveness towards generalizability. The code and a unified way to access 18 available datasets is online at https://github.com/semantic-systems/KGQA-datasets as well as https://github.com/semantic-systems/KGQA-datasets-generalization.

2. LREC

   Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis

   Perevalov, A., Yan, X., Kovriguina, L.,  Jiang, L., Both, A., and Usbeck, R.

   In Proceedings of the 13th Language Resources and Evaluation Conference, 2022

   Abs PDF

   Data-driven systems need to be evaluated to establish trust in the scientific approach and its applicability. In particular, this is true for Knowledge Graph (KG) Question Answering (QA), where complex data structures are made accessible via natural-language interfaces. Evaluating the capabilities of these systems has been a driver for the community for more than ten years while establishing different KGQA benchmark datasets. However, comparing different approaches is cumbersome. The lack of existing and curated leaderboards leads to a missing global view over the research field and could inject mistrust into the results. In particular, the latest and most-used datasets in the KGQA community, LC-QuAD and QALD, miss providing central and up-to-date points of trust. In this paper, we survey and analyze a wide range of evaluation results with significant coverage of 100 publications and 98 systems from the last decade. We provide a new central and open leaderboard for any KGQA benchmark dataset as a focal point for the community - this https URL. Our analysis highlights existing problems during the evaluation of KGQA systems. Thus, we will point to possible improvements for future evaluations.

2019

1. ICSC

   End-to-End Product Taxonomy Extension from Text Reviews

   Jiang, L., Biran, O., Tiwari, M., Weng, Z., and Benajiba, Y.

   In 2019 IEEE 13th International Conference on Semantic Computing (ICSC), 2019

   Abs

   Product ontologies - consisting of a taxonomic categorization of product types and lists of attributes that types and products have - are invaluable for analyzing sales, opinions and ratings of items on e-commerce sites. Unfortunately, many international and smaller sites lack such ontologies, and instead feature only coarse high-level categories. We present a Siamese neural model which utilizes such coarse categories to learn a fine-grained hierarchical categorization of products, and jointly extract lists of product attributes from text reviews. We show that our model retains a high accuracy on the categorization task for unseen products and unseen category depths, and as a side effect learns to extract useful product attributes.

2. ICSC

   Siamese Networks for Semantic Pattern Similarity

   Benajiba, Y., Sun, J., Zhang, Y.,  Jiang, L., Weng, Z., and Biran, O.

   In 2019 IEEE 13th International Conference on Semantic Computing (ICSC), 2019

   Abs

   Semantic Pattern Similarity is an interesting, though not often encountered NLP task where two sentences are compared not by their specific meaning, but by their more abstract semantic pattern (e.g., preposition or frame). We utilize Siamese Networks to model this task, and show its usefulness in determining SQL patterns for unseen questions in a database-backed question answering scenario. Our approach achieves high accuracy and contains a built-in proxy for confidence, which can be used to keep precision arbitrarily high.