# Elements of World Knowledge (EWOK): A cognition-inspired framework for evaluating basic world knowledge in language models

Authors: Anna A. Ivanova, Aalok Sathe, Benjamin Lipkin, Unnathi Kumar, Setayesh Radkani, Thomas Hikaru Clark, Carina Kauf, Jennifer Hu, R. T. Pramod, Gabriel Grand, Vivian C. Paulun, Maria Ryskina, Ekin Aky"urek, Ethan Wilcox, Nafisa Rashid, Leshem Choshen, Roger Levy, Evelina Fedorenko, Joshua B. Tenenbaum, Jacob Andreas
Venue: CoRR (2024)

## Abstract

 The ability to build and reason about models of the world is essential for situated language understanding. But evaluating world modeling capabilities in modern AI systems—especially those based on language models—has proven challenging, in large part because of the difficulty of disentangling conceptual knowledge about the world from knowledge of surface co-occurrence statistics. This paper presents Elements of World Knowledge (EWoK), a framework for evaluating language models’ understanding of the conceptual knowledge underlying world modeling. EWoK targets specific concepts from multiple knowledge domains known to be important for world modeling in humans, from social interactions (help, deceive) to spatial relations (left, right). Objects, agents, and locations in the items can be flexibly filled in, enabling easy generation of multiple controlled datasets. We then introduce EWoK-core-1.0, a dataset of 4,374 items covering 11 world knowledge domains. We evaluate 20 open-weights large language models (1.3B–70B parameters) and compare them with human performance. All tested models perform worse than humans, with results varying drastically across domains. Performance on social interactions and social properties was highest and performance on physical relations and spatial relations was lowest. Overall, this dataset highlights simple cases where even large models struggle and presents rich avenues for targeted research on LLM world modeling capabilities.

## Links
- arXiv: https://arxiv.org/abs/2405.09605
- PDF: https://arxiv.org/pdf/2405.09605
- HTML: https://arxiv.org/html/2405.09605
- Hugging Face: https://huggingface.co/papers/2405.09605
- alphaXiv: https://www.alphaxiv.org/abs/2405.09605
- DOI: https://doi.org/10.48550/ARXIV.2405.09605
- Semantic Scholar: https://www.semanticscholar.org/paper/269790884
- Publisher: https://doi.org/10.48550/arXiv.2405.09605

## How to cite
@article{DBLP:journals/corr/abs-2405-09605,
author       = {Anna A. Ivanova and
                  Aalok Sathe and
                  Benjamin Lipkin and
                  Unnathi Kumar and
                  Setayesh Radkani and
                  Thomas Hikaru Clark and
                  Carina Kauf and
                  Jennifer Hu and
                  R. T. Pramod and
                  Gabriel Grand and
                  Vivian C. Paulun and
                  Maria Ryskina and
                  Ekin Aky{"{u}}rek and
                  Ethan Wilcox and
                  Nafisa Rashid and
                  Leshem Choshen and
                  Roger Levy and
                  Evelina Fedorenko and
                  Joshua B. Tenenbaum and
                  Jacob Andreas},
  title        = {Elements of World Knowledge {(EWOK):} {A} cognition-inspired framework
                  for evaluating basic world knowledge in language models},
  journal      = {CoRR},
  volume       = {abs/2405.09605},
  year         = {2024},
  url          = {https://doi.org/10.48550/arXiv.2405.09605},
  doi          = {10.48550/ARXIV.2405.09605},
  eprinttype    = {arXiv},
  eprint       = {2405.09605},
  timestamp    = {Sun, 04 Aug 2024 01:00:00 +0200},
  biburl       = {https://dblp.org/rec/journals/corr/abs-2405-09605.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}