Overview
A language model can answer a question fluently while silently assuming a setting the reader never chose. A typical wedding custom, familiar meal, or everyday object can mean something different in different places. This study turns the problem into a controlled test. It names a cultural setting, varies the language of the question, and compares how several models respond.
Key Findings
A language model answering a question about a custom has several clues to work with. The name of a country is one. The language of the question is another. The researchers vary both and look for occasions when the model follows the language more strongly than the place named in the prompt.
Change the place and the language
The authors distinguish culture-specific knowledge from general commonsense reasoning under a cultural context. One experiment asks a model to choose a plausible completion for a statement associated with a named country. A second asks the model to infer the country from a cultural statement. Later experiments use generic statements from GenericsKB-Best to see how country context affects verification and country association.
Country names are experimental labels. Each one encompasses communities with varied practices. The cultural question-answering test uses GeoMLAMA assertions associated with the United States, China, India, Iran, and Kenya. English provides a common comparison language. Chinese, Hindi, Farsi, and Swahili let the researchers test whether asking in a language associated with a country helps.
An answer can depend on the language of the question
| Prompt language | US | China | India | Iran | Kenya | Average |
|---|---|---|---|---|---|---|
| English | 0.78 | 0.78 | 0.79 | 0.51 | 0.57 | 0.69 |
| Hindi | 0.38 | 0.35 | 0.59 | 0.40 | 0.37 | 0.42 |
| Farsi | 0.49 | 0.51 | 0.64 | 0.52 | 0.49 | 0.53 |
| Swahili | 0.54 | 0.66 | 0.59 | 0.38 | 0.38 | 0.51 |
Selected rows from Table 3 of the NAACL paper. They report GPT-3.5-turbo accuracy on the cultural commonsense QA task. Farsi is slightly ahead of English for the Iran questions, while English leads on average across the five countries among the tested languages. Table 2 shows that the direction can change with the model. GPT-4 scores 0.89 in Chinese and 0.85 in English for the China questions.
A model can lose accuracy for several reasons. It may know less about a subject, struggle to follow an instruction in a particular language, or encounter a translation that changes the cues. The benchmark measures the combined effect. A follow-up experiment can hold the meaning fixed with bilingual review and vary the prompt format to separate some of those explanations.
When the model assigns a country to a general claim
The association experiments turn the question around. The culture-specific task removes the country name or demonym from 700 CANDLE assertions. Items in which translation already removed that cue are filtered out. For Kenya, GPT-4’s accuracy in guessing the masked country rises from 0.45 in English to 0.61 in Swahili, even though English was often stronger on the earlier completion task. The two experiments put different demands on the same language cue. Table 4
The other experiment gives the model a general commonsense statement and examines which country the model associates with it. The distinction matters. Identifying where a particular food is traditionally eaten can require cultural knowledge. Treating a broadly shared practice as characteristic of one country can instead reveal what the model habitually associates with that place.
The cost of switching languages
The verification task asks whether a general statement holds when the model is asked to consider a specific country. Changing language sometimes alters model accuracy even with the same named country. The paper reports drops of up to about 20 percentage points on its general commonsense experiments. Read that alongside the language-specific differences in the table above. Together the experiments separate the model’s knowledge of a practice, its response to a cultural cue, and its ability to reason across languages.
Questions & Answers
The opening answers are simulated author responses based on the paper.
Q: Does naming a country solve the problem?
A: Naming the country gives the model useful context when the question depends on local practice. Large accuracy gaps remained. In our general verification task, naming a country reduced LLAMA2 accuracy on universally true statements. Local knowledge and the ability to use it in the language of the question each warrant testing.
Q: Could a translated prompt be measuring translation quality?
A: Yes. We used translation and checked a subset by back-translation. Some phrasing differences can survive that check. A bilingual evaluation with independently written prompts could test whether the result holds for natural questions in each language.
Q: What would count as progress beyond higher accuracy?
A: A system should also be able to recognize that several answers may be appropriate, say what context it assumed, and ask for more context when a precise answer depends on it. Those behaviors need their own evaluation.
Q: What can the country-by-language table explain about a gap?
A: Comparing several prompt languages for each country shows where benchmark accuracy changes. A gap can reflect cultural knowledge, translation, or how a model follows instructions in a given language. The curated assertions and country labels also cover only part of each culture. Small differences within the table are descriptive, including the one-point difference for Iran in Table 3.
Future Ideas
A language-pair test before a help desk launches
For a university help desk, build a small collection of questions where the right answer depends on a local rule or practice. Bilingual staff write equivalent questions in two languages and label acceptable answers and questions that require clarification. Run the same assistant on both, and route cases for review when the factual answer changes or either response omits the context that determines it.
Compare this queue with random review and a single-language confidence rule at the same staffing budget. Measure how many materially wrong answers each catches, along with how often the system creates extra work for common requests. Use the results to decide which topics need bilingual sources or human escalation before launch.
Before scoring, have bilingual staff agree on which missing facts materially change an answer and record when several locally appropriate answers are possible.
Sources
Original abstract
Large language models (LLMs) have demonstrated substantial commonsense understanding through numerous benchmark evaluations. However, their understanding of cultural commonsense remains largely unexamined. In this paper, we conduct a comprehensive examination of the capabilities and limitations of several state-of-the-art LLMs in the context of cultural commonsense tasks. Using several general and cultural commonsense benchmarks, we find that (1) LLMs have a significant discrepancy in performance when tested on culture-specific commonsense knowledge for different cultures; (2) LLMs’ general commonsense capability is affected by cultural context; and (3) The language used to query the LLMs can impact their performance on cultural-related tasks.Our study points to the inherent bias in the cultural understanding of LLMs and provides insights that can help develop culturally-aware language models.