Large Language Models are Geographically Biased

Large Language Models are Geographically Biased

2024 | Rohin Manvi, Samar Khanna, Marshall Burke, David Lobell, Stefano Ermon
Large Language Models (LLMs) exhibit geographic bias, where their training data leads to systemic errors in geospatial predictions. This bias is particularly evident in their ratings of locations, which show strong monotonic correlation with ground truth data for objective topics like population density and infant mortality rate. However, LLMs also display biases on sensitive subjective topics such as attractiveness and morality, often rating areas with lower socioeconomic conditions less favorably. The study introduces a bias score to quantify these geographic biases, revealing significant variation among existing LLMs. The bias score incorporates the mean absolute deviation of output ratings and Spearman's rank correlation with an anchoring bias distribution, such as infant mortality rate. The results show that LLMs like GPT-4 Turbo and Llama 2 70B exhibit less bias compared to other models. The findings highlight the importance of addressing geographic bias in LLMs to ensure fairness and accuracy in their applications.Large Language Models (LLMs) exhibit geographic bias, where their training data leads to systemic errors in geospatial predictions. This bias is particularly evident in their ratings of locations, which show strong monotonic correlation with ground truth data for objective topics like population density and infant mortality rate. However, LLMs also display biases on sensitive subjective topics such as attractiveness and morality, often rating areas with lower socioeconomic conditions less favorably. The study introduces a bias score to quantify these geographic biases, revealing significant variation among existing LLMs. The bias score incorporates the mean absolute deviation of output ratings and Spearman's rank correlation with an anchoring bias distribution, such as infant mortality rate. The results show that LLMs like GPT-4 Turbo and Llama 2 70B exhibit less bias compared to other models. The findings highlight the importance of addressing geographic bias in LLMs to ensure fairness and accuracy in their applications.
Reach us at info@study.space
[slides] Large Language Models are Geographically Biased | StudySpace