This paper surveys 146 papers analyzing "bias" in natural language processing (NLP) systems, finding that their motivations are often vague, inconsistent, and lacking in normative reasoning. Despite the fact that analyzing "bias" is an inherently normative process, the papers' proposed quantitative techniques for measuring or mitigating "bias" are poorly matched to their motivations and do not engage with the relevant literature outside of NLP. The authors propose three recommendations to guide work analyzing "bias" in NLP systems: (1) ground work in the relevant literature outside of NLP that explores the relationships between language and social hierarchies; (2) provide explicit statements of why system behaviors described as "bias" are harmful, in what ways, and to whom; and (3) examine language use in practice by engaging with the lived experiences of members of communities affected by NLP systems. The authors argue that these recommendations are essential for developing a more comprehensive understanding of "bias" in NLP systems and for ensuring that the work is grounded in the relationships between language and social hierarchies. The paper also highlights the importance of considering the power relations between technologists and communities affected by NLP systems and the need for deeper engagement between these groups. The authors conclude that without this grounding, researchers and practitioners risk measuring or mitigating only what is convenient to measure or mitigate, rather than what is most normatively concerning.This paper surveys 146 papers analyzing "bias" in natural language processing (NLP) systems, finding that their motivations are often vague, inconsistent, and lacking in normative reasoning. Despite the fact that analyzing "bias" is an inherently normative process, the papers' proposed quantitative techniques for measuring or mitigating "bias" are poorly matched to their motivations and do not engage with the relevant literature outside of NLP. The authors propose three recommendations to guide work analyzing "bias" in NLP systems: (1) ground work in the relevant literature outside of NLP that explores the relationships between language and social hierarchies; (2) provide explicit statements of why system behaviors described as "bias" are harmful, in what ways, and to whom; and (3) examine language use in practice by engaging with the lived experiences of members of communities affected by NLP systems. The authors argue that these recommendations are essential for developing a more comprehensive understanding of "bias" in NLP systems and for ensuring that the work is grounded in the relationships between language and social hierarchies. The paper also highlights the importance of considering the power relations between technologists and communities affected by NLP systems and the need for deeper engagement between these groups. The authors conclude that without this grounding, researchers and practitioners risk measuring or mitigating only what is convenient to measure or mitigate, rather than what is most normatively concerning.