This paper presents bootstrap resampling methods for computing statistical significance in machine translation evaluation. The authors address the question of whether differences in test results between translation systems indicate true differences in system quality. They validate their methods using the BLEU score, a widely used automatic evaluation metric for machine translation. Even for small test sizes, their methods provide confidence that observed differences are real. The paper discusses the challenges of evaluating machine translation systems, which have shifted from human judgment to automatic metrics like BLEU. BLEU measures n-gram overlap with reference translations and has been shown to correlate with human judgment. However, complex metrics like BLEU do not lend themselves to analytical techniques for assessing statistical significance, so the authors propose bootstrap resampling methods. Bootstrap resampling is a statistical technique that involves repeatedly sampling data to estimate the distribution of a statistic. The authors use this method to estimate the confidence intervals for BLEU scores, allowing them to determine if differences in test results are statistically significant. They validate their approach by comparing different systems on various test sets and show that their estimated significance levels are accurate. The authors also discuss paired bootstrap resampling, which is used to compare two systems. By repeatedly sampling data from a single test set, they can estimate the probability that one system outperforms another. This method is particularly useful for small test sets, where traditional statistical methods may not be reliable. The paper concludes that bootstrap resampling provides a reliable way to assess the statistical significance of test results in machine translation evaluation. This method allows researchers to draw more confident conclusions about the performance of different translation systems. The authors hope that their methods will become standard practice in published machine translation research.This paper presents bootstrap resampling methods for computing statistical significance in machine translation evaluation. The authors address the question of whether differences in test results between translation systems indicate true differences in system quality. They validate their methods using the BLEU score, a widely used automatic evaluation metric for machine translation. Even for small test sizes, their methods provide confidence that observed differences are real. The paper discusses the challenges of evaluating machine translation systems, which have shifted from human judgment to automatic metrics like BLEU. BLEU measures n-gram overlap with reference translations and has been shown to correlate with human judgment. However, complex metrics like BLEU do not lend themselves to analytical techniques for assessing statistical significance, so the authors propose bootstrap resampling methods. Bootstrap resampling is a statistical technique that involves repeatedly sampling data to estimate the distribution of a statistic. The authors use this method to estimate the confidence intervals for BLEU scores, allowing them to determine if differences in test results are statistically significant. They validate their approach by comparing different systems on various test sets and show that their estimated significance levels are accurate. The authors also discuss paired bootstrap resampling, which is used to compare two systems. By repeatedly sampling data from a single test set, they can estimate the probability that one system outperforms another. This method is particularly useful for small test sets, where traditional statistical methods may not be reliable. The paper concludes that bootstrap resampling provides a reliable way to assess the statistical significance of test results in machine translation evaluation. This method allows researchers to draw more confident conclusions about the performance of different translation systems. The authors hope that their methods will become standard practice in published machine translation research.