This paper proposes a test-time Forward-Optimization Adaptation (FOA) method for adapting a pre-trained model to unseen test samples with distribution shifts, without requiring backpropagation or modifying model weights. FOA operates by learning a new input prompt through a derivative-free covariance matrix adaptation (CMA) evolution strategy, combined with an unsupervised fitness function that measures test-training statistic discrepancy and model prediction entropy. Additionally, a back-to-source activation shifting scheme is introduced to align model activations from out-of-distribution (OOD) samples with the source in-distribution (ID) domain, further enhancing adaptation performance. FOA is designed to work on quantized 8-bit Vision Transformers (ViT) and outperforms gradient-based TENT on full-precision 32-bit ViT, achieving up to 24-fold memory reduction on ImageNet-C. The method is particularly effective in resource-constrained environments such as smartphones and FPGAs, where backpropagation is not feasible. Extensive experiments on four benchmarks (ImageNet-C, ImageNet-R, ImageNet-V2, ImageNet-Sketch) demonstrate that FOA achieves state-of-the-art performance in terms of classification accuracy and expected calibration error (ECE), outperforming both gradient-free and gradient-based test-time adaptation methods. FOA's key contributions include a novel paradigm for test-time adaptation that avoids backpropagation and model weight modification, a forward-only adaptation approach using prompt adaptation and activation shifting, and a new fitness function that ensures stable prompt learning under online unsupervised settings. The method is also effective on quantized models, demonstrating its versatility and efficiency in real-world applications. Overall, FOA provides a practical and efficient solution for test-time adaptation in scenarios where computational resources are limited.This paper proposes a test-time Forward-Optimization Adaptation (FOA) method for adapting a pre-trained model to unseen test samples with distribution shifts, without requiring backpropagation or modifying model weights. FOA operates by learning a new input prompt through a derivative-free covariance matrix adaptation (CMA) evolution strategy, combined with an unsupervised fitness function that measures test-training statistic discrepancy and model prediction entropy. Additionally, a back-to-source activation shifting scheme is introduced to align model activations from out-of-distribution (OOD) samples with the source in-distribution (ID) domain, further enhancing adaptation performance. FOA is designed to work on quantized 8-bit Vision Transformers (ViT) and outperforms gradient-based TENT on full-precision 32-bit ViT, achieving up to 24-fold memory reduction on ImageNet-C. The method is particularly effective in resource-constrained environments such as smartphones and FPGAs, where backpropagation is not feasible. Extensive experiments on four benchmarks (ImageNet-C, ImageNet-R, ImageNet-V2, ImageNet-Sketch) demonstrate that FOA achieves state-of-the-art performance in terms of classification accuracy and expected calibration error (ECE), outperforming both gradient-free and gradient-based test-time adaptation methods. FOA's key contributions include a novel paradigm for test-time adaptation that avoids backpropagation and model weight modification, a forward-only adaptation approach using prompt adaptation and activation shifting, and a new fitness function that ensures stable prompt learning under online unsupervised settings. The method is also effective on quantized models, demonstrating its versatility and efficiency in real-world applications. Overall, FOA provides a practical and efficient solution for test-time adaptation in scenarios where computational resources are limited.