This paper explores an effective field theory that describes the dynamics of massive particles interacting electromagnetically or gravitationally, focusing on the expansion about their mass ratio, known as the self-force (SF) expansion. The authors derive an effective action that includes the lighter particle and the photon or graviton, propagating in a Coulomb or Schwarzschild background. The 0SF dynamics are described using the background field method, while the 1SF dynamics are supplemented by a "recoil operator" that accounts for the wobble of the heavy particle. The formalism leverages the analytic expressions for classical backgrounds and particle trajectories to extract multiloop integrands for perturbative scattering. The authors verify their formalism by studying the two-loop classical scattering of scalar particles in electromagnetism and gravity, and present new calculations for the two-loop classical scattering of dyons and particles interacting with additional scalar or vector fields. The recoil operator is crucial for correctly reproducing the results of perturbation theory in a flat background, and the authors demonstrate its importance through explicit calculations.This paper explores an effective field theory that describes the dynamics of massive particles interacting electromagnetically or gravitationally, focusing on the expansion about their mass ratio, known as the self-force (SF) expansion. The authors derive an effective action that includes the lighter particle and the photon or graviton, propagating in a Coulomb or Schwarzschild background. The 0SF dynamics are described using the background field method, while the 1SF dynamics are supplemented by a "recoil operator" that accounts for the wobble of the heavy particle. The formalism leverages the analytic expressions for classical backgrounds and particle trajectories to extract multiloop integrands for perturbative scattering. The authors verify their formalism by studying the two-loop classical scattering of scalar particles in electromagnetism and gravity, and present new calculations for the two-loop classical scattering of dyons and particles interacting with additional scalar or vector fields. The recoil operator is crucial for correctly reproducing the results of perturbation theory in a flat background, and the authors demonstrate its importance through explicit calculations.