The paper constructs random walks on simple Lie groups that converge quickly to the Haar measure for all moments up to order \( t \). Specifically, a step of the walk on the unitary or orthogonal group of dimension \( 2^n \) is a random Pauli rotation \( e^{i\theta P/2} \). The spectral gap of this random walk is shown to be \( \Omega(1/t) \), which is the best known bound for a random walk on the permutation group on \( \{0, 1\}^n \). This implies that the walk gives an \( \varepsilon \)-approximate unitary \( t \)-design in depth \( \mathcal{O}(nt^2 + t \log 1/\varepsilon) d \), where \( d = \mathcal{O}(\log n) \) is the circuit depth to implement \( e^{i\theta P/2} \). The proof uses quadratic Casimir operators of Lie algebras. The results have implications for circuit complexity and seed length, and can be adapted to special orthogonal groups.The paper constructs random walks on simple Lie groups that converge quickly to the Haar measure for all moments up to order \( t \). Specifically, a step of the walk on the unitary or orthogonal group of dimension \( 2^n \) is a random Pauli rotation \( e^{i\theta P/2} \). The spectral gap of this random walk is shown to be \( \Omega(1/t) \), which is the best known bound for a random walk on the permutation group on \( \{0, 1\}^n \). This implies that the walk gives an \( \varepsilon \)-approximate unitary \( t \)-design in depth \( \mathcal{O}(nt^2 + t \log 1/\varepsilon) d \), where \( d = \mathcal{O}(\log n) \) is the circuit depth to implement \( e^{i\theta P/2} \). The proof uses quadratic Casimir operators of Lie algebras. The results have implications for circuit complexity and seed length, and can be adapted to special orthogonal groups.