Whether due to the sheer size of a computational domain, the fine resolution required, or the multiples scales and stochasticity of the dynamics, the dimensionality of a system must often be reduced so that problems of interest become computationally tractable. In this paper, we develop retractions for time-integration schemes that efficiently and accurately evolve the dynamics of a system’s low-rank approximation. Through differential geometry, we analyze the error incurred at each time-step due to the high-order curvature of the manifold of fixed-rank matrices. We first obtain a novel, explicit, computationally inexpensive set of algorithms that we refer to as perturbative retractions and show that the set converges to an ideal retraction that projects optimally and exactly to the manifold of fixed-rank matrices by reducing what we define as the projection-retraction error. Furthermore, each perturbative retraction itself exhibits high-order convergence to the best low-rank approximation of the full-rank solution. Using perturbative retractions, we then develop a new class of integration techniques that we refer to as dynamically orthogonal Runge–Kutta (DORK) schemes. DORK schemes integrate along the nonlinear manifold, updating the subspace upon which we project the system’s dynamics as it is integrated. Through numerical test cases, we demonstrate our schemes for matrix addition, real-time data compression, and deterministic and stochastic partial differential equations. We find that DORK schemes are highly accurate by incorporating knowledge of the dynamic, nonlinear manifold’s high-order curvature, and they are computationally efficient by limiting the growing rank needed to represent the evolving dynamics.