Quenches in isolated quantum systems are currently a subject of intense study. Here, we consider quantum few-mode systems that are integrable in their classical mean-field limit and become dynamically unstable after a quench of a system parameter. Specifically, we study a Bose-Einstein condensate (BEC) in a double-well potential and an antiferromagnetic spinor BEC constrained to a single spatial mode. We study the time dynamics after the quench within the truncatedWigner approximation (TWA), focus on the role of motion near separatrices, and find that system relaxes to a steady state due to phase-space mixing. Using the action-angle formalism and a pendulum as an illustration, we derive general analytical expressions for the time evolution of expectation values of observables and their long-time limits. We find that the deviation of the long-time expectation value from its classical value scales as O(1 / ln N), where N is the number of atoms in the condensate. Furthermore, the relaxation of an observable to its steady-state value is a damped oscillation. The damping is Gaussian in time with a time scale of O[(ln N)(2)]. We also give the quantitative dependence of the steady-state value and the damping time on the system parameters. Our results are confirmed with numerical TWA simulations.