Spin qubits in silicon quantum dots can have long coherence times, yet their manipulation relies on the exchange interaction, through which charge noise can induce decoherence. Charge traps near the interface of a Si heterostructure lead to fluctuations in the quantum-dot confinement and barrier potentials, which cause gating errors and two-spin dephasing. We quantify these effects in Si double quantum dots using a realistic model of noise. Specifically, we consider both random telegraph noise from a few traps good for dots grown on submicron wafers and 1/f noise from many traps good for larger wafers appropriate for quantum dot arrays. We give estimates of gate errors for single-spin qubit architectures and dephasing in singlet-triplet qubits.