This paper presents a nonperturbative treatment of strong-coupling induced effects in atom-field systems which cannot be seen in traditional perturbative treatments invoking compromising assumptions such as the Born-Markov, rotating wave, or Fermi Golden rule. We consider an atom whose internal degrees of freedom are modeled by a harmonic oscillator which is bilinearly coupled to a scalar quantum field, representing one of the two polarizations of an electromagnetic field. Because the whole system is Gaussian we can solve this problem exactly. Using the open quantum system conceptual framework and the influence functional formalism we derive the dynamics of the reduced density matrix for the atom which enables the calculation of atomic transition probability and other relevant physical quantities. Finding an exact solution to this problem has the distinct advantage of enabling one to capture fully the strong coupling regime and discover interesting effects such as spontaneous ground state excitation [R. Passante, T. Petrosky, and I. Prigogine, Long-time behaviour of self-dressing and indirect spectroscopy, Physica (Amsterdam) 218A, 437 (1995).] which is unfathomable in perturbative treatments. The conventional description of atomic-optical activities is predicated on the assumption that the state of the total atom-field system is a product state of the atomic excitations and the photon number states, an assumption which is valid only for vanishingly weak coupling. The correct energy eigenfunctions to use should be that of the Hamiltonian of the combined atom-field system. Other features associated with finite to strong coupling effects such as resonance peak broadening and transition from a gapped to a gapless spectrum can all be understood from this perspective. Finally, to put the issues in a proper perspective we take the perturbative limit of the exact results and compare them with those from conventional time-dependent perturbation theory (TDPT). This enables one to pin-point where the deficiencies of TDPT lie as one removes the ultraweak coupling assumption.