Recently it has been demonstrated that an ensemble of trapped ions may serve as a quantum annealer for the number-partitioning problem [Nat. Commun. 7, 11524 (2016)]. This hard computational problem may be addressed by employing a tunable spin-glass architecture. Following the proposal of the trapped-ion annealer, we study here its robustness against thermal effects; that is, we investigate the role played by thermal phonons. For the efficient description of the system, we use a semiclassical approach, and benchmark it against the exact quantum evolution. The aim is to understand better and characterize how the quantum device approaches a solution of an otherwise difficult to solve NP-hard problem.