Fracton phases exhibit striking behavior which appears to render them beyond the standard topological quantum field theory (TQFT) paradigm for classifying gapped quantum matter. Here, we explore fracton phases from the perspective of defect TQFTs and show that topological defect networks-networks of topological defects embedded in stratified 3+1-dimensional (3+1D) TQFTs-provide a unified framework for describing various types of gapped fracton phases. In this picture, the subdimensional excitations characteristic of fractonic matter are a consequence of mobility restrictions imposed by the defect network. We conjecture that all gapped phases, including fracton phases, admit a topological defect network description and support this claim by explicitly providing such a construction for many well-known fracton models, including the X-cube and Haah s B code. To highlight the generality of our framework, we also provide a defect network construction of a fracton phase hosting non-Abelian fractons. As a byproduct of this construction, we obtain a generalized membrane-net description for fractonic ground states as well as an argument that our conjecture implies no topological fracton phases exist in 2+1-dimensional gapped systems. Our paper also sheds light on techniques for constructing higher-order gapped boundaries of 3+1D TQFTs.