We propose an experimental scheme to directly detect the Chern number of the topological Haldane model realized with optical lattices based on the time-of-flight imaging. The Haldane model can be realized with two-component fermionic atoms loaded in the optical lattices consisting of two sublattices and each of the sublattices hosts atoms with a specific component of the atoms. We find that the Bloch wave function of the lower band can be measured with the time-of-flight imaging and thus the Chern number can be directly obtained from the Bloch function. We numerically simulate the Chern number of the Haldane model and its modified version in both honeycomb lattice and brick-wall lattice, and show that the proposed approach is general and feasible in practical experiments. (C) 2015 Optical Society of America