Abstract: The passive containment cooling system may serve to remove the heat from the containment after some accidents by means of natural circulation. However, it may also suffer from the two-phase flow instability during the operation. The mathematical model and constitutive relationships were set up for an open natural circulation system. The conservation equations were linearized by using the perturbation method. The transfer function of the mass flow rate of the system was derived in terms of the enthalpy change of the heating section through Laplace transformation. Based on the Nyquist stability criterion, the influence of thermal-hydraulic parameters on the stability of the natural circulation system was analyzed. The results show that the flow stability of the system is mainly dominated by the relationship between the void fraction and the quality of the two-phase flow. In the studied range, with the increase of the quality, the sensitivity of the void fraction to the quality is weakened, and the system tends to be stable.