Abstract: As the fourth generation nuclear power technology, high temperature gas-cooled reactor (HTGR) has good prospects due to its inherent safety and high temperature. Two-phase flow stability is one of the most important thermal hydraulic issues for steam generators. For the steam generator of HTR-PM (high temperature gas-cooled reactor-pebble module), the size of the inner tube diameter is uniform. While for steam generator of HTR-10 (10 MW high temperature gas-cooled reactor), the inner tube diameter of single-phase preheating region is smaller than that of two-phase and superheated regions. A model with variable tube diameter will be established to predict the stability boundary of HTR-PM and HTR-10 steam generators. The heat transfer tube of the steam generator is divided into three regions, and they are single-phase preheating region, two-phase flow region and superheated region. Based on the existing frequency domain theoretical model, a new frequency domain theoretical model with different inner tube diameters for the single-phase preheating region and two-phase region was established for analyzing two-phase flow stability. System stability was determined by the Nyquist curve of the open transfer function. The prediction of the new proposed model is in good agreement with the engineering test results of HTR-10 (variable tube diameter) and HTR-PM (uniform tube diameter) once through steam generators. Then, the effects of friction factor difference between helical tube and inclined straight tube, the effects of helical diameter difference, and the effects of different tube diameters for single-phase preheating and two-phase regions on two-phase flow stability were analyzed. For systems with high inlet resistance coefficients, when other conditions are the same, the friction factor of an inclined straight tube is smaller than that of a helical tube, so the system with an inclined straight tube (the inclined angle is the same as the helical angle of the helical tube) is more stable than the system with a helical tube. When other conditions are the same, the friction factor decreases with the increase of helical diameter, so increasing the helical diameter can slightly improve the stability of the system when the value of inlet resistance coefficients is large. Both the difference between helical tube and inclined straight tube and the effect of helical diameter can be attributed to the effect of friction factor on stability boundary. If friction factor changes significantly, stability boundary will be affected more significantly. Compared with the design of uniform inner tube diameter, the system will become slightly stable when the inner diameter of single-phase preheating regions is smaller than that of two-phase and superheated regions. The main reason is that decreasing tube inner diameter will increase the frictional pressure drop of the single-phase preheating regions.