Abstract: Due to the advantages of long service life, compact structure, strong environmental adaptability and high inherent safety, the heat pipe reactor power supply system is worth studying. The NUSTER2.0 dual-mode heat pipe reactor non-nuclear prototype built by Xi’an Jiaotong University adopts the integrated design of “simulated reactor core-high temperature heat pipe-Stirling generator-thermoelectric power generation”. The combination of the dynamic module coupled with Stirling and the static module coupled with thermoelectric power generation device can meet the diversified demands of energy. In this paper, the steady-state experiment of NUSTER2.0 was carried out to verify the design, and the system coupling simulation of the prototype was carried out by COMSOL multiphysics to analyze its operation and mechanical characteristics. In the 15 kW thermal power steady-state experiment, the generating power of the dynamic module is 2 019 W, and the static module is 493.2 W. The overall thermoelectric conversion efficiency is 16.7%, which verifies the engineering feasibility of the design. The system simulation model is coupled with the thermal resistance network model of high temperature heat pipe, the equivalent thermal conductivity model of thermoelectric power generation device and the Stirling adiabatic analysis program. The heat source of fuel rod and the heat sink of cooling water were simulated by heat flux boundary and convective boundary respectively. Under 15 kW thermal power, the steady-state simulation results are in good agreement with the experimental results on both the heat pipe temperature distribution and the main operating parameters of the dynamic and static modules, while the maximum relative error is 6.73%, which verifies that the coupled model is reliable. The variable thermal power steady-state simulation determines that the limit due to Stirling is 16.5 kW, and predicts that the static module will reach the peak efficiency near 25 kW. The system start-up transient simulation obtains the start-up characteristics under the 3.5-10.15 kW stepped power platform. The prototype start-up strategy based on the combination of power platform cold start and thermal insulation restart was also proposed. The thermal-mechanical coupling simulation indicates that the stress danger zone of the system is the coupling between the heat pipe and assembly matrix. Under 15 kW thermal power, the maximum stress at the coupling of heat pipe-core matrix and heat pipe-collector matrix are 160 MPa and 97 MPa, respectively. In the experiment, the self-adjusting function of the Stirling insulation ring can significantly relieve the 366 MPa stress between it and the Stirling matrix. The thermal expansion rate of the reactor core is 1.21%, which will cause negative feedback on the reactivity. The simulation results provide the numerical basis for the operation safety analysis and further experimental design of the NUSTER2.0 prototype.