Abstract: After Fukushima nuclear accident, the passive concept is widely adopted in the thermal safety design of advanced nuclear reactors. The in-pool passive residual heat removal system (PRHRS) can also provide security guarantee for the domestic reactor HPR1000 without driving of the external power. Although RELAP5 code has been extensively validated for thermal-hydraulic transients in nuclear reactor, it underestimates the heat exchanging power and the system flow rate in the in-pool PRHRS due to the lack of specialized model for in-pool heat exchanger. Therefore, the model development for the in-pool heat exchanger is necessary for accurate simulation of PRHRS and safety assessment of the reactor system. In this study, the in-pool bundle boiling model and large-diameter tube condensation model were selected and implemented in RELAP5 code as the in-pool heat exchanger model. Firstly, the relevant heat transfer models in original RELAP5 were evaluated. The results show that Chen correlation developed initially for in-tube boiling is not suitable for bundle boiling heat transfer. Thus Cooper correlation developed initially for in-pool boiling considering the surface roughness is used to replace Chen correlation for the bundle geometry. On the other hand, the calculation in original RELAP5 for in-tube condensation encounters some faults under the condition with low gas speed in the natural circulation of PRHRS. The Shah method modified through fluid viscosity and pressure is chosen for the calculation of condensation heat transfer coefficient in the large-diameter tube. Secondly, the separate effect validation was carried out for the two heat transfer models. For the validation of in-pool bundle boiling model, the root-mean-square error of the heat transfer coefficient predicted by the improved RELAP5 is reduced to 16.2%, compared to the value of 53.8% obtained by the original code. For the validation of in-tube condensation model, the root-mean-square error of the heat flux predicted by the improved RELAP5 is reduced from 10.9% to 7.6%. Both models are improved effectively. After that, the influence of nodalization method was analyzed based on the in-pool energy removal system for emergency operation (PERSTO) facility. It is shown that the nodalization for the heat exchanger and pool has important influence on the thermal-hydraulic simulation of PERSTO. The pool should be modelled as at least two hydraulic components and the volume size of the heat exchanger along the flow direction should be less than 0.09 m. Finally, the simulation capabilities of the original RELAP5 and the improved code were compared based on the transient parameters of the integral experimental facility. The heat exchanger power and the flow rate in the natural circulation predicted by the improved RELAP5 are consistent with the experimental data. Then the improved code in this study is confirmed for thermal-hydraulic analysis of in-pool PRHRS after separate effect validation and integral effect validation. This study can provide the basic technology support for the structure design of PRHRS and safety analysis of the reactor system.