Abstract: Heat-pipe-cooled reactors offer many advantages, such as modularity and no flow loops. In recent years, it has attracted the attention of many scholars. Solid cores involve complex thermal-mechanical coupling behavior over their operating lifetime. The deformation of solid core structure has significant influence on its heat transfer performance. This paper introduced the gap heat transfer model by Ross and Stoute, which is an important part of the heat transfer process in the core. The creep and swelling behavior of materials used in solid core under high temperature and irradiation environment was briefly described. In this paper, a simplified finite element model of solid core was developed based on the secondary development of ABAQUS by programming subroutines, which considered the effect of gap heat transfer, creep and swelling behavior in the core. Based on the Ross-Stoute gap heat transfer model, the subroutine GAPCON was developed to simulate the gap heat transfer behavior in the core. The subroutine CREEP was developed to simulate the complex mechanical behavior of core components under different temperatures and radiation environments. Typical characteristics of temperature and stress distribution in each component during the normal operation period of 5 years were obtained. The thermal-mechanic coupling behaviors of solid core and the interaction between components were analyzed. The high temperature area appears in the center of the fuel pellets, and the temperature drops along the radial direction of the fuel pellets. This distribution of temperature field results in the inner extrusion and the outer tension of fuel pellets. The annular tensile stress region appears obviously outside the fuel pellets. These temperature and stress distribution characteristics can partly explain the radial cracking behavior of cylindrical fuel pellets. Furthermore, the designed size of the solid core was optimized based on finite element and machine learning to obtain lower temperature and stress peaks. The parametric modeling technique of ABAQUS was used to generate 5 800 sets of data, and the sample database was established for the training and testing of deep neural network machine learning. A surrogate model of core design parameters, maximum temperature and Mises stress in the reactor was established. The surrogate model had good predictive accuracy. Based on the surrogate model, pareto front was obtained by NSGA-Ⅱ algorithm and the optimized design parameters were obtained. The results of core temperature and stress optimized by surrogate model were verified by finite element method. Compared with the initial design, the maximum temperature decreases by 8.44% and the maximum stress decreases by 34.43%. The performance of the core is improved.