Abstract: With mankind’s growing desire to explore the universe, deep space exploration missions have become a hot topic in scientific research and technology. These missions are often required to operate for long periods of time in extreme environments, placing high demands on energy systems. Especially in deep space far away from the sun, solar energy resources become scarce and more reliable and efficient energy solutions need to be found. Facing the need for high-power space nuclear power systems for deep space exploration missions, a dual-mode nuclear thermal propulsion system based on the regenerative Brayton cycle was designed. To meet the design needs of the dual-mode nuclear thermal propulsion system, mathematical modeling of the system was carried out from the propulsion and power generation parts, and the composition of each component and the main influence parameters of the two modules were analyzed. With regard to the selection of the Brayton cycle system, the specific heat capacity ratio, density, viscosity and thermal conductivity of He-Xe, He-sCO₂ and He-N₂O, which are the common work materials in the space Brayton cycle, were compared to investigate their thermodynamic properties and transportation characteristics. The aforementioned properties of He-Ar mixed media were proposed and investigated, as well as the effects of helium mass fraction on the two main properties of He-Ar mixed media, namely specific heat capacity ratio and viscosity. In order to address the problem of conflicting optimal choices for each system performance parameter, the expressions derived above were used to optimize the system by using the non-dominated sorted whale optimization algorithm as the objective function for specific impulse, thermal efficiency and cyclic work. Finally, the main parameters affecting the dual-mode nuclear thermal propulsion system, the optimal choice of the nuclear thermal propulsion part, the changes in the properties and advantages and disadvantages of each work material at different temperatures and pressures, the advantages of the He-Ar hybrid work material in terms of its thermodynamic properties and the shortcomings of its transport properties were found, and the optimal solution of the Pareto front for the system’s multi-objective optimization was obtained.