Abstract: The performance of space nuclear reactors and spacecraft is significantly influenced by the size and mass of shield, therefore shielding design and optimization are crucial for the development of space nuclear power systems. In this paper, shielding design and optimization process was proposed, and the effectiveness was verified through an optimized shielding design for the Jupiter Icy Moons Orbiter (JIMO) reactor. Building upon the open lattice reactor concept of the JIMO project, the neutronic design of the JIMO reactor was supplemented. The shielding design and optimization for the reactor employed a layered combination of beryllium (Be), boron carbide (B₄C), tungsten (W), and lithium hydride (LiH). Considering the radial distribution of radiation dose, the Monte Carlo method was utilized to compute the neutron flux and the photon dose at the payload. Additionally, the neutron dose at the leading edge of the LiH was taken into account. Given the computational cost associated with the Monte Carlo method, the stepwise optimization approach was proposed after the analysis of the coupled transport characteristics of neutrons and photons and the shielding design principles. The stepwise optimization and analysis revealed several key findings. Firstly, the multi-layer Be-B₄C configuration, compared to the single-layer Be plus single-layer B₄C arrangement, effectively reduces reflected neutrons, thereby diminishing the mass required for photon shielding material. This reduction results from a decrease in the secondary photons generated within the leading-edge structural materials. Secondly, due to the lower photon doses at the outer edge resulting from the strong penetration ability of photons, the mass of the photon shielding material can be reduced by decreasing the photon shielding radius. Thirdly, a shielding configuration with a Be to B₄C thickness ratio of 7:3 demonstrates excellent shielding effectiveness while maintaining a relatively small mass. Finally, placing W at 30 cm from the shielding leading edge not only reduces the photon shielding radius but also decreases the generation of the secondary photons, leading to an optimal shielding mass. The optimized shielding demonstrates equivalent radiation attenuation capabilities to the JIMO shielding, encompassing the attenuation of neutron flux and photon dose at the payload and neutron absorption dose at the leading edge of LiH. Simultaneously, the shielding mass is reduced by 98.41 kg. This underscores the effectiveness of the shielding optimization process. The developed shield and the associated design process have the potential to serve as a valuable reference for future shielding design and optimization for space nuclear reactors.