Abstract: To meet the demand for lightweight and high-efficiency neutron shielding materials in small modular reactors, a high-performance neutron shielding composite was developed in this study using hydrogen-rich epoxy resin AFG-90H as the polymer matrix to slow down fast neutrons, and nano-Gd₂O₃ as the neutron-absorbing phase. However, poor interfacial compatibility between inorganic fillers and the organic polymer matrix makes agglomeration likely, resulting in a decrease in the nuclear shielding performance of the composite material. Previous studies show that coupling agents form chemical bonds with the surface of inorganic fillers and the polymer matrix through reactive groups at both ends of their molecules, thereby constructing molecular bridges at the interface between the two phases and effectively improving interfacial compatibility and bonding strength. Based on the above findings, nano-Gd₂O₃ was modified in this study using silane, titanate, zirconate, and aluminate coupling agents, respectively, to prepare a series of M-Gd₂O₃/AFG-90H composites. The effects of different coupling agents on the microstructure, thermal stability, mechanical properties, and neutron shielding performance of the composites were systematically studied. The experimental results show that modification with coupling agents can significantly improve the uniformity of Gd₂O₃ dispersion and the interfacial bonding strength in the epoxy resin matrix, thereby enhancing the mechanical properties, temperature resistance, and neutron shielding performance of the composites. Among these, silane coupling agent modification yields the best results. Compared with the unmodified composite, the 5% weight loss temperature (T₅) of the modified composite increases from 324.9 ℃ to 349.5 ℃, the Shore hardness increases from 85.4 HD to 92.5 HD, and the tensile strength increases from 28.1 MPa to 47.8 MPa. The neutron shielding performance of the composites was practically measured using the D-T neutron source. The half-value layer thickness of the silane-modified composite decreases from 3.43 mm to 1.72 mm. Compared to the unmodified system, the thickness required to achieve the same shielding effect decreases by approximately 50%, indicating that silane modification is an effective strategy for enhancing the neutron shielding performance of Gd₂O₃/AFG-90H composites. Mechanistic analysis shows that silane coupling agents optimize the interfacial compatibility between inorganic particles and the organic matrix through chemical bonding, effectively suppressing filler agglomeration and the neutron leakage effect, thereby achieving a synergistic improvement in both mechanical properties and shielding performance. This study addresses the engineering requirements for lightweight and highly efficient shielding materials in advanced nuclear energy systems and developed a series of lightweight M-Gd₂O₃/AFG-90H composites with effective neutron shielding performance, elucidating the effects of coupling agent modification on the mechanical properties, thermal stability, and shielding performance of the composites. The findings provide reliable data support for the lightweight design of shielding materials.