Abstract: Neutron radiography, also called neutron imaging, is a significant radiographic non-destructive testing (NDT) technology. In the process of neutron imaging, electronic equipment, space fluctuation of neutrons, collimator structure, high-energy particle irradiation imaging detector, and other factors can introduce different degrees of degradation on the neutron radiographic images, mainly in the form of Gaussian noise, Poisson noise, geometric unsharpness (blur), white spot noise, and other image distortions, especially in compact neutron radiography systems with limited neutron flux. Most of the existing distortion suppression algorithms for neutron radiographic images require some prior information on distortion, such as noise intensity and blur intensity. Moreover, the distortion-suppression effect depends greatly on the estimation accuracy of the distortion parameters. However, the distortion model of neutron radiographic images contains a variety of mixed noise and blurring. Suppressing some single distortion may introduce or aggravate other kinds of distortions. Therefore, it is difficult to effectively improve the image quality with a single distortion-suppression scheme. In order to solve this issue, the multiple-degradation models of neutron radiographic images were first analyzed. Then, a large-scale image distortion dataset consistent with the real distortion types of neutron radiography was constructed, including different levels of Gaussian noise, Poisson noise, white spot noise, and Gaussian blur. A novel image restoration network based on the generative adversarial network (GAN) and coordinate attention mechanism was designed. Through end-to-end training, multiple distortions of real neutron radiographic images can be effectively suppressed by the proposed method. Moreover, because of the different imaging mechanisms and distortion types of natural images and neutron radiographic images, a no-reference image quality assessment scheme was further proposed for neutron radiographic images in this paper. A series of experiments on the real neutron radiographic images show that the proposed restoration method achieves good performance in the subjective evaluation. Meanwhile, the objective quality assessment method also verifies the effectiveness of the proposed method. Specifically, according to the subjective scores of 30 groups of researchers on the effects of different restoration schemes, the proposed image restoration scheme achieves the best performance in subjective evaluation. At the same time, based on the quantitative analysis of the objective image quality assessment method proposed in this paper, it can be seen that the quality score of the proposed restoration scheme increases by more than 10% compared with other deep learning and traditional image restoration methods, thus demonstrating the effectiveness of the proposed restoration and quality assessment for the compact neutron radiography system.