Abstract: Due to the high complexity and high risk of nuclear power system, its design and operation require high safety and reliability. Anomaly detection is an important technology to improve the inherent safety of nuclear power system. Aiming at the special problems of nuclear power system, such as many types of faults, small probability of faults and lack of fault samples, based on the included angle cosine distance calculation theory and normal historical operation data, an algorithm to detect system anomalies was proposed in this paper, which can effectively deal with the practical problem of insufficient fault samples. The data test results show that the algorithm can accurately detect the minor anomalies and faults of the system based on the limited parameters. It can be used as the entry link algorithm in the fault diagnosis system of nuclear power system and has a good practical application prospect.