Abstract: The flow instability of natural circulation system under rolling motion has a significant influence on the safety of marine nuclear power system. The predict complex instability of flow rate under rolling motion was optimized using genetic algorithm and neural networks. The largest Lyapunov exponent of the flow rate time series was calculated by small-data method to acquire the maximum predictable time. The multi-step prediction of the flow rate time series was achieved by back propagation (BP) neural network with single hidden layer. The forecast result agrees well with the experiment data for the prediction with small number of time steps. However, the BP neural network could be trapped in local optimal solution. To overcome this drawback, genetic algorithm was applied to optimize the initial thresholds and weights of the BP neural network. Hence, the non-linear prediction ability of BP neural network was largely improved. A practical and relatively accurate method for natural circulation flow instability prediction is provided.