Abstract: Nonlinear evolution characteristics of natural circulation flow instabilities under rolling motion conditions were studied. Experimental time series of different flow states were analyzed with frequency spectrum and reconstructed phase space. The geometric invariants, including correlation dimension (CD), Kolmogorov entropy (K entropy) and maximal Lyapunov exponent (MLE), were determined based on phase space reconstruction theory. Evolution characteristics of natural circulation flow instabilities under rolling motion conditions were studied based on the results of geometric invariants. The results indicate that the values of the geometric invariants firstly enhance and then weaken as non-dimensional power increases. The system changes from limit cycle to chaotic oscillations via period doubling bifurcation, and finally returns to steady flow. The nonlinear characteristics of the system firstly enhance and then weaken which is caused by the mutual feedback and different degrees of coupling of thermal driving force, flow resistance and the additional force caused by rolling motion.