Abstract: The tube bundle structures of steam generators in nuclear power plants are prone to damage and failure under long-term flow impact, threatening the safe operation of the reactor system. Although there have been many numerical studies on the flow-induced vibration of tube bundle structures, the establishment of efficient and reliable two-way fluid-structure interaction models based on the refinement method and the clarification of the transformation mechanism of different flow-induced vibration mechanisms are two key scientific issues that need to be solved urgently. Thus, in order to study the flow-induced vibration behavior of the tube bundle structures, accurately predict the critical flow velocity for the occurrence of fluid-elastic instability, clarify the transformation mechanism of each flow-induced vibration mechanism, and then provide data support for the design and optimization of the tube bundle structures of the steam generators. A fluid-structure interaction computational model was established based on the large eddy simulation (LES) method, the secondary development of user-defined functions (UDF), combined with the local dynamic mesh deformation technology using the square tube bundle structures as the object of study. Based on the refined fluid-structure interaction model of the tube bundle structures, the flow vibration calculations were carried out for the square tube bundle structures with only the center tube as the elastic tube and compared with the experimental results. The calculated streamwise and transverse vibration amplitudes as well as the critical flow velocity are in good agreement with the experimental values. The dominant mechanism of flow-induced vibration in the streamwise and transverse vibration as well as the transformation mechanism were analyzed and investigated by combining the vibration spectral response, vibration amplitude and the largest Lyapunov exponent. The results show that the established fluid-structure interaction computational model can accurately predict the critical flow velocity for the occurrence of fluid-elastic instability in the square tube bundle structures as well as the behavior of flow-induced vibration. The calculated critical flow velocities for fluid-elastic instability are in good agreement with the experimental values and the associated envelopes. The transverse vibration instability of the center tube is mainly caused by a combination of vortex-induced vibration and fluid-elastic instability, while fluid-elastic instability also occurs in the streamwise vibration. The vibrations of the center tube at different flow velocities show a weak chaotic state. When the vibration is dominated by a single mechanism, the largest Lyapunov exponent does not change much, while when the mechanism is transformed, the trend of the largest Lyapunov exponent changes. The dominant mechanism of the flow-induced vibration can be discriminated on this basis.