Abstract: The flexibly connected rotor system with a high rotating speed under flexible bearings easily causes the rotor dynamic instability, and the internal damping in connections is the primary factor. The complex modulus internal damping model in rotor connections was achieved by using the Kelvin-Voigt viscoelasticity theory, and the imaginary part in the complex modulus which stands for the internal damping was obtained through the relation between the loss factor in Kelvin-Voigt viscoelasticity theory and the modal test. The dynamic equation of the flexibly connected rotor system was established using Lagrangian method. With the experiment results the model was qualitatively verified, and the influences of internal damping in connections and bearing coefficients of the vibration absorber on rotor dynamic instability were investigated. The results show that increasing the internal damping in connections will tremendously reduce the rotor dynamic stability, increasing the stiffness and decreasing the effective mass in the vibration absorber will improve the rotor dynamic stability and the stability design needs taking both the connections and vibration absorber into consideration.