Abstract: The nonlinear effect analysis based on accelerator’s high order transfer mapping has advantages of clear physical view, symplectic and accuracy. This method is frequently applied to beam dynamics study of synchrotrons. Appling high order transfer mapping method to accelerator with complex magnetic field is difficult. For example, cyclotron’s orbit varies with the beam energy and its magnetic field is too complex to decompose into elements with clear transfer maps, such as dipole, quadrupole and sextupole magnets. RungeKutta or other numerical methods are usually used to simulate the orbit dynamics. However, numerical methods are incapable of analyzing high order nonlinear resonance clearly. In order to extend the application extent of the transfer mapping method, a novel neural network which simulating Lie operator was proposed in this paper. Every layer of the novel neural network has explicit physical meaning. The fully connected layer represents the linear transfer matrix while the Lie operator layer represents the nonlinear effects. A well-trained neural network can predict not only the nonlinear orbit of charged particle, but also the transfer matrix and Deprit decompose factor. To validate the effectiveness of this novel neural network, a thin lens lattice section was used as an example. Numerical orbit tracking data were used to train the neural network and the trained neural network is capable of predicting the nonlinear orbit of charged particle and works well on the test data set. Root mean square loss function of the test data set is less than 8×10-4. The transfer matrix and Deprit decompose factor predicted by the novel neural network are entirely consistent with analytic results. The neuralnetworkbased nonlinear effect analysis method was applied to a 70 MeV isochronous fixed field alternating gradient (FFAG) accelerator which has four fold symmetry magnetic fields. This FFAG accelerator was designed to produce more than 5 mA proton beam. The radial tune footprint excursion varies from 1.1 to 1.4 while the beam is accelerated and the 3vr=4 resonance driven by the four fold symmetry magnetic fields is crossed. The 3vr=4 resonance is an intrinsic third order nonlinear resonance whose beam blowup effect should be carefully estimated, especially for high intensity accelerators. The neuralnetworkbased nonlinear effect analysis method can clearly show the phase space distortion effect caused by any nonlinear resonance. As a datadriven algorithm, the limitation of this neuralnetworkbased method is that it will require high quality data to train the network. Future works are also discussed in this paper.