The X format is broken and the animation doesn't play

Now that you mention it, I seem to recall that that’s normal behaviour in Panda: Actors to which you don’t keep a reference don’t animate.

(This doesn’t, however, speak to animations not working with “.x” files.)