Not being a developer for Panda, nor having worked with Panda for any real amount of time, I can’t really say what’s happening in respect to shadows.
I think the main reason for your shadow fragments is that you’re using a rather low resolution shadow map, from too far away. Simple fixes would be to either move the light closer, or increase the shadow map resolution (which then impacts on FPS).
Alternatively, there are one or two things on the forums that might help you further (another option would be to write some shadow shaders yourself, which I assume would fall under the category of ‘too much effort/work’).
PSSM: [PSSM Shadows)
[Parallel-Split Shadow Maps (PSSM): Shadows for large scenes)
PCSS: [PCSS soft shadows)
I haven’t tested any of these myself. For my simple projects I’m mostly skirted around the shadow issue so far.
Anyway, hope that helps.