Olivia is based on Ubuntu Raring. If you have the amd64-version running, you can use my build: Panda3D downloads for several distros
If you run i386 or really want to compile yourself, here is a package list for orientation:
sudo apt-get install bison flex libboost-python-dev cmake libxxf86dga-dev libode-dev libvrpn-dev libfftw3-dev libtiff5-dev libpng12-dev libssl-dev libgtk2.0-dev libsquish-dev nvidia-cg-dev libopenal-dev libcv-dev libhighgui-dev libopencv-dev libeigen3-dev libfcollada-dev libavdevice-dev libavfilter-dev libbullet-dev fftw-dev libfmodex-dev libosmesa6-dev libcogl-dev
There can be errors, but you should get the point with it. You will need this PPA launchpad.net/~panda3d/+archive/ppa
You need to set up the PPA as you would run Ubuntu Quantal.