Unified Audio-Video subsystem using FFMpeg and OpenAL

I am in the process of creating a unified audio/video subsystem for panda. On the surface, this will be just another AudioManager and another MovieTexture. However, the two will both pull data from an underlying “Movie” class which does the file IO and yields up video frames and audio samples. The advantage of having an common underlying file reader is that it means the audio system can play the same file as the video system, at the same time. That, in turn, means that you will be able to create cut-scene movies with both video and audio! It also means you’ll be able to synchronize the video to the audio accurately.

Furthermore, since the AudioManager will be based on OpenAL, the audio system will be truly free software, for the first time. In the past, there have been other audio managers based on OpenAL. Most of them, however, did not support MP3, OGG, or AVI — not suprising, since OpenAL doesn’t include decoders for these formats. But since the underlying Movie layer (based on ffmpeg) can handle all of that, all these formats will be supported.

I should also mention that you shouldn’t take the word “movie” too literally. I count an MP3 file as a “movie” with good audio, but blank video.

i cant wait for this to be finished. I have tried to use pygame sound mixer to play an mp3 then used my own openAL (ctypes) lib to play wav in 3d. It did not sound good with different sound systems. But then sound on linux is crap any ways.

Update: the video portion of the system is now fully operational. Time to get started with the audio.

Great news! Keep up the good work.

Update: the code for this is finished. I also wrote a new sample program, “Media Player,” to demonstrate how it works. I’ve successfully played several DivX music videos with it, I haven’t seen any problems. Lip synchronization is very reliable, even if you pause, resume, or change the play rate of the audio. Caching behavior is good too: it loads audio data when it is actually playing, discards it when the sound is stopped, and maintains a cache of recently-used sounds. It can stream big files, no problem. I’m very happy with it.

I’m going to include it in 1.4.1. I know that normally, one wouldn’t include a major new feature in a non-x.x.0 release, but I really need to release this so that my students can use it, and furthermore, you can turn it off just by setting some config variables, so I think it’s ok.

Well, will be nice to see that in action. :slight_smile:

I wonder if it will allow some sort of streaming media?

Regards, Bigfoot29

It has hooks for streaming media, but it isn’t implemented yet.

Sounds great! (no pun intended) Is it in CVS so we can give it a try?

Yes, it is. The CVS head has the new code. To activate it, you have to set a few config variables:

use-movietexture #t
audio-library-name p3openal_audio

I am in the process of creating a CVS tag panda3d_1_4_1 as well. I think I have the tag in the right place. That, too, contains the code.

Oh, wait… one more thing. I recompiled OpenAL myself and called it “pandaopenal32.dll” in order to avoid DLL hell. You might have to do the same thing to get this to work, either that or dork around with makepanda to change the library name.

good think this is moving do you have the manual right or is it the same way as the current sound system works?

It’s almost exactly the same. Only difference is that “loadSfx” and “loadTexture” can now both handle AVI / MOV etc. Then, there’s one new function: MovieTexture.synchronizeTo(AudioSound). When you play the audio, the texture follows along and stays in sync.

Josh, there is still a problem.
I just downloaded Panda3D 1.4.1 (the dapper deb) and tested the Media Player sample, after I found out that I needed to put this in Config.prc:

audio-library-name p3openal_audio
use-movietexture #t

At first it plays fine, but at a random moment in the movie it segfaults. Every time I run it, it segfaults at a different moment. I -sometimes- get this too:

alsa_blitbuffer: Could not write audio data to sound device: Input/output error

but less often than the segfaults.

The strace output gives not much special, except the last lines:

mmap2(NULL, 141504512, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xaa8fc000
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
Process 15471 detached

Another thing: when I close the window, panda3d isn’t terminated. The window is closed, but I need to press ^Z on the commandline to have it closed entirely.

Update: I’m pretty sure that this is because the free implementation of OpenAL was a disaster about nine months ago. I think the current head revision of OpenAL might be better. I’m going to try it.

… update: the new version works much better. Still not perfect, but close.

Do you know if doppler shifting works in the current head revision of OpenAL? It didn’t work for me a few months ago …

Well, I don’t know that, but I do know that AL_PITCH works. Since pitch-shifting and doppler are very closely related, it seems likely to me that if one works, so would the other.

When I was playing around with this pitch worked, but doppler didn’t. Therefore I implemented doppler myself in the audioSound class.

However, this is really something that OpenAL should be able to do itself. I hope the linux implementation will be improved by someone…

the only thing i get with the new openAL is odd clicking. I dont think its an openAl problem because i use the same openAl with ctypes and it works fine. I will investigate some more.

I found the cause of the clicking. Under linux, calling alcMakeContextCurrent causes a “pop.” I’ve rearranged the code so that we don’t call that function any more.

wow great! The clicking was my only problem

PS: where is the new download zip?