Microphone/voice stream on UDP?

C++ coding using Panda3D
1

Hi,

I’ve set up a voice communication between 2 Panda C++ programs running on 2 different machines. In order to do so I’m using a streaming of microphone input over the LAN via UDP packets.

It sort of works, with anyway a couple of issues:

some delay & stuttering
a kind of unexplained loop in the audio-receive reader (it seems that former received samples in the buffer are being read again - after about 4s)
echo (I’ll take care about this one)

Since the documentation is somewhat difficult to grasp, I’m wondering what’s missing in set-up, buffering or whatever…

Here is the relevant part of the code:

// SET UP AUDIO STUFF
PT(MicrophoneAudio)	    mic = MicrophoneAudioDS::get_option(2);		// micro 44100hz mono
PT(MicrophoneAudioCursorDS) mic_cursor = DCAST(MicrophoneAudioCursorDS,mic->open());	// audio cursor on micro stream

// create audiomanager
PT(AudioManager) AM = AudioManager::create_AudioManager();
PT(AsyncTask) audio_listen_task = new GenericAsyncTask("AudioUpdate", &audio_task, (void*) AM);
taskMgr->add(audio_listen_task);

// create audio send & rciv tasks
PT(AsyncTask) audio_send_task = new GenericAsyncTask("AudioSend", &audio_task_send, (void*) NULL);
taskMgr->add(audio_send_task);

PT(AsyncTask) audio_receive_task = new GenericAsyncTask("AudioReceive", &audio_task_receive, (void*) AM);
taskMgr->add(audio_receive_task);

// open remote sound stream
PT(UserDataAudio) remoteAudio  = new UserDataAudio(44100,1,true);  // remote sound input, 1 channel, remove after read
PT(AudioSound)    remote_sound = AM->get_sound(remoteAudio,false); // positional = false, AudioManager::StreamMode::SM_???
remote_sound->init_type();				           // is this needed? for what?

And the async processing

// PER FRAMe ASYNCHRONOUS TASKS
AsyncTask::DoneStatus audio_task_send(GenericAsyncTask* task, void* datanull){

#define MIC_FRAME_SIZE	2048	// mike buffers are 2048 samples (2bytes)
  PN_int16 samples[MIC_FRAME_SIZE];
 
  int nb_audio_samples_ready = mic_cursor->ready();
  if (nb_audio_samples_ready < MIC_FRAME_SIZE) return AsyncTask::DS_cont;

  // one mono mic sample is 2*bytes
  while (nb_audio_samples_ready >= MIC_FRAME_SIZE) {
    int nb_read = MIC_FRAME_SIZE;
    mic_cursor->read_samples(nb_read,samples); // read samples from microphone
    send_voice_on_UDP((char *)samples,nb_read);  
    nb_audio_samples_ready -= nb_read;
  }
  return AsyncTask::DS_cont;
}


AsyncTask::DoneStatus audio_task_receive(GenericAsyncTask* task, void* data){
  char datagram[VOICE_PACKET_SIZE];

  while (true) {
    int bytes_received = recvfrom(RecvSocketVoice, (char *) datagram, VOICE_PACKET_SIZE, 0, (SOCKADDR *)&SenderAddr, &SenderAddrSize);
    if (bytes_received < VOICE_PACKET_SIZE) break;
    int nb_audio_samples = bytes_received/2; // since 1 mono sample = 16bits
    remoteAudio->append((PN_int16 *) datagram, nb_audio_samples);
  }
  return AsyncTask::DS_cont;
}


AsyncTask::DoneStatus audio_task(GenericAsyncTask* task, void* data){
  AudioManager *AMgr = (AudioManager*)data;
  AMgr->update();	// local sound
  return AsyncTask::DS_cont;
}

Being stuck with something that is 85% working and still not confident if done properly, I suspect some audio set-up is not correct.

In particular, what the use of:

AudioManager::StreamMode::SM_xx ??? (heuristic, stream, …)
remote_sound->init_type();

Is there a way to insure that the Audio reader really drops each sample once played?

Any hint is more than welcome!

Thanks fy support

2

Looking at the code, it appears that AudioManager::StreamMode is respected by the OpenALAudioManager only, and it seems to be a hint as to whether to load a soundfile entirely into memory, or stream it from disk.

AudioSound::init_type() is part of Panda’s dynamic typing system, and there is no reason whatsoever to call it on your new sound instance. That doesn’t even make sense. It’s meant to be called on the class object at static init time, and Panda already does that.

David

3

ok, I get rid of AudioSound::init_type() (which anyway appeared to make no difference)

(1) should I leave int mode=SM_heuristic or SM_sample as get_sound parameter?

(2) How is it that some samples are played again after circa 4s, I would have thought that:
PT(UserDataAudio) remoteAudio = new UserDataAudio(44100,1,true);
meant that once audio samples (fed in with remoteAudio->append((PN_int16 *) datagram, nb_audio_samples); ) was played it won’t show up again!!!

4

(1) I don’t think this will make a difference for samples that you are feeding directly.

(2) I don’t know, sorry–I have little experience with this feature. It does sound like a bug, but whether the bug is in OpenAL itself or in Panda’s OpenALAudioManager is difficult to say from here. Are you in a position to explore the underlying code to research this further?

David

5

thks David,
I’ll investigate over this week-end.

I have a strange feeling that this is somewhat related to the mike and audio reader buffering, ie here we have open a monophonic microphone, with a sampling rate of 44.1khz, at 2 bytes/sample. The recording is done through 64 rolling buffers of 2048 samples (@1/20s per 2048 samples -> roughly 3 seconds). So I’m getting close to the approximate 4s… but this doesn’t make sense to explain why would the Audio reader loops back, unless it uses too buffers in a rolling mode…

6

Done additional investigation, but can’t really fing the root cause of the buffer replay issue, since the phenomenon varies over time (ie it sometimes works ok…)

Nevertheless, what I’ll try and do is to target a full implementation of pbx sip protocol, so that the features will be more comprehensive than a mere point to point voice exchange.

If anybody is interested to contribute to this development, I’ll be more than happy to join forces!