process UDP packets from eye tracker for research in autism

General Discussion
#1

Hello,

I’m using Panda in my research to better understand the cognition of children with autism who have very limited communication. I am a psychologist investigating what inferences can be made based on eye gaze behavior.

Children will watch/interact with the Panda app (computer 1) and an eye tracker (computer 2) estimates the gaze vector of the child and interpolates the screen coordinates of where the child is looking. This data can be sent out via TCP or UDP back to Panda. For simplicity, imagine the tracker simply sends a framenumber, and X, Y screen coordinates of where the child is looking (lots of other parameters are available).

I would ultimately like a class where one can specify the port no, and that has methods to get the current value of framenumber, gazeX, gazeY (and eventually any parameter of interest: head rotation, pupil dilation, etc. - all of which can easily be sent by the eye tracker). What I don’t know is whether you would instantiate this class within the main Panda DirectObject class (like most of the samples are set up) or elsewhere.

I’ve chosen Panda because of many of the tasks I am building should be quite simple. However, getting this networking worked out is an important preliminary step. Also, I will need to

Any input / ideas would be very welcome. The details of what I’ve done are below.

Thank you.

Jeff Munson, PhD
University of Washington, Seattle
jeffmun AT u DOT Washington DOT edu

Problem 1: Making the UDP data available in the Panda loop.

I can successfully read the UDP data with a simple Python nonblocking socket and then “print” it (see Example 1 below). However, whenever I try to use this data within the main class for the Panda program I can’t access it (see example 2 below which adds the tutorial scene and places the socket stuff into the Task Manager). I simply want to print the framenumber, gazeX, gazeY vars to the screen. Once I have access to them I will use them for collision detection, etc.

You can also note that in Example 1 the code is processing every other frame sent by the eye tracker (8965, 8967, 8969, etc. - framenumber is the first parameter sent). This is fine given the C# middleware I am using (I’ll talk about that in the next section). However, Example 2, that uses the taskMgr, has repetitions of the data and big jumps between processed frames (after 9454 repeats for a while it jumps to 9522 )

I’ve read the Panda manual about the Task Manager, and perhaps I need a truly multithreaded app. I am looking for ideas. The eye tracker sends data out at about 60Hz or about every 16-20ms. Lost packets aren’t as big a problem – I want Panda to have the latest data because the gaze data reflects what was on the screen about 50 ms prior as the eye-tracker needs time to process where gaze is directed so I need to keep lag at a minimum.

Problem 2: Bitstring conversion.

The UDP data which I can successfully read is actually a hack. I have a C# app in the middle that is getting the raw binary data from the eye tracker and then converting it to a string with the pipe “|” character delimiting each value. I had a functional C# class from the eye tracker company that does the conversion of the UDP packet data.

I’ve tried using the Python BitString library but I’ve had little success. I have the API documentation with the C++ header information that specifies the construction of the data for each available parameter. Each packet header contains the info that specifies what parameter from the eye tracker it contains.

The bottom line is that I need a solution that allows for fast reading of UDP data and that allows straight-forward access to these values within a Panda app. I can flexibly specify the port and the parameters that are sent from the eye-tracker. At this point I only need to process data from a single source.

####################################################################

### Example 1:
### No rendering

from socket import *
import direct.directbase.DirectStart

# Set the socket parameters
host = "localhost"
port = 4998
buf = 1024
addr = (host,port)

# Create socket and bind to address
UDPSock = socket(AF_INET,SOCK_DGRAM)
UDPSock.bind(addr)

# Receive messages
while 1:

    data,addr = UDPSock.recvfrom(buf)
    if not data:
        print "Client has exited!"
        break
    else:
        #print "\nReceived message '", data,"'"
        print globalClock.getRealTime(), data

# Close socket
UDPSock.close()

####################################################################

OUTPUT FOR EXAMPLE 1

DirectStart: Starting the game.
Known pipe types:
wglGraphicsPipe
(all display modules loaded.)

5.41296575673 8965|0|605|512|Screen26_SE5_3|0|0|0|0|0|0|0|0|
5.44873770646 8967|0|603|512|Screen26_SE5_3|0|0|0|0|0|0|0|0|
5.48429247197 8969|0|593|504|Screen26_SE5_3|0|0|0|0|0|0|0|0|
5.5224726992 8971|0|600|499|Screen26_SE5_3|0|0|0|0|0|0|0|0|
5.57194379461 8974|0|602|503|Screen26_SE5_3|0|0|0|0|0|0|0|0|
5.62606686219 8977|0|612|505|Screen26_SE5_3|0|0|0|0|0|0|0|0|
5.66200420815 8979|0|624|515|Screen26_SE5_3|0|0|0|0|0|0|0|0|
5.69867336462 8981|0|669|549|Screen26_SE5_3|0|0|0|0|0|0|0|0|
5.7367169366 8983|0|1122|543|Screen26_SE5_3|0|0|0|0|0|0|0|0|
5.76785128465 8985|0|1265|564|Screen26_SE5_3|0|0|0|0|0|0|0|0|
5.80107830165 8987|0|1217|583|Screen26_SE5_3|0|0|0|0|0|0|0|0|

####################################################################
####################################################################
####################################################################

### Example 2
### Render the tutorial environment, place UDP into taskMgr

import direct.directbase.DirectStart
#import math
import socket, traceback
from direct.task import Task
from direct.gui.OnscreenText import OnscreenText
from pandac.PandaModules import TextNode

font = loader.loadFont("cmss12")

host = ''
port = 4998

s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.setblocking(0)
s.bind((host, port))


def myProcessGazeData():
    try:
        k = s.recv(1024) 
        gazedata = k.split('|')
        print globalClock.getRealTime(), gazedata
        return gazedata
        
    except:
        return "pass"
        pass


class Demo():

    def GetPacketTask(task):
        #add network packet receive task here
        mygazedata = myProcessGazeData()
        if mygazedata != "pass" :
            myframenumber = mygazedata[0]
            mygazeX = mygazedata[1]
            mygazeY = mygazedata[2]
        return Task.cont


    # Function to put instructions on the screen.
    def addInstructions(pos, msg):
        return OnscreenText(text=msg, style=1, fg=(1,0,0,1), font = font,
        pos=(-1.3, pos), align=TextNode.ALeft, scale = .10)

    #Load the first environment model
    environ = loader.loadModel("models/environment")
    environ.reparentTo(render)
    environ.setScale(0.25,0.25,0.25)
    environ.setPos(-8,42,0)

    taskMgr.add(GetPacketTask, "GetPacketTask")

    # Post the gaze data
    inst1 = addInstructions(0.90,"Framenumber = ")  #  I WANT TO PRINT DATA HERE e.g, str(framenumber)
    inst2 = addInstructions(0.80,"Gaze X ="  )
    inst3 = addInstructions(0.70,"Gaze Y =" ) 
        
    
t = Demo()

run()

########################################################################
OUTPUT FOR EXAMPLE 2

DirectStart: Starting the game.
Known pipe types:
wglGraphicsPipe
(all display modules loaded.)
4.66029838752 [‘9454’, ‘0’, ‘668’, ‘989’, ‘Screen26_SE5_3’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’,’’]
4.76648718048 [‘9454’, ‘0’, ‘668’, ‘989’, ‘Screen26_SE5_3’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’,’’]
4.79641796332 [‘9454’, ‘0’, ‘668’, ‘989’, ‘Screen26_SE5_3’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’,’’]
4.80224129834 [‘9454’, ‘0’, ‘668’, ‘989’, ‘Screen26_SE5_3’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’,’’]
4.81705719292 [‘9454’, ‘0’, ‘668’, ‘989’, ‘Screen26_SE5_3’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’,’’]
4.83914562100 [‘9454’, ‘0’, ‘668’, ‘989’, ‘Screen26_SE5_3’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’,’’]
4.87189301086 [‘9454’, ‘0’, ‘668’, ‘989’, ‘Screen26_SE5_3’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’,’’]
4.94528701913 [‘9522’, ‘0’, ‘612’, ‘1038’, ‘Screen26_SE5_3’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’,’’]
5.02549992467 [‘9526’, ‘0’, ‘0’, ‘0’, ‘Screen26_SE5_3’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’,’’]
5.22064339639 [‘9538’, ‘0’, ‘745’, ‘901’, ‘Screen26_SE5_3’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’,’’]
5.28638850320 [‘9541’, ‘0’, ‘1040’, ‘828’, ‘Screen26_SE5_3’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’,’’]
5.33677965868 [‘9544’, ‘0’, ‘1025’, ‘834’, ‘Screen26_SE5_3’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’,’’]
5.40268233500 [‘9549’, ‘0’, ‘1026’, ‘847’, ‘Screen26_SE5_3’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’,’’]
5.55170069957 [‘9557’, ‘0’, ‘1012’, ‘878’, ‘Screen26_SE5_3’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’,’’]
5.63141281360 [‘9560’, ‘0’, ‘0’, ‘0’, ‘Screen26_SE5_3’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’,’’]
5.66849403927 [‘9564’, ‘0’, ‘1167’, ‘921’, ‘Screen26_SE5_3’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’,’’]
5.76626137880 [‘9569’, ‘0’, ‘1123’, ‘880’, ‘Screen26_SE5_3’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’,’’]
5.83253684223 [‘9573’, ‘0’, ‘0’, ‘863’, ‘Screen26_SE5_3’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’,’’]
5.98264497620 [‘9581’, ‘0’, ‘1079’, ‘770’, ‘Screen26_SE5_3’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’,’’]
6.09743918524 [‘9587’, ‘0’, ‘1069’, ‘788’, ‘Screen26_SE5_3’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’, ‘0’,’’]

########################################################################

#2

i once used pythons udp stuff. i guess panda’s building integratesbetter into the panda itself. since i did simmilar stuff you might want to look at it under : https://discourse.panda3d.org/viewtopic.php?t=6530

converting strings to the approriate type of data is easy.

 myframenumber= int( mygazedata[0] )

you can use that with pretty much every type suited. such as float() hex() str() etc.

#3

thomasEgi may have more experience with panda’s networking code, but i got the feeling that it does more then a normal network socket does. using python’s socket’s is probalby equally good, but it’s not really platform independant (non-blocking-socket dont work well on some platforms).

this may help, add “taskMgr.step()” in the while loop.


from socket import *
import direct.directbase.DirectStart

# Set the socket parameters
host = "localhost"
port = 4998
buf = 1024
addr = (host,port)

# Create socket and bind to address
UDPSock = socket(AF_INET,SOCK_DGRAM)
UDPSock.bind(addr)

# Receive messages
while 1:

    data,addr = UDPSock.recvfrom(buf)
    
    # this makes the panda rendering happen
    taskMgr.step()
    
    if not data:
        print "Client has exited!"
        break
    else:
        #print "\nReceived message '", data,"'"
        print globalClock.getRealTime(), data

# Close socket
UDPSock.close()

i didnt really see any other questions, but i may havent read it carefully enougth.

#4

btw. one thing that came into my mind right now:

the first example you made doesnt give repating packages as it’s non blocking, and thus waits for a new package to arrive. if you stop receiving packages from the network it will also stop the application from running.

the second example does non-blocking reads, it the readings are faster then the sending it’s obvious that you will get the same data multiple times. this can also happen because of network lags, or package drops. but it’s definately the better solution.

for the binary data conversion you may want to look into this:
docs.python.org/library/struct.html

#5

you could just check this out:
it came out from a kijaro snippet I adapted for your interesting project - it is fast and easy to use and use the panda api to manage network stuff.
I packed here a working sample with a client (TCP actually but you may easily turn to UDP) and a server that simulate (roughly) your tracker moving the mouse in his window - the blue square you’ll see is the kid glare.
Hope you’ll find useful.

#6

The sample astelix put together was incredibly helpful.

I needed to set rawmode to True and I got it to work. I will check out the other links regarding the binary data conversion so I can get rid of the C# middleware application and do everything in Panda.

Thank you for all your replies, this has been an amazing success as my first post on the Forums.

#7

hey happy to hear that!
keep us up to date on how it goes this interesting project