Motion capture real time

Scripting Issues
1

Hello,

Ok i’ve pretty much finished my game now and am very chuffed with how much I’ve achieved within 6 weeks. The only problem I have is connecting my avatar up to the motion sensor suit we have. I have tried the method shown on the manual page but nowhere.

As I dont have the suit available to me 24hours a day I have been experimenting with simply applying a pre-recorded .bvh file to my avatar.

I cannot seem to find any support out there on the web so if someone could point me in the right direction that would be great.

As for the real time mechanics, I believe the suit uses InterSense sensors and writes to .AC format which can then be converted to bvh in realtime (not 100% on this). If some info could be provided for how I go about getting this into panda in realtime would be great.

2

Not sure which method you’ve tried, but in principle, all you need to do is control all of the joints of your model, and connect each joint to the corresponding input. You also need to be sure your coordinate systems and initial joint transforms are the same.

If that’s not working for you, then I couldn’t begin to speculate on what’s going wrong without more specifics.

David

3

This seems to be the problem, I’m not entirely sure how I can connect the joints of my model to the input from the motion sensor suit. I have exposed all the joints and I’m pretty sure my model is good to go. The suit came prepackaged with software that manipulates a provided avatar and can also map to a character in motion builder but im confused on how I can relay this to my panda armature.

I’ve tried following the instructions on the manual page and imported the VRHandler script but get a coordiate error. Is there somewhere on the web that delves into this subject more so than what is catered for in the manual?

4

Well, what kind of data do you get out of your VR software? If you have a matrix, or a translate/scale/rotation trio, or something along those lines, you just have to set that on the controlled node for each joint.

Nothing I know of talks about this specifically in context of motion capture data, but it’s a pretty generic function that you’re trying to do. You’ve see the Eve demo in the panda samples directory that shows controlling a joint in Panda, right?

David

5

I have downloaded the SDK for the intersense trackers and I’m slowly reading through this to get a better sense of whats happening. I have exposed and set controlJoint for all my bones so the rig is ready to go.

I dont really get the technique of how I go about connecting the rotation matrix I’m getting and setting that to my bones, I’m using the mouseWatcher function shown in eve at the moment but I want to move from this to my suit.

Where can I find the VRHandler script and is there anywhere with documentation on this online?

Many thanks.

6

I don’t know what “VRHandler” refers to.

The rotation matrix you get is just that, a matrix. It’s the same kind of matrix you put on a joint with node.setMatrix(). You have to have a rig already set up that has rotation matrices in the same places corresponding with your motion-capture suit, then it’s just a matter of calling getMatrix() and setMatrix() for all of the corresponding joints.

Note that if your VR software is not giving you any translation information, just rotation information, it’s very important for you to set up your rig so that you’re only replacing every other joint, since the alternate joints will need to contain the translation information.

The stuff you’re trying to do is kind of outside the scope of Panda’s tutorials. Panda is, after all, a game engine, or a graphics engine. Panda will be happy to handle the rendering for you. And we’ll be happy to help you with the technical details of applying a matrix to a particular joint. But as to the philosophy and general strategy of hooking up your motion-capture data to your rig, you’re kind of expected to understand how that works already. :slight_smile:

David

7

Ok, for the moment I’m working with some pre-recorded actions that I got done today. The program stores all the rotation data as a .bvh file. Hopefully I can get my head around getting my character responding to this and then move into real-time.

Is there a way to look into the .bvh file and essentially ask to take a particular matrix and apply it to a particular joint. I have looked through the manual and online but cannot find any info on set & getMatrix.

Many thanks

8

Sorry, my bad–the method name is setMat and getMat. Once you have made a joint handle via controlJoint(), you call setMat() on the joint handle to apply the matrix to the joint. (The joint handle is just a NodePath. In addition to setMat(), you can also call setTransform(), setPos(), or setHpr(); these are all ways to apply a transform to a NodePath.)

This is also demonstrated in the “Looking and Gripping” demo that ships with Panda.

David

9

OK so I’m trying to write a parser that can take the motion values from a .bvh file and apply it to my bones.

I can read a line from the .bvh file (rot values) and store this in a multi-list. Eg, for one list entry I get 66 channels of motion data for that frame. I get 66 values as there are 22 bones in total each with x,y,z rot values.

[-0.010283, 95.249999, -0.010271, 0.000000, 0.000000…

The problem I have is that I want to extract from this list the first 3 values and apply that to my chest bone (using controlJoint and then setHpr). However, I cannot seem to work out how to do this. Furthermore is there a way to say: the first 3 values will be the chestHpr, the next 3 the rightShoulder hpr etc.

I have tried to use reshape, setshape functions but these can only be applied to arrays, and I have a list, plus I dont know if this is the best way of doing things as well.

Any help appreciated.

10

if you’r using python. you can do

yourList = eval( theloadedline )

this will tell panda to automatically evaluate the string you pass as argument.
afterwards you can access your floatvalues with

 yourlist[0]

or 1,2,3,4,5 whatever entry you need.

please note. eval() is not very secure so you should only use it on data which you can trust.
if you need a saver alternative you can use string operators to remove the brackets, and split the string at each “,” into a list 66 small strings. where you can use “float(yourSplitString)” to get a float out of the string-values.

11

thanks for the quick reply, I don’t know what I did but each value in the line are being placed in each subsequent index in the list :slight_smile:. I converted these to floats and it works :slight_smile:

Just wondering if there is a way I can dynamically create variables in a for loop.

At the mo i’m writing:

self.list1 = self.values[0]
self.float1 = float(self.list1)
            
self.list2 = self.values[1]
self.float2 = float(self.list2)
        
        self.chestC.setHpr(Point3(self.float[1],self.float[2],self.float[3]))

is there a way I can say for n in 66:

create self.list’n’ = self.values[n-1]
create self.float’n’ = float(self.list’n’)

etc…

if you get what i mean (saves me writing out a new float value for every joint’s Hpr)

Thanks again, Jake

12

It’s possible, but I would rather recommend storing a list of lists.
E.g:

self.lists = []
self.lists.append([1,2,3])

If you really want to do it your way, it would look something like this: (I strongly recommend against it)

for n in range(66):
  self.__setattr__("list%d" % n, self.values[n-1])
  self.__setattr__("float%d" % n, float(self.__getattr__("list%d" % n)))

I guess you could also do it with self.dict[“varname”].

13

Yeah thanks, tht was the first method I tried but I was trying to turn all values in the array to floats and was getting errors (i believe!?). Found my way round it eventually, thanks for the help guys

14

OK i think I’m getting closer but am having a problem getting my character in panda to match the true BVH animation. I believe it has to do with the joint transforms.

When the model is imported and played most of the joint animations dont match at all (apart from the chest that seems perfect). If I delete the transform matrix, of the leftShoulder for example, in the egg file I get a closer animation to my orginal but also get lots of deformations (i’m guessing this is because the joint isn’t placed to match the joint as in the suit)

What exactly does the 16 values refer to and how will I have to manipulate this or my original model to get a closer match to my original animation? The .bvh file comes with x,y,z values for the offset of each joint from the chest joint. I’ guessing I have to match each joint’s transform matrix to this offset but I have 16 figures from which to cover!

Thanks again for all your help, just really need to get this bvh parser working so I can go into uni and play my game in this suit!

15

I have matched my bvh data onto my model and the result is very close to the motion I recorded. I believe that all my rotation channels are matched up to my model’s joints correctly.

One problem I have is that sometimes the captured data for the x,y,z channels will sometimes invert by 180degrees. This will happen for all channels at the same time so I would expect to see no change in what is displayed. When this flip occurs the character will then turn in the opposite direction, much like a mirror image.

I assumed that when all HPR angles are increased or decreased by 180degrees the result should be the same as the initial position. Am I missing something?

16

Not necessarily. The HPR angles interplay with each other. You can load up a model (say zup-axis.egg) and reparent it to the scene to see what different HPR angles sum up to. For instance, it’s true that Hpr(0, 0, 0) is the same rotation as Hpr(180 + 0, 180 + 0, 180 + 0); but it’s not true that Hpr(10, 20, 30) is the same rotation as Hpr(180 + 10, 180 + 20, 180 + 30).

David