Quake3 BSP loader (proof of concept)

ThomasEgi · December 17, 2009, 7:16pm

does panda need a Q3 bsp loader?

definetly, great addition
i dont care…
no… complete waste of time to implement it

0 voters

i almost proudly announce:

the first prototype for loading quake3 mapfiles into panda.
for now it’s a stand-alone script written in python. converting the bsp into an egg.

supported features so far are:
-vertices and trifans
-vertexcolors,normals, uv (both texture and lightmap)
-texture handling

what’s still missing:
-a LOT. especially triangle-geometry (see the holes in the map)
-loading of lightmaps
-handling of special-effects / shaders
-loading of empties
-the entire PVS and BSP stuff.
-porting it to c++ and integrating into the core engine.

currently converting an average q3 map takes a bit less then a second.
to give you an idea about it’s current status:

if you feel like helping feel free to contact me.

blenderkid · December 17, 2009, 9:01pm

This is a good idea. I’d like to know more about it.

I don’t know much about q3 level-editing, but I ever liked to play it

How are the maps created? Using blender?

blenderkid

ThomasEgi · December 17, 2009, 9:27pm

i took some original quake3 bsp maps for testing purpose. you can create q3 maps with all q3 map editors. there should be TONS of info on the internet. radiant might be amongst the more famous ones.

i continued to add stuff. i can now correctly load and display independant triangle-mesh-faces. without texture for now. so the holes in the map are closing more and more

[EDIT] textures now load on pretty much all surfaces… only a few triangles are missing aswell as curved surfaces.

ThomasEgi · December 17, 2009, 9:47pm

ok. here for the first “dirty” pieces of code.
they are quite a mess but i aimed for functionality so far.

this code can convert most (no billboards, no curved surfaces) geometry including most textures (they are assumed to be .jpg files) into an egg file which can be loaded by panda directly.

so far there are no nodes, no hiarchy no BSP-calculations and no potential visibility stuff in there. just getting the geometry with textures. i’ll try to add more features. next would the loading of lightmaps.

so here’s the code, in case my train crashes tomorrow

#python based reader for quake3 BSP files according to http://www.mralligator.com/q3/
#written by Thomas Egenhofer, released under BSD license.
#if you have any questions contact me: antitron@web.de

import struct
infile = open('/media/ext-hdd/Modellstuff/test/bsptests/maps/q3dm17.bsp',"rb")

outfile = open('./outfile.egg',"w")
outfile.write("<CoordinateSystem> { Z-Up } \n")
outfile.write("<Comment> {   }  \n")


#start reading the mapfile.. first comes the header. it contains magic number, version number
#and some information where to find what other data.
#that information is stored in 17 "Lumps". each containing different data.
magic = infile.read(4)
versionnumber = struct.unpack("<i" , infile.read(4))[0]
print "magic number:", magic
print "version number" , hex(versionnumber)



#for the entities lump in the bsp file.[ offset from file beginning, length of the data]

def readLumpEntry():
    lumpEntry= [ struct.unpack("<i", infile.read(4))[0] ,
                 struct.unpack("<i", infile.read(4))[0] , ]
    print lumpEntry                 
    return lumpEntry
print "entities lump",
Entities    = readLumpEntry()   ;print "Textures lump", 
Textures    = readLumpEntry()   ;print "Planes lump",        
Planes      = readLumpEntry()   ;print "Nodes lump",       
Nodes       = readLumpEntry()   ;print "Leafs lump",
Leafs       = readLumpEntry()   ;print "LeafFaces lump",
Leaffaces   = readLumpEntry()   ;print "LeafBrushes lump",             
Leafbrushes = readLumpEntry()   ;print "Models lump",            
Models      = readLumpEntry()   ;print "Brushes lump",           
Brushes     = readLumpEntry()   ;print "Brushsizes lump",
Brushsides  = readLumpEntry()   ;print "Vertices lump",
Vertices    = readLumpEntry()   ;print "MeshVerts lump",
Meshverts   = readLumpEntry()   ;print "Effects lump",          
Effects     = readLumpEntry()   ;print "Faces lump",       
Faces       = readLumpEntry()   ;print "Lightmaps lump",             
Lightmaps   = readLumpEntry()   ;print "Lightvols lump",           
Lightvols   = readLumpEntry()   ;print "Visdata lump",           
Visdata     = readLumpEntry()

print " END OF HEADER "
print 

textures = []
####handling textures
def readTexture(offset):
    infile.seek(offset)
    texturename = infile.read(64).rstrip("\0")
    #global textures
    textures.append(texturename)
    sourceFlags = struct.unpack("<i", infile.read(4))[0]
    contentFlags= struct.unpack("<i", infile.read(4))[0]
    #print 
    #print "texturename:" , texturename
    #print "sourceFlags:", hex(sourceFlags) , "contentFlags:",hex(contentFlags)
    
def readTextures(offset,length):
    numtextures = length/(64+4+4) #thats 64chars,4byte int, 4 byte int  
    print "Number of Textures",numtextures
    for i in range(0,numtextures):
        readTexture(offset+ i*(64+4+4) )
    
readTextures(Textures[0],Textures[1])
print "END OF TEXTURES"
###### end of texture handling

for i in textures:
    outfile.write("<Texture> tex"+str(textures.index(i))+" { \n\t./"+i+".jpg \n\t<Scalar> uv-name { Diffuse } \n}" )


###handling meshvertices ### //nasty offsets values in the vertex-list later on.. i still dont get what it's good for..
meshVertsList=[]
def readMeshverts(offset,length):
    infile.seek(offset)
    numMeshverts = length /  4
    print 
    print "number of meshVertices:", numMeshverts
    for i in range(0,numMeshverts):
        meshVertsList.append(struct.unpack("<i", infile.read(4))[0])

readMeshverts(Meshverts[0],Meshverts[1]) 
print meshVertsList

#### handling vertices YAY!.. we def need those
def readVertex(offset):
    infile.seek(offset)
    vertexPos =  [  struct.unpack("<f", infile.read(4))[0],
                    struct.unpack("<f", infile.read(4))[0],
                    struct.unpack("<f", infile.read(4))[0] ]
                    
    vertexUVDif = [ struct.unpack("<f", infile.read(4))[0],
                    struct.unpack("<f", infile.read(4))[0] ]
                    
    vertexUVLight=[ struct.unpack("<f", infile.read(4))[0],
                    struct.unpack("<f", infile.read(4))[0] ]
                 
    vertexNormal =[ struct.unpack("<f", infile.read(4))[0],
                    struct.unpack("<f", infile.read(4))[0],
                    struct.unpack("<f", infile.read(4))[0] ]   
     
    vertexRGBA  = [ struct.unpack("<B", infile.read(1))[0],                          
                    struct.unpack("<B", infile.read(1))[0],
                    struct.unpack("<B", infile.read(1))[0],
                    struct.unpack("<B", infile.read(1))[0] ]
                    
    #print "vertex Pos:",vertexPos
    #print "vertex UV Diffusive:",vertexUVDif
    #print "vertex UV Lightmap:",vertexUVLight
    #print "vertex normal:", vertexNormal
    #print "vertex color:",vertexRGBA
    ####writing to egg file:
    outfile.write("\t\t\t"+repr(vertexPos[0])+ " " +repr(vertexPos[1]) +" "+repr(vertexPos[2]) +"\n" )
    outfile.write("\t\t\t <Normal> { " +repr(vertexNormal[0])+ " " +repr(vertexNormal[1]) +" "+repr(vertexNormal[2]) +"}\n" )
    outfile.write("\t\t\t <RGBA> { " +repr(vertexRGBA[0]/255.)+ " " +repr(vertexRGBA[1]/255.) +" "+repr(vertexRGBA[2]/255.)+ " " +repr(vertexRGBA[3]/255.) +" }\n" )
    outfile.write("\t\t\t <UV> Diffuse { " +repr(vertexUVDif[0])+ " " +repr(1-vertexUVDif[1])+" }\n" )
    outfile.write("\t\t\t <UV> Lightmap { " +repr(vertexUVLight[0])+ " " +repr(vertexUVLight[1])+" }\n" )
    outfile.write("\t\t\t}\n")
    
def readVertices(offset , length):
    outfile.write("<Group>{\n")
    outfile.write("\t<VertexPool> BSPmap {\n")    
    
    numvertices = length / ( 3*4 + 2*4 + 2*4 + 3*4 +4*1) 
    print 
    print "number of numvertices:", numvertices
    for i in range(0,numvertices):
        #print 
        #print "vertexNr:" ,i
        
        outfile.write("\t\t<Vertex> %i { \n"%i)
        
        readVertex(offset+ i* ( 3*4 + 2*4 + 2*4 + 3*4 +4*1) )
    outfile.write("\t\t}\n")
readVertices(Vertices[0],Vertices[1]) 
print "END OF VERTEXDATA" 
#### end of handling vertices #####


#### handling of faces #####
def readFace(offset):
    infile.seek(offset)
    
    texture         = struct.unpack("<i", infile.read(4))[0]
    effect          = struct.unpack("<i", infile.read(4))[0]
    facetype        = struct.unpack("<i", infile.read(4))[0]
    vertex          = struct.unpack("<i", infile.read(4))[0]
    nVertices       = struct.unpack("<i", infile.read(4))[0]
    meshVertex      = struct.unpack("<i", infile.read(4))[0]
    nMeshVerts      = struct.unpack("<i", infile.read(4))[0]
    lightmapIndex   = struct.unpack("<i", infile.read(4))[0]
    lightmapStart   =[  struct.unpack("<i", infile.read(4))[0],
                        struct.unpack("<i", infile.read(4))[0] ]
    lightmapSize    =[  struct.unpack("<i", infile.read(4))[0],
                        struct.unpack("<i", infile.read(4))[0] ]
    lightmapOrigin  =[  struct.unpack("<f", infile.read(4))[0],
                        struct.unpack("<f", infile.read(4))[0],
                        struct.unpack("<f", infile.read(4))[0] ]
    
    lightmapVecs    =[ [struct.unpack("<f", infile.read(4))[0],
                        struct.unpack("<f", infile.read(4))[0],
                        struct.unpack("<f", infile.read(4))[0] ],
                        
                       [struct.unpack("<f", infile.read(4))[0],
                        struct.unpack("<f", infile.read(4))[0],
                        struct.unpack("<f", infile.read(4))[0] ] ]
                        
    normal          =[  struct.unpack("<f", infile.read(4))[0],
                        struct.unpack("<f", infile.read(4))[0],
                        struct.unpack("<f", infile.read(4))[0] ]
    
    patchSize       =[ struct.unpack("<i", infile.read(4))[0],
                       struct.unpack("<i", infile.read(4))[0] ]
                                               
    """
    print 
    print "texture index:",     texture
    print "effect index:",      effect
    print "face type:",         facetype
    print "vertex index:",      vertex
    print "number of vertices:",nVertices
    print "mesh vertex:",       meshVertex
    print "number of mesh vertices:", nMeshVerts
    print "lightmap index:",    lightmapIndex
    print "lightmap Start:",    lightmapStart
    print "lightmap Size:",     lightmapSize
    print "lightmap Origin:",   lightmapOrigin
    print "lightmap Vectors:",  lightmapVecs
    print "face normal:",       normal
    print "patch size:",        patchSize
    """
    """
    if facetype == 1 :
        outfile.write("\t\t<Polygon> {\n")
        outfile.write("\t\t\t<VertexRef> { ")
        for j in range(0,nVertices):
            outfile.write(str(vertex+nVertices-j-1)+" ")
        outfile.write("<Ref> { BSPmap } } \n")
        outfile.write("<TRef> { tex"+str(texture)+" }\n") 
        outfile.write("\t\t}\n")
    """
       
    if facetype == 3 or facetype==1:
        for n in range(0,nMeshVerts/3):
            outfile.write("\t\t<Polygon> {\n")
            outfile.write("\t\t\t<VertexRef> { ")
            for j in range(0,3):
                outfile.write(str(vertex+ meshVertsList[2+meshVertex+3*n-j] )+" ")
            outfile.write("<Ref> { BSPmap } } \n")
            outfile.write("<TRef> { tex"+str(texture)+" }\n")
            outfile.write("\t\t}\n")
        
    print facetype,vertex,nVertices,meshVertex,nMeshVerts
def readFaces(offset , length):
    numfaces = length / (12*4 + 12*4+2*4) 
    print 
    print "number of numFaces:", numfaces
    for i in range(0,numfaces):
        #print 
        #print "faceNr:" ,i
        readFace(offset+ i* (12*4 + 12*4+2*4) )
readFaces(Faces[0],Faces[1]) 
outfile.write("}")
outfile.close()
print "END OF FACEDATA" 
#### end of handling faces ####

#print textures

ThomasEgi · December 18, 2009, 12:32am

small update before i go to bed… i just added lightmap support including automatic unpacking from the bsp file.

code can be found here:
http://home.arcor.de/positiveelectron/files/bspreader.py.zip

PS: code is still heck of a mess but it’s functinality is gradually improving

[EDIT] disabled vertexlight when using lightmaps, also fixed a tiny lightmap-assignment bug. it now looks pretty close to what it does in the original game.

treeform · December 18, 2009, 2:08am

I think i looked into writing some thing like this before. This is definitely great addition.

Xidram · December 18, 2009, 7:07am

Yeah, this is awesome. Not only would this be another format supported by Panda, which in itself would be great, but also this is a rather popular map format used in plenty of games.

ThomasEgi · December 19, 2009, 12:10am

tiny update:
+added support for loading curved surface patches (thought they are not tesselated yet so not really curved at all)

this makes geometry-loading itself complete. the tesselation code aside which will increase prettyness a lot.

since i need to access vertex-data for the tesselation i need to do a major cleanup/rewrite.

i updated the link to the zip-file in case you feel like playing around with it.

ThomasEgi · December 19, 2009, 8:40pm

small status update on curved surfaces with tesselation but no uv corrds and thus no texturing: (curved stuff is colored deep red)

this was done using pandas internal nurbs surfaces. but looks like i still have to write my own tesselation code due to texture/UV

Praios · February 13, 2010, 4:48pm

you could use Automatic Texture Coordinates for the nurbs surface or - but im not sure how that works - project the textures

radu · February 14, 2010, 10:10am

Great stuff. Latest version available at the same link?

ov3rcl0ck · February 15, 2010, 4:26pm

Will this be available in up coming version of panda? I’d really like to have my Modeler use quake map creator or just use the quake format. It will really help develope content faster.

Thanks.

ThomasEgi · February 15, 2010, 9:18pm

i havent turned it into a fully-fledged converter. it would still require some work to make it versitale.
for now it’s still very basic and needs more work with texture naming, texture combine modes for transparent stuff, uv mapping for curved surfaces (you’r ok if you dont use them). propper CLI interface is also missing. and you have to manually unpack textures from the .pak file in question.
all those things are quite resonable. the hard part was getting the geometry right. please note that this does not support BSP itself. the entire geometry is exported as one big model for now.so you might want to run the eggoctree script on it.

aside from that the latest version can be found at
http://thomaseg1.p3dp.com/samples/

if you have serious need for a certain feature i might be able to add it within a reasonable ammount of time.

ThomasEgi · January 17, 2011, 2:09am

small updated:

thomaseg1.p3dp.com/samples/bsp2egg.py

now comes with a proper cli. also added experimental dumping of the collision solids.

the thing is. those are convex-solids, ODE supports those, so should panda.
in panda’s apiref they show up on the c++ side but not on the python end!

any chance someone simply forgot to wrap it up in python? and if so. pretty please for a bugfix