Flat file or Database?


I’m aiming to create a single player open world spacesim. I’ve been messing with Panda for a few weeks - and I’m liking it so far :smiley:

I’m currently wondering which would be better - to store data of planets/fleets/quests in a flat file, or in a database?

I’m thinking of a DB to hold information that won’t change - system/galaxy/dialog, and one that holds changable information on quests/fleets/npcs

Is there much of a difference between the two, in regards to ease of reading/writing and performance?


My general rule of thumb is, use whatever solution is easiest, unless there is a compelling reason to use a more difficult solution.

In this case, I’d recommend using a flat file, since that’s generally less work than a database system (you don’t need to ensure a separate database program is installed and running). In fact, Python provides a particularly easy file format for reading and writing arbitrarily complex Python objects to a flat file via the pickle module.


Thanks for that David.

I had a look at Pickle, seems like a very easy way to save data.

Although Pickle seems more suited to saving the current state of objects, maybe I need to look at XML for storing dialog etc.

If anybody else is looking at this, I wrote up some quick code using Pickle below.

# This is a fairly simple example of using Pickle to save a list of Zoos to a file, and then 
# reading the saved Zoos, which are printed to the screen.

from pandac.PandaModules import *
import direct.directbase.DirectStart
from direct.gui.OnscreenText import OnscreenText
import pickle
import pprint


def genLabelText(text, i):
    winHeight = base.win.getYSize() 
    ppu = 16
    return OnscreenText(text = text, pos = (-1.3, .95-.05*i), fg=(1,1,1,1),
                      align = TextNode.ALeft, scale = (2.0 * ppu) / winHeight)

class Panda:
    def __init__(self, n): 
        self.name = n
class Zoo:
    def __init__(self, n): 
        self.name = n
        self.pandas = []

    def addPanda(self, p):

class World:
    def __init__(self):
        # Lets create some data - A couple of Zoos with some Pandas in them
        self.zoos = [];
        self.zoos.append(Zoo("Red Panda Zoo"))
        self.zoos.append(Zoo("Perth Zoo"))
        #Specify the file to create
        output = open('zoo.pkl', 'wb')        
        #Create a pickle object from the open file
        p = pickle.Pickler(output)
        # Dump the object to the file
        # Close the data stream
        # Reopen the file
        input = open('zoo.pkl', 'rb')

        # Put the pickled data into a new variable
        self.zoosFromFile = pickle.load(input)

        genLabelText("The data below was saved and read using Pickle", 0)

        # Output the data to screen        
        line_num = 2
        for zoo in self.zoosFromFile:            
            self.zooText = genLabelText(str(zoo.name), line_num)
            for panda in zoo.pandas:
                line_num += 1
                self.zooText = genLabelText("- " + str(panda.name), line_num)
            line_num += 1
#Create the world        
w = World()

#Run it

Keep two things in mind:

  • The cPickle module has been tested to be 1000 times faster (no exaggeration) than the pickle module - since cPickle is implemented in C, while the behavior barely differs.
  • It is possible to pickle harmful pieces of code, that are executed when unpickling it. So do not use pickle for anything that other users might be able to alter. Google for ‘python pickle unsafe’ and you’ll find everything about it.

Hmm, interesting reading.

I might choose YAML. Its sounds a bit more secure, and has a function that looks to do some sanitising of data (safe_load). Its also more human readable then pickle and less bulky then XML.


i am a fan of yaml myself. Going flat file will also make your game more modable if you want the player to have that ability

I think the paranoia about pickle is often overstated. For the purposes of saving local files, pickle is a perfectly fine choice, and is just as safe as anything else. To put it in perspective: if a hacker wants to corrupt your code, he doesn’t have to mess around with sabatoging your data files; he can just modify the code directly.

The only time you’d want to avoid using pickle for security reasons is if you’re running a program on a secure server that’s loading data files from the internet, or passing data files between users, or something along those lines.

Of course, it’s also true that pickle can’t (directly) store things that are pointers to non-Python objects like dialogs.