"With open ..." and log-files

Scripting Issues
1

So, I recently learned that it’s relatively unsafe to simply open a file, assign the return-value to avariable, and then work with the resultant file-object. That is, to do something like this:

myFile = open(myFileName, "r")
myFile.readline()
# And so on

Instead, it’s apparently safer to use a “with”-statement, along these lines:

with open(myFileName, "r") as myFile:
    myFile.readline()
    # and so on

This apparently guarantees correct file-closing, outside of some edge-cases.

For the most part, switching to using “with” shouldn’t be a major problem, I think. The point at which I’m uncertain, however, is in the case of log-files.

In one of my projects, I’m redirecting log-output to a text-file, like this:

sys.stdout = open("logfile.txt", "w")
sys.stderr = open("logfile.txt", "w")

The code in question happens at the start of my main module, amongst that module’s import-statements.

In this case, how would I implement the “with”-pattern? Would I enclose the entire program in it, the whole thing indented? Something else? Or is it perhaps better to no redirect output in this way at all?

2

You may want to look into Python’s logging module and the corresponding howto article. If you don’t want to get into all of that, you can use sys.excepthook to close the file handles when the program is about to exit from an exception.

3

Hmm… Based on a cursory look, the logging module wouldn’t catch output sent to stdout or stderr, and thus wouldn’t catch any of Panda’s output or any uncaught errors–is that right?

If so, then sys.excepthook seems like by far the preferable option–not only would it allow me to catch pretty much everything, but it looks far, far simpler.

Are there any caveats with using sys.excepthook? Is there a reason that it would be a poorer choice?

4

How exactly do your sources describe it is “more safe” to use a with-block? The only situation I can imagine is if the functioning of the program relies on the file being explicitly closed at the end of a function that can throw exceptions. Unless you absolutely require the written contents to be flushed to disk immediately (eg. due to being read by a different part of the same program), neither “with” nor an explicit close() call are actually needed to “safely” close a file when you’re done using it, as Python files are closed automatically upon being garbage collected.

You shouldn’t try to use “with” in cases where doing so is clearly inconvenient or inappropriate, such as when assigning to sys.stderr/stdout.

5

I only have it second-hand myself, from @Moguri here: How do I compile a game built in panda3d version 1.10.0?

From what I see in the link that Moguri gave in the above-linked post, the basic problem is that if something goes seriously wrong while the file is open, using “with” guarantees that the file will nevertheless be closed; conversely, eschewing “with” may leave the file open, potentially leading to “memory issues” or the file being corrupted.

6

I don’t know what something as vague as “memory issues” means, but assuming “something goes seriously wrong” means that a Python exception is triggered in the function that is supposed to close the file, it will still be closed properly at the end of the function if it holds the last reference to that file object (or more generally, when the last reference goes away, or when the program exits), and there is no corruption or “memory issue”. If “seriously wrong” is interpreted as the Python interpreter crashing entirely, then some data you may have written to a writable file may indeed not have been flushed properly to the file, but the operating system will still close the open file descriptors. This is all defined behaviour.

I think that people are simply trying to say that it’s good practice to close your files explicitly when you’re done with them (even if just in the interest of writing self-documenting code), and with is inarguably the best way to do this in a variety of common scenarios (including in the specific scenario that @Moguri was commenting on), but I don’t think anyone is trying to suggest that any usage of file objects without use of with is an anti-pattern—that would actually make a lot of ways to interact with files impossible in Python!

7

What you say is somewhat reassuring–although I’d like to hear Moguri’s thoughts on the matter, should they see this discussion.

That said, the page to which Moguri linked explicitly calls non-“with” usage an “anti-pattern”, both by inclusion (it’s a document covering anti-patterns) and in the text itself.

8

I’ve not heard of QuantifiedCode so can’t comment on its reliability, but let’s look at the official Python documentation for why with exists and when it should be used with files:

It is good practice to use the with keyword when dealing with file objects. The advantage is that the file is properly closed after its suite finishes, even if an exception is raised at some point. Using with is also much shorter than writing equivalent try - finally blocks:

What this is essentially saying is that this code:

with open("file", "w") as fp:
    fp.write(...)

is intended as a short-hand to replace this code:

fp = open("file", "w")
try:
    fp.write(...)
finally:
    fp.close()

…which would be written as such to ensure that fp.close() still runs even if fp.write(...) fails with an exception.

In the case of assigning to sys.stdout/sys.stderr, you actually only care about the file being closed at the end of the program, so an explicit close() is not actually needed in this case as Python takes care of this during interpreter shutdown at the very latest. Though, if you wish, you can add an explicit close() during your game’s shutdown routine; you just need to take extra care to first assign stdout/stderr to something sensible before you do so.

The only way to do what you want to do with a with block is to wrap your entire game’s run loop with it; but then you actually still need a try…finally block to reassign the old stdout/stderr back, so the with short-hand doesn’t actually make things easier in that case.

9

Hmm… Fair enough.

But if an exception is raised and not handled, would the “finally”-block ever be called?

10

Yes. The finally block is always called.

11

Oh–fair enough, then! That is rather reassuring; thank you. :slight_smile:

I’ll still wait to hear Moguri’s thoughts before deciding, I think, since it was they who originally linked to that site and recommended against the non-“with” approach, but you’ve made a convincing argument.

12

I agree with @rdb. Use with when it makes sense. While it is “safer” in the general case, it really isn’t worth the hassle in this case.

13

Ah, fair enough! That’s a little less alarming than the page to which you previously linked, I feel. ^^;

But I am reassured, then. I’ll likely go through my code and replace where is seems reasonable to do so, but not worry about cases such as the logging redirection.

Thank you to all who gave aid in this thread! :slight_smile: