Building With Dynmically-Imported Modules

General Discussion
1

I’m struggling a little to produce a working distributable build in a game that dynamically imports certain elements.

Specifically, I have a development file structure that looks something like this:


<Root-dir>
|___ <Sub-dir>
|    |
|    |__ "__init__.py"
|    |
|    |_____<Sub-dir 2>
|          |
|          |__ "__init__.py"
|          |
|          |__ secondary module (Python scripts, sub-directories, assets, etc.)
|
|
|__ main program files (Python scripts, sub-directories, assets, etc.)

The main program, located in the root directory, dynamically discovers (via the VFS) and imports the secondary module, located in a sub-directory of a sub-directory. During development, this works as expected.

However, when I make a distributable build, things become more complicated: Naturally, the dynamically-imported module isn’t automatically detected during the build, and thus isn’t included in it.

My attempts to make use of the “include_modules” build-option have thus far failed–and I’ve used a number of values, including “<Sub-dir1>”, “<Sub-dir1>.*”, and “<Sub-dir1>.<Sub-dir2>.*”.

I’ve also tried just adding “<Sub-dir1>” to “include_patterns”, hoping to simply copy the whole thing in and have it be imported as in the development version. Here, the importation works–but it seems that the secondary module is then unable to import any modules from the main directory. :/

(That is, the following no longer works: Assume that within the root directory itself, there is a Python file named “Cat.py”. Assume further that within <Sub-dir2> there is a Python file named “Kitten.py”. During development, having “Kitten.py” import from “Cat.py” works–but in the built version it no longer does.)

Does anyone know where I might be going wrong, and how I might gain a working distributable build with the structure that I have…?

2

I tried dynamically importing stuff and didn’t really had a good time.

That being said, I just recently did something with templates. Which I also wanted to keep around relative to the build. The solution I found was the use of

import importlib.resources as pkg_resources
from . import templates
#...
my_temp = pkg_resources.read_text(templates, "template.html")

and in the setup.py it needed a hint that there is something I wanted to bundle as a resource.

package_data={'package_name': ['templates/*.html']},

(Root-dir)
setup.py
<package_name dir>
|    main.py
|    __init__.py
|____<templates dir>
|        template.html

Maybe that helps?

3

I believe you want <Sub-dir1>.**

4

Hmm… That seems to help indeed–thank you! :slight_smile:

But it also seems to means that the VFS can no longer discover the module (presumably because it’s frozen into the executable).

(I want the VFS to be able to see it in order to detect the presence of such secondary modules. Building them into executable isn’t my long-term intent–I want them to be located in separate “.mf” files eventually–but seemed like a potential short-term solution to making a demo.)

Hmm… That does look like it might be worth investigating, thank you. :slight_smile:

[edit 3]
Hmm… Looking briefly at the Python API on the matter, it looks like the module that you mention is more intended for non-code resources, while the main of my problem is with the dynamic importation of code.
[/edit 3]

[edit]
Perhaps it would help for me to explain less-abstractly (if still in broad strokes) what I’m attempting to do:

At the root, I have the “core” of the game, where I have implemented all of the game’s base logic–movement, player-handling, scenes, etc.

But I want specific game-elements to be modular, allowing not only the “default adventure”, but potentially others, too. To this end, there is a sub-folder named “Content” (“<Sub-dir1>”, above), which in turn contains sub-folders for each individual “adventure”.

Note that these sub-directories contain not only assets, but Python code too–classes for specific enemies and items, for example. Naturally, these inherit from base-classes found in the “game-core”.

(In the long run I want these modules to be held within multifiles–but as shown in another thread, that comes with its own complications…)

The “core” then uses the VFS to scan the “Content” directory, looking for modules to potentially make use of (via importation).

Hence my problem: Freezing these modules (aside from making them no longer modular) prevents the VFS from finding them; but if I simply have them be copied into place, they can no longer import from their frozen base-modules…

[edit 2]
As a fallback position for the purposes of my initial demo, if called for I daresay that I can comment-out the dynamic importation and just statically import the current secondary module. I’d rather not, but it would allow me to move forward, at least.

Still, should the project keep going I would presumably eventually want a solution…

5

Ah, okay, then we want to get the 'include_patterns' approach (treat the py files as assets) working and ignore 'include_modules’.

For your example with Cat.py and Kitten.py, does Kitten.py attempt a relative import, or does it just do import Cat?

6

Checking, in the case of the first error that I get, it seems to be a simple “from Cat import Cat” statement.

(Which gets me an error along the lines of “ModuleNotFoundError: No module named ‘Cat’”.)

I don’t think that I have any relative imports within the module–I think that it’s generally as the above–but I wouldn’t like to swear to it.

7

So, I had an idea, and I’d like to run it past the thread before I attempt it, if I may:

If I’m not much mistaken, the classes being imported by the sub-module (i.e. “Cat”, above) are all gameplay classes (e.g. the “Enemy” class, the “Projectile” class, etc.); I believe that none of the central game-classes (e.g. the ShowBase-derived “game” class, the main menu, etc.) are so imported.

As a result, perhaps I could package up the files that hold those gameplay classes into a multifile, to be dynamically mounted and imported by one of the central game-classes. That should, I think, mean that they’re not frozen into the final executable, and thus remain available to the sub-module for importation.

What do you think? Does that sound like it would work? And would there be potentially-undesirable consequences to doing so? (e.g. Performance considerations.)