Importing a Non-Dynamic Script From a Dynamically Imported One

This is essentially a continuation of this old thread.

So, as mentioned in the thread above, I have a setup in which certain modules are not frozen for distribution, as the project’s “base” modules are.

Instead, they’re kept more-or-less as-is, and imported dynamically by the “base” program.

As far as that goes, it seems to work.

However, I’ve now found that such modules don’t seem to be able to import anything from the base, frozen modules–which is a bit of a problem for me.

For example, consider the following hierarchy:

<base-dir>
   |
   |___ base_script.py
   |
   |___ some_class.py
   |
   |___<sub-dir>
         |
         |___ nonfrozen_class.py
         |
         |___ __init__.py
         

There, as the name suggests, “nonfrozen_class.py” is not frozen by setuptools when a distributable is made.

Further, presume that “base_script.py” imports “nonfrozen_class.py”, and further that “nonfrozen_class.py” imports “some_class.py”.

Outside of a frozen, distributable build, the above works.

Within a frozen, distributable build, however, the above crashes when “nonfrozen_class.py” attempts to import “some_class.py”. :/

I’ve attached here a simple test-program that–on my machine at least–demonstrates the issue.
importTester.zip (1.9 KB)

It contains the following files:

  • setup.py
  • requirements.txt
  • core.py
  • simpleClass.py
  • SubClasses/secondary_dynamic.py
  • SubClasses/__init__.py

The main program is “core.py”.

This program dynamically imports from “secondary_dynamic.py”, which in turn (statically) imports from “simpleClass.py”.

If you simply run “core.py”–i.e. without building a frozen version–you should see the following output:

Moggy Instantiated!
A Simple Class Instantiated!
I am a Moggy

The first line is printed by the class in “secondary_dynamic.py” on being instantiated by “core.py”. The second line is then printed by the class in “simpleClass.py” on being instantiated by the class in “secondary_dynamic.py”. And finally, the third line is printed by the class in “secondary_dynamic.py” at a function-call from the core program.

However, if you then attempt to build the program and run it, you should see something like the following:

Traceback (most recent call last):
  File "__main__", line 4, in <module>
  File "importlib", line 126, in import_module
  File "importlib._bootstrap", line 1050, in _gcd_import
  File "importlib._bootstrap", line 1027, in _find_and_load
  File "importlib._bootstrap", line 1006, in _find_and_load_unlocked
  File "importlib._bootstrap", line 688, in _load_unlocked
  File "importlib._bootstrap_external", line 883, in exec_module
  File "importlib._bootstrap", line 241, in _call_with_frames_removed
  File "/home/thaumaturge/Documents/My Game Projects/BugHunt/ImportTester/Frozen/dist/Import Tester-1.0.0_manylinux2014_x86_64/Import Tester/SubClasses/secondary_dynamic.py", line 2, in <module>
    from simpleClass import SimpleClass
  File "importlib._bootstrap", line 1027, in _find_and_load
  File "importlib._bootstrap", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'simpleClass'

As you can see, the code seems to get into “secondary_dynamic.py”, as intended–but from there doesn’t seem to be able to get back to the frozen modules.

Does anyone know how I might get this working…?

[edit]
*Bump*

I did have a thought: Might this work better if the dynamically-imported script were itself independently frozen…?

That is, if one had the dynamically-imported script compiled to a library file or some such–still referencing things from the main program, mind you–that could be dropped into place…?

You don’t need to froze your dynamically imported Python script, but there are several caveat that you need to take care of. First, the freezing mechanism uses a concept called import chasing: The frozen code will track the import statements of your code and freeze only the explicitly imported modules. If a module is not used in your base code, it won’t be included in the generated bundle.

So, you must explicitly import all the modules and packages that your dynamic script might need (this also include any part of the standard Python library that your dynamic code would need but your base code doesn’t use)

Simply adding :

import simpleClass

before your import_module line make your example work

Also, using import_module has as consequence that any dynamic code you need to load at runtime must be located next to your executable, inside the app bundle, which is not user friendly. Worse, on macOS, a user can not open the app bundle easily.

It’s possible to load a module using its path via importlib but you need to create a spec and load the module from it.

Ah, interesting! Thank you for that! :slight_smile:

(Of course! I’d forgotten that apparently-unused modules were simply not frozen…)

Hmm… Okay, so let me ask if I may: Are there any downsides to importing absolutely every core module (which is a fair few, I think) in my main program-script…?

[edit]
An alternative occurred to me, and at least in the basic example given above it seems to work:

Instead of importing additional modules within code, one can simply tell setuptools to include them via the “include_modules” option.

Indeed, adding the following to the “options” dictionary of the “setup.py” file in the example-program seems to do the job:

            "include_modules" : {
                TITLE : "simpleClass"
            },

(Noting that “TITLE” is defined further up in that same file.)
[/edit]

Hmm… So, how “next to” does it have to be? Can it be within a sub-directory of a sub-directory of the main program directory?

As to being user-friendly, I’m not sure of where else I might have such modules be located that would be all that much more user-friendly…

(I don’t know how many users know much about their user-folders.)

This, however, is troublesome. (Should I want to support Mac.)

What do Mac games do when they want to allow drop-in DLC…?

I think that I might look this up further…

(Although if you have any recommended resources, please do mention them!)

It only consume a few MB of disk and the startup is a bit slower if you do explicit import in your code, using the hidden import statement of the setup.py is faster as the modules won’t be executed until explicitly imported.

The import will be relative to your main executable, so if you have several subdirectories, you will need to include their name in the fully qualified module name you will import as Python will consider them as packages.

Also, on macOS, in an app bundle, the executable is not in the same directory as the resource files, so you can not use this module import I believe (unless not following the Apple rules)

You could ask you user, or your app could load and store those script or DLC in the user app directory ($HOME/.config/my-app or $HOME/.local/share/my-app on Linux, $HOME/Library/my-app on macOS, UserData on Windows and so on)

Well, the examples of the importlib are a good starting point : importlib — The implementation of import — Python 3.12.4 documentation although a bit dry. In my own app I’m using a variant of third example “Importing a source file directly”, mapping to your example it would be :

import importlib
import os

module_path = "SubClasses/secondary_dynamic.py"
module_name, _ = os.path.splitext(os.path.basename(module_path))
spec = importlib.util.spec_from_file_location(module_name, module_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)

mog = module.Moggy()
mog.describe()

Okay, that’s good, in both cases. :slight_smile:

Fair enough–that’s as expected, I believe!

*sigh* I really don’t like Apple and the hoops that they demand of devs. :/

(I might well prefer to just not support Apple.

And for now I’m only building my demo for Linux and Windows anyway.)

Hum… Until we have the option of a full installer for distributables, that seems likely to be awkward to implement. At the least for the module that comes with the game!

(I suppose that, for the module that comes with the game, I could have the application transfer the relevant files on first being run…)

Ah, thank you for all of that! Based thus far on your example, the process looks fairly simple. :slight_smile:

So, I think that for now–for the purposes of the demo that I’m currently working towards–I’ll stick with having my modules within (a sub-directory of a sub-directory of) the application directory.

But, looking towards actual release, I will likely want an installer anyway, and so might then look to using the module-spec functionality to place my modules in the user’s app-directory.

Thank you so much for your help! :slight_smile:

1 Like