Deferred Pipeline w/ Physically Based Shading

wini · May 25, 2015, 11:51pm

Hi TobiasSpringer,

Amazing work on this one! btw, I ran into a bit of a problem here… I downloaded a previous version and it works ok then recently I downloaded a recent build in your repo and I’m now getting these errors:

Known pipe types:
  wglGraphicsPipe
(all display modules loaded.)
Traceback (most recent call last):
  File "main.py", line 478, in <module>
    app = Main()
  File "main.py", line 81, in __init__
    self.renderPipeline.create()
  File "C:\renderpipeline\Code\RenderingPipeline.py", line 440, in create
    self.lightManager = LightManager(self)
  File "C:\renderpipeline\Code\LightManager.py", line 107, in __init__
    self._initLightCulling()
  File "C:\renderpipeline\Code\LightManager.py", line 197, in _initLightCulling
    self._makeRenderedLightsBuffer()
  File "C:\renderpipeline\Code\LightManager.py", line 211, in _makeRenderedLightsBuffer
    self.renderedLightsBuffer.setupBufferTexture(bufferSize, Texture.TInt, Texture.FR32i, GeomEnums.UHDynamic)
AttributeError: 'panda3d.core.Texture' object has no attribute 'setupBufferTexture'

I tried the default 1.9 build (Win32) in the download sections and also in the link that you provided, still the same errors.

Any ideas? thanks.

tobspr · May 26, 2015, 5:07am

wini:

Hi TobiasSpringer,

Amazing work on this one! btw, I ran into a bit of a problem here… I downloaded a previous version and it works ok then recently I downloaded a recent build in your repo and I’m now getting these errors:
Known pipe types:
  wglGraphicsPipe
(all display modules loaded.)
Traceback (most recent call last):
  File "main.py", line 478, in <module>
    app = Main()
  File "main.py", line 81, in __init__
    self.renderPipeline.create()
  File "C:\renderpipeline\Code\RenderingPipeline.py", line 440, in create
    self.lightManager = LightManager(self)
  File "C:\renderpipeline\Code\LightManager.py", line 107, in __init__
    self._initLightCulling()
  File "C:\renderpipeline\Code\LightManager.py", line 197, in _initLightCulling
    self._makeRenderedLightsBuffer()
  File "C:\renderpipeline\Code\LightManager.py", line 211, in _makeRenderedLightsBuffer
    self.renderedLightsBuffer.setupBufferTexture(bufferSize, Texture.TInt, Texture.FR32i, GeomEnums.UHDynamic)
AttributeError: 'panda3d.core.Texture' object has no attribute 'setupBufferTexture'
I tried the default 1.9 build (Win32) in the download sections and also in the link that you provided, still the same errors.

Any ideas? thanks.

Hey, thanks! rdb added buffer textures recently, you will have to use a development build from https://github.com/panda3d/panda3d. I think the buildbot versions should work, too, you can grab them here: https://www.panda3d.org/download.php?platform=win&version=devel&sdk, so you don’t have to compile it on your own

wini · May 26, 2015, 7:18am

Thanks, but now I’m getting this error…

Traceback (most recent call last):
  File "Tut-Roaming-Ralph.py", line 16, in <module>
    from Code.RenderingPipeline import RenderingPipeline
  File "../../Code\RenderingPipeline.py", line 4, in <module>
    from panda3d.core import PTAVecBase3f, PTAFloat, PTALMatrix4f, PTAInt, SamplerState
ImportError: cannot import name SamplerState

Using this package on the very top: buildbot.panda3d.org/downloads/d … 27-503.exe

tobspr · May 26, 2015, 9:36am

wini:

TobiasSpringer:

Hey, thanks! rdb added buffer textures recently, you will have to use a development build from https://github.com/panda3d/panda3d. I think the buildbot versions should work, too, you can grab them here: https://www.panda3d.org/download.php?platform=win&version=devel&sdk, so you don’t have to compile it on your own

Thanks, but now I’m getting this error…
Traceback (most recent call last):
  File "Tut-Roaming-Ralph.py", line 16, in <module>
    from Code.RenderingPipeline import RenderingPipeline
  File "../../Code\RenderingPipeline.py", line 4, in <module>
    from panda3d.core import PTAVecBase3f, PTAFloat, PTALMatrix4f, PTAInt, SamplerState
ImportError: cannot import name SamplerState
Using this package on the very top: buildbot.panda3d.org/downloads/d … 27-503.exe

Oh, it seems the buildbot versions on that page are outdated I think there are newer buildbot verions somewhere, but right now I can’t find the link - I will ask rdb about it

Edit: seems there are no recent buildbot versions right now. Unfortunately you would have to build panda from source

wini · May 26, 2015, 3:59pm

Thanks, I’m now doing a x64 build but it is taking really slow, is this normal? What compile options do you recommend?

I’m doing the default with –everything --installer

tobspr · May 27, 2015, 5:33am

If you disable eigen, it should be much faster I’m using the following command:

call makepanda\makepanda --use-sse2 --verbose --nothing --use-pandatool --platform-sdk=win71 --threads=6 --optimize=3 --use-bullet --use-gl --use-direct --use-openal --use-zlib --use-png --use-jpeg --use-squish --use-freetype --use-pview --use-nvidiacg --use-python --use-openssl --use-pandaphysics --use-pandaparticlesystem --distributor="tobspr"

You might want to adapt threads to match your cpu cores For reference, a full panda build takes about 43 mins on my core i7 3820

wini · May 27, 2015, 6:33am

Thanks! really helpful tip

wini · May 27, 2015, 7:03am

Hey Tobias, so I have managed to run the main sample? main.py

…But running the other samples I get path errors!

Traceback (most recent call last):
  File "Tut-Roaming-Ralph.py", line 343, in <module>
    w = World()
  File "Tut-Roaming-Ralph.py", line 77, in __init__
    self.renderPipeline.create()
  File "../../Code\RenderingPipeline.py", line 440, in create
    self.lightManager = LightManager(self)
  File "../../Code\LightManager.py", line 123, in __init__
    self._loadIESProfiles()
  File "../../Code\LightManager.py", line 165, in _loadIESProfiles
    self.iesLoader.loadIESProfiles("Data/IESProfiles/")
  File "../../Code\IESLoader.py", line 60, in loadIESProfiles
    files = os.listdir(directory)
WindowsError: [Error 3] The system cannot find the path specified: 'Data/IESProfiles/*.*'

and is it too low for that simple demo for just around 25fps? I have a GTX650 graphics card

BTW, how do I set it in window mode? Please advise with settings, thanks.

tobspr · May 27, 2015, 8:27am

It is 24 ms, so thats arround 41 fps Notice that its a high poly demo scene with arround 1 Million vertices so I think thats actually quite okay

Oh, seems I forgot to fix some pathing there, will have a look at it!

It should be in window mode by default. You can change the settings in Config/configuration.prc. Notice that win-fixed-size is currently required, as resizing the window is not supported yet.

wini · May 27, 2015, 9:37am

Ok gotcha! thanks.

so I was wondering, since this build is a “complete graphics pipeline” as you posted, would you consider adding optimization techniques like Blending dynamic shadows and lightmaps (perhaps a built-in lightmapper ), similar to UDK’s lightmass? It would be really cool to have these methods since it (Baked GI Lightmaps) is been truly tested and utilized by most game engines (Unity, Unreal, etc…)

tobspr · May 27, 2015, 12:00pm

Well, I’m not a fan of baked lightmaps, however, I will maybe think of any dynamic solution

wini · May 27, 2015, 11:54pm

Thanks, this will be really awesome!

croxis · June 21, 2015, 6:35pm

This may be me doing something wrong with the latest version of the pipeline, but it looks like Globals.load() isn’t being called, or at the very least base isn’t being set.

Traceback and log

~/s/ITF git:master ❯❯❯ python2 main.py                                                                                             ⏎ ✱ ◼
:ITF(debug): Loading space drive
:SpaceDrive: Init SpaceDrive
:SpaceDrive(debug): Bitness = 64
:SpaceDrive: Loading Sandbox
Known pipe types:
  glxGraphicsPipe
(all display modules loaded.)
:net(error): Unable to open TCP connection to server 127.0.0.1 on port 5185
:pstats(error): Couldn't connect to PStatServer at localhost:5185
:ITF: Setting up Solar System Body Simulator
:SpaceDrive(debug): Setting up system: <class 'spacedrive.orbit_system.OrbitSystem'>
:SandBox(debug): Adding system <spacedrive.orbit_system.OrbitSystem object at 0x7fb9712e6450>
PipelineSettings      Loading ini-file from pipeline.ini
:SpaceDrive(debug): Cache Directory: /home/croxis/.cache/spacedrive
MountManager          Set base path to '/usr/lib/python2.7/site-packages/SpaceDrive-1.2-py2.7.egg/spacedrive/renderpipeline'
RenderingPipeline     Setting up render pipeline
RenderingPipeline     Checking required Panda3D version ..
MountManager          Setting up virtual filesystem.
:express(warning): Attempt to mount /usr/lib/python2.7/site-packages/SpaceDrive-1.2-py2.7.egg/spacedrive/renderpipeline/Config, not found.
MountManager          Mounting /home/croxis/.cache/spacedrive/Shaders as PipelineTemp/
RenderingPipeline     Setting up globals
GUIManager            Creating GUI ..
AtmosphericScattering  Precomputing ..
AtmosphericScattering  Disabled 1  windows while rendering
:display:gsg:glgsg(error): Shader input source1 is not present.
:display:gsg:glgsg(error): Shader input source2 is not present.
AtmosphericScattering  Finished precomputing, also reenabled windows.
Traceback (most recent call last):
  File "main.py", line 233, in <module>
    spacedrive.init_graphics(debug_mouse=False)
  File "/usr/lib/python2.7/site-packages/SpaceDrive-1.2-py2.7.egg/spacedrive/__init__.py", line 132, in init_graphics
    sandbox.render_pipeline.create()
  File "/usr/lib/python2.7/site-packages/SpaceDrive-1.2-py2.7.egg/spacedrive/renderpipeline/Code/RenderingPipeline.py", line 414, in create
    self.initialRenderPass = InitialRenderPass()
  File "/usr/lib/python2.7/site-packages/SpaceDrive-1.2-py2.7.egg/spacedrive/renderpipeline/Code/RenderPasses/InitialRenderPass.py", line 16, in __init__
    self.scene = Globals.base.render
AttributeError: 'NoneType' object has no attribute 'render'

And my relevant code:

def init_graphics(system=GraphicsSystem,
                  component=celestial_components.CelestialComponent,
                  debug_mouse=False):
    """Sets up multipass rendering. Rendering is done in this order:
    Skybox, Suns, Atmospheres, Celestial bodies, ships"""
    log.warning("TODO: Finish Implement")
    vfs.mount_loop(os.path.join(os.path.dirname(__file__), 'Shader/'),
                   'Shader',
                   VirtualFileSystem.MF_read_only)
    # sandbox.base.camLens.set_far(20000000)
    #sandbox.base.camLens.set_far(2000000)
    sandbox.render_pipeline = RenderingPipeline(sandbox.base)

    sandbox.render_pipeline.loadSettings('pipeline.ini')
    #TODO: Make platform options
    cache_dir = sandbox.appdirs.user_cache_dir('spacedrive', 'croxis')
    log.debug("Cache Directory: " + cache_dir)
    if not os.path.exists(cache_dir):
        os.makedirs(cache_dir)
    if not os.path.exists(os.path.join(cache_dir, 'Shaders')):
        os.makedirs(os.path.join(cache_dir, 'Shaders'))
    sandbox.render_pipeline.getMountManager().setBasePath(
        os.path.join(os.path.dirname(__file__), 'renderpipeline'))
    sandbox.render_pipeline.getMountManager().setWritePath(
        os.path.join(cache_dir, 'Shaders'))
    sandbox.render_pipeline.create()
    init_system(system, component)

    if not debug_mouse:
        sandbox.base.disableMouse()

tobspr · June 22, 2015, 6:46am

It seems the issue is that the Globals is imported differently at two different places, thus not being set I think. It gets included as “From Globals import Globals” in the main-file and “From Code.Globals import Globals” in the render pass file. That issue only happens when the render pipeline is not in the main folder I think. I will have to see what I can do against that, I didn’t test the case where the render pipeline is imported externally, thats why I didn’t mention this bug yet

Did you check out the experimental or the master branch? It might be worth trying the experimental branch, as I included a lot of changes there.

Btw:

Makes me struggle, are you sure the path you set is correct?

croxis · June 22, 2015, 6:48pm

I just haven’t added the Config directory to my setup.py file yet so it isn’t installed with the package.

Yes renderpipeline is in its own directory (a git subtree actually).

You could drop in a blank init.py (or even not blank) and make the Code directory a package.

TonyF · June 25, 2015, 5:52am

Hi All, Im a newbie here so be gentle.

I saw this the other day and it looks absolutely fantastic.

I compiled panda today using the latest git source.

Looks like i am having a few issues with the demos similar to a post above.

All samples failed to find the data directory (Main demo was fine)
Temporarily modded the lightmanager to use a relative path
Main Demo / Sample 1 / Sample 2 / Sample 3.
This demo gives a grayish looking budda

Errors are:
:display:gsg:glgsg(error): Shader input source1 is not present.
:display:gsg:glgsg(error): Shader input source2 is not present.
:display:gsg:glgsg(error): Shader input currentTex is not present.
:display:gsg:glgsg(error): Shader input lastTex is not present.

Sample 2 Fails with:
AttributeError: RenderingPipeline instance has no attribute ‘enableDefaultEarthScattering’

Im not sure where this function is supposed to come from as i cant find it it my editor

Sample 3 Fails with:
ImportError: No module named BetterShader

Im stumped here as i cannot find that module anywhere either

Im i doing something wrong or is it all a bit broken at the moment (Definitely on my end)

Edit:

Decided to try the experimental branch as well. Looks like the pathing issue is fixed however i still run much of anything.

Samples 2 and 3 have the issues as above and the main demo dosnt show much of anything:

:display:gsg:glgsg(error): An error occurred while linking GLSL shader program!
Vertex shader(s) failed to link, fragment shader(s) failed to link.
Vertex link error: INVALID_OPERATION.
ERROR: error(#275) Symbol “materialColor” is defined with 2 different types between two stages
fragment link error: INVALID_OPERATION.
ERROR: error(#275) Symbol “materialColor” is defined with 2 different types between two stages

namic · June 30, 2015, 6:33pm

When running the 04-Carousel sample, i got this:

Traceback (most recent call last):
  File "Tut-Carousel.py", line 37, in <module>
    from Code.BetterShader import BetterShader
ImportError: No module named BetterShader

When running the 01-Simple, i got this:

  File "main.py", line 8, in <module>
    from Code.RenderingPipeline import RenderingPipeline
  File "../../Code/RenderingPipeline.py", line 14, in <module>
    from LightManager import LightManager
  File "../../Code/LightManager.py", line 17, in <module>
    from RenderTarget import RenderTarget
  File "../../Code/RenderTarget.py", line 15, in <module>
    from GUI.BufferViewerGUI import BufferViewerGUI
  File "../../Code/GUI/BufferViewerGUI.py", line 9, in <module>
    from BetterButton import BetterButton
  File "../../Code/GUI/BetterButton.py", line 2, in <module>
    from direct.gui.DirectButton import DirectButton
  File "/usr/share/panda3d/direct/gui/DirectButton.py", line 7, in <module>
    from DirectFrame import *
  File "/usr/share/panda3d/direct/gui/DirectFrame.py", line 7, in <module>
    from DirectGuiBase import *
  File "/usr/share/panda3d/direct/gui/DirectGuiBase.py", line 656, in <module>
    if config.GetBool('record-gui-creation-stack', __debug__):
NameError: name 'config' is not defined

I’m running on a latest Panda3D clone from master.

Also, when running main.py, on the root of the project, i got this:

:display:gsg:glgsg(error): Shader input currentTex is not present.
:display:gsg:glgsg(error): Shader input lastTex is not present.

Lot’s of them. Any ideas?

tobspr · June 30, 2015, 7:24pm

I just updated the samples, can you try if it works for you now?

namic · June 30, 2015, 7:28pm

They all give me this now:

Traceback (most recent call last):
  File "main.py", line 8, in <module>
    from Code.RenderingPipeline import RenderingPipeline
  File "../../Code/RenderingPipeline.py", line 14, in <module>
    from LightManager import LightManager
  File "../../Code/LightManager.py", line 17, in <module>
    from RenderTarget import RenderTarget
  File "../../Code/RenderTarget.py", line 15, in <module>
    from GUI.BufferViewerGUI import BufferViewerGUI
  File "../../Code/GUI/BufferViewerGUI.py", line 9, in <module>
    from BetterButton import BetterButton
  File "../../Code/GUI/BetterButton.py", line 2, in <module>
    from direct.gui.DirectButton import DirectButton
  File "/usr/share/panda3d/direct/gui/DirectButton.py", line 7, in <module>
    from DirectFrame import *
  File "/usr/share/panda3d/direct/gui/DirectFrame.py", line 7, in <module>
    from DirectGuiBase import *
  File "/usr/share/panda3d/direct/gui/DirectGuiBase.py", line 656, in <module>
    if config.GetBool('record-gui-creation-stack', __debug__):
NameError: name 'config' is not defined

TonyF · July 1, 2015, 5:09am

I tried the updated samples and that has made them work for me which is great.

One problem i have as well is that when the G-Illum is activated it makes all textures black.

Not sure what is happening there