Reverse-z : Non-standard Depth/zBuffer?


#1

I am interested in using a non-standard depth buffer to deal with a large dynamic range in distance in a single scene. The best solution out there seems to be using reversed-z buffers with floating point depth values. It is great because there is no slowdown because it can still use the hardware depth pipeline (unlike for things like log or linear depths).

The main trick is to put zNear= the far distance and zFar = the nearer distance.

e.g. self.cam.node().getLens().setNearFar(5000,0.1)

OpenGL is fine with this – does Panda3d object? Even if it does I assume a workaround would not be too much work but there are additional steps as well.

There is a really good overview here:

https://nlguillemot.wordpress.com/2016/12/07/reversed-z-in-opengl/

The author lists 5 key steps to make it work:
1) Depth buffer must use 0 to 1 not -1 to+1 (glClipControl does this)
2) Float 32 bit z-buffer – takes advantage of float/exp representation to preserve resolution near 0
3) Depth buffer Clear to 0 (far)
4) Flip depth comparison to greater than
5) Special perspective matrix (because the scene is mirror-reversed otherwise)

There are a few interesting articles showing just how effective this is in practice. Quite simply its amazing and takes you from about 4 decs depth precision to over 7 decades (i.e. better than 10 million dynamic range).
e.g.
https://developer.nvidia.com/content/depth-precision-visualized
http://dev.theomader.com/depth-precision/

Are there any road blocks to doing this in Panda? For example, I didn’t see how glClipControl was exposed to the user in Panda. I have also struggled a bit convincing the renderer to use a 32 bit float (not fixed) depth buffer. The last awkward part seemed to be the need to generate your own perspective matrix and thus forgo the nice automatically made matrix features of Panda.

Thanks.


#2

I have heard of this technique, and am happy to consider how it can best be implemented in Panda3D. It does unfortunately rely on OpenGL functionality that has been added in 4.5, so cannot be used if compatibility with older hardware is your goal.

  1. Adding a config variable to change the OpenGL depth range convention using glClipControl should be fairly trivial, and am happy to check in such a change shortly. In the meantime, you could hack this using PyOpenGL to make the call.

  2. Creating a floating point depth buffer is supported in Panda3D, but I don’t think this works for the default window framebuffer, merely because it doesn’t seem like GLX/WGL gives us a way to explicitly request float depth buffer. You can however render the scene to a GraphicsBuffer, which you create by passing a FrameBufferProperties on which you called fbp.setFloatDepth(True). Let me know if you need help with this.

  3. buffer.setClearDepth(0)

  4. render.setAttrib(DepthTestAttrib.make(DepthTestAttrib.M_greater))

  5. Inverting the near/far clip should work in theory (we should fix it if it doesn’t), but I don’t think that’s all there is to it, since by default it maps depth to the -1…1 depth range. The DirectX renderer already needs to adjust the projection matrix to take this into account. I think we will also need whatever variable will control glClipControl to also make this adjustment to the projection matrix.

    You could in the meantime use a MatrixLens to supply your own projection matrix.

I think I will also look into adding a config variable that automatically reverses the projection matrix, depth func and depth clear if this functionality is available. If you think we need greater control than a global config setting, perhaps we will need to add a special “depth mode” setting to FrameBufferProperties that controls this.

FWIW, passing float("inf") as far distance is explicitly allowed in Panda3D.


#3

Thanks for having a look.

That’s a really good point regarding older hardware. OpenGL 4.5 is 2014 or so so hopefully becoming well supported in drivers now. In particular, DirectX uses 0 to 1 for the depth buffer so hardware has been supporting 0 to 1 for a long time. They also mention an extension which may help for older hardware: https://www.khronos.org/registry/OpenGL/extensions/ARB/ARB_clip_control.txt


#4

That extension was introduced when OpenGL 4.5 was released; it is standard practice to release an extension version of all functionality alongside every core release. Even if a device can support this extension without supporting the rest of OpenGL 4.5, it would still need a recent enough driver.

Also note that not all hardware is capable of supporting OpenGL 4.5. In particular, pre-DX11 hardware is generally unlikely to support it. But they may still be able to support the GL_ARB_clip_control functionality.

According to this report, only 21% of submissions support GL_ARB_clip_control, but this may vary drastically depending on audience. It does appear that there exist pre-OpenGL 4.x cards that support this extension nonetheless, so it will certainly still help to check for the extension.

There is however a vendor-specific extension (GL_NV_depth_buffer_float) that lets us produce the same result by allowing us to call glDepthRangedNV(-1, 1), which cancels out the default OpenGL depth remapping. It at least covers NVIDIA GPUs from GeForce 8 upwards, which is pretty much all NVIDIA GPUs one needs to care about at this point. It also appears that AMD has added support for it to their drivers, as confirmed by the above report. No Intel hardware, though.


#5

Okay, it’s in:

You can now set gl-depth-zero-to-one true in Config.prc, which calls glClipControl to activate GL_ZERO_TO_ONE, and also automatically adjusts the projection matrix to project to the [0,1] range instead of the default [-1,1].

I had also implemented the fallback implementation using GL_NV_depth_buffer_float, but I have commented it out for now because there’s one more piece missing: it requires adding an additional clip plane on the near (or far, in case of reverse rendering) side, as not doing so would yield negative depth values. I don’t have time for that today, but I may get around to it soon.

I have also verified that a reverse near/far works in Panda3D. However, I do not believe setting the near distance to infinity works yet. I will have to look into that as well.

With this setting, you still need to change the depth clear value, the depth test attribute, flip the near/far range and use an FBO with float depth, but this is all possible via the Panda3D API. Let me know if you run into any trouble or need any help.


#6

Thanks a lot – I’ll check it out.


#7

Thanks again! I tested the new code. Reverse-z depth buffers work really well in Panda. I made a quick demo that gets a factor of 100 more range in the depth buffer without z-fighting (based on the shader-terrain sample) with a water layer than z-fights with the terrain depending on the depth buffer method.

https://drive.google.com/open?id=1aEwXX-1lYZtUWOP871bw-OrmqYokAu9_

Two versions to try:
main_zfight.py     standard depth buffers with Near/Far 0.01 to 5000 (has z-fighting issues)
main_reversez.py    reverse z with Near/Far 0.0001 to 5000 with no problems

Based on tests in the other links above it should be able to do even better in different set-ups. Note that my quick hack approach to creating a float depth buffer via a GraphicsBuffer is extremely ugly – I added a modified renderSceneInto and createBuffer. I am sure there is a better/neater way to do this. If you can give me some pointers on a more direct way to use GraphicsBuffer I can make a much cleaner demo.

Also – it works really well but doesn’t fail very gracefully. That is, if I try to make zNear even smaller (say 0.00001) the whole scene just disappears. I guess some kind of round-off could be zeroing all the depths at this point as its a pretty impressive ratio.


#8

I tried to check the code, but I get the error message:

File "main.py", line 81, in __init__
skybox.set_shader (skybox_shader)
TypeError: NodePath.set_shader () argument 1 must be Shader, not NoneType

Full console output:

D:\Panda3d\reversez>D:\Panda3d\Panda3D-1.10.0-x64\python\python.exe main.py
Known pipe types:
  wglGraphicsPipe
(all display modules loaded.)
:display:gsg:glgsg(debug): HAS EXT WGL_ARB_pixel_format 1
:display:gsg:glgsg(debug): HAS EXT WGL_ARB_multisample 1
:display:gsg:glgsg(debug): HAS EXT WGL_ARB_create_context 1
:display:gsg:glgsg(debug): GL_VENDOR = NVIDIA Corporation
:display:gsg:glgsg(debug): GL_RENDERER = GeForce GTX 1050/PCIe/SSE2
:display:gsg:glgsg(debug): GL_VERSION = 4.6.0 NVIDIA 398.36
:display:gsg:glgsg(debug): GL_VERSION decoded to: 4.6
:display:gsg:glgsg(debug): GL_SHADING_LANGUAGE_VERSION = 4.60 NVIDIA
:display:gsg:glgsg(debug): Detected GLSL version: 4.60
:display:gsg:glgsg(debug): HAS EXT GL_ARB_compatibility 1
:display:gsg:glgsg(debug): Using compatibility profile
:display:gsg:glgsg(debug): GL Extensions:
  GL_AMD_multi_draw_indirect             GL_AMD_seamless_cubemap_per_texture
  GL_AMD_vertex_shader_layer             GL_AMD_vertex_shader_viewport_index
  GL_ARB_ES2_compatibility               GL_ARB_ES3_1_compatibility
  GL_ARB_ES3_2_compatibility             GL_ARB_ES3_compatibility
  GL_ARB_arrays_of_arrays                GL_ARB_base_instance
  GL_ARB_bindless_texture                GL_ARB_blend_func_extended
  GL_ARB_buffer_storage                  GL_ARB_clear_buffer_object
  GL_ARB_clear_texture                   GL_ARB_clip_control
  GL_ARB_color_buffer_float              GL_ARB_compatibility
  GL_ARB_compressed_texture_pixel_storage
  GL_ARB_compute_shader                  GL_ARB_compute_variable_group_size
  GL_ARB_conditional_render_inverted     GL_ARB_conservative_depth
  GL_ARB_copy_buffer                     GL_ARB_copy_image
  GL_ARB_cull_distance                   GL_ARB_debug_output
  GL_ARB_depth_buffer_float              GL_ARB_depth_clamp
  GL_ARB_depth_texture                   GL_ARB_derivative_control
  GL_ARB_direct_state_access             GL_ARB_draw_buffers
  GL_ARB_draw_buffers_blend              GL_ARB_draw_elements_base_vertex
  GL_ARB_draw_indirect                   GL_ARB_draw_instanced
  GL_ARB_enhanced_layouts                GL_ARB_explicit_attrib_location
  GL_ARB_explicit_uniform_location       GL_ARB_fragment_coord_conventions
  GL_ARB_fragment_layer_viewport         GL_ARB_fragment_program
  GL_ARB_fragment_program_shadow         GL_ARB_fragment_shader
  GL_ARB_fragment_shader_interlock       GL_ARB_framebuffer_no_attachments
  GL_ARB_framebuffer_object              GL_ARB_framebuffer_sRGB
  GL_ARB_geometry_shader4                GL_ARB_get_program_binary
  GL_ARB_get_texture_sub_image           GL_ARB_gl_spirv
  GL_ARB_gpu_shader5                     GL_ARB_gpu_shader_fp64
  GL_ARB_gpu_shader_int64                GL_ARB_half_float_pixel
  GL_ARB_half_float_vertex               GL_ARB_imaging
  GL_ARB_indirect_parameters             GL_ARB_instanced_arrays
  GL_ARB_internalformat_query            GL_ARB_internalformat_query2
  GL_ARB_invalidate_subdata              GL_ARB_map_buffer_alignment
  GL_ARB_map_buffer_range                GL_ARB_multi_bind
  GL_ARB_multi_draw_indirect             GL_ARB_multisample
  GL_ARB_multitexture                    GL_ARB_occlusion_query
  GL_ARB_occlusion_query2                GL_ARB_parallel_shader_compile
  GL_ARB_pipeline_statistics_query       GL_ARB_pixel_buffer_object
  GL_ARB_point_parameters                GL_ARB_point_sprite
  GL_ARB_polygon_offset_clamp            GL_ARB_post_depth_coverage
  GL_ARB_program_interface_query         GL_ARB_provoking_vertex
  GL_ARB_query_buffer_object             GL_ARB_robust_buffer_access_behavior
  GL_ARB_robustness                      GL_ARB_sample_locations
  GL_ARB_sample_shading                  GL_ARB_sampler_objects
  GL_ARB_seamless_cube_map               GL_ARB_seamless_cubemap_per_texture
  GL_ARB_separate_shader_objects         GL_ARB_shader_atomic_counter_ops
  GL_ARB_shader_atomic_counters          GL_ARB_shader_ballot
  GL_ARB_shader_bit_encoding             GL_ARB_shader_clock
  GL_ARB_shader_draw_parameters          GL_ARB_shader_group_vote
  GL_ARB_shader_image_load_store         GL_ARB_shader_image_size
  GL_ARB_shader_objects                  GL_ARB_shader_precision
  GL_ARB_shader_storage_buffer_object    GL_ARB_shader_subroutine
  GL_ARB_shader_texture_image_samples    GL_ARB_shader_texture_lod
  GL_ARB_shader_viewport_layer_array     GL_ARB_shading_language_100
  GL_ARB_shading_language_420pack        GL_ARB_shading_language_include
  GL_ARB_shading_language_packing        GL_ARB_shadow
  GL_ARB_sparse_buffer                   GL_ARB_sparse_texture
  GL_ARB_sparse_texture2                 GL_ARB_sparse_texture_clamp
  GL_ARB_spirv_extensions                GL_ARB_stencil_texturing
  GL_ARB_sync                            GL_ARB_tessellation_shader
  GL_ARB_texture_barrier                 GL_ARB_texture_border_clamp
  GL_ARB_texture_buffer_object           GL_ARB_texture_buffer_object_rgb32
  GL_ARB_texture_buffer_range            GL_ARB_texture_compression
  GL_ARB_texture_compression_bptc        GL_ARB_texture_compression_rgtc
  GL_ARB_texture_cube_map                GL_ARB_texture_cube_map_array
  GL_ARB_texture_env_add                 GL_ARB_texture_env_combine
  GL_ARB_texture_env_crossbar            GL_ARB_texture_env_dot3
  GL_ARB_texture_filter_anisotropic      GL_ARB_texture_filter_minmax
  GL_ARB_texture_float                   GL_ARB_texture_gather
  GL_ARB_texture_mirror_clamp_to_edge    GL_ARB_texture_mirrored_repeat
  GL_ARB_texture_multisample             GL_ARB_texture_non_power_of_two
  GL_ARB_texture_query_levels            GL_ARB_texture_query_lod
  GL_ARB_texture_rectangle               GL_ARB_texture_rg
  GL_ARB_texture_rgb10_a2ui              GL_ARB_texture_stencil8
  GL_ARB_texture_storage                 GL_ARB_texture_storage_multisample
  GL_ARB_texture_swizzle                 GL_ARB_texture_view
  GL_ARB_timer_query                     GL_ARB_transform_feedback2
  GL_ARB_transform_feedback3             GL_ARB_transform_feedback_instanced
  GL_ARB_transform_feedback_overflow_query
  GL_ARB_transpose_matrix                GL_ARB_uniform_buffer_object
  GL_ARB_vertex_array_bgra               GL_ARB_vertex_array_object
  GL_ARB_vertex_attrib_64bit             GL_ARB_vertex_attrib_binding
  GL_ARB_vertex_buffer_object            GL_ARB_vertex_program
  GL_ARB_vertex_shader                   GL_ARB_vertex_type_10f_11f_11f_rev
  GL_ARB_vertex_type_2_10_10_10_rev      GL_ARB_viewport_array
  GL_ARB_window_pos                      GL_ATI_draw_buffers
  GL_ATI_texture_float                   GL_ATI_texture_mirror_once
  GL_EXTX_framebuffer_mixed_formats      GL_EXT_Cg_shader
  GL_EXT_abgr                            GL_EXT_bgra
  GL_EXT_bindable_uniform                GL_EXT_blend_color
  GL_EXT_blend_equation_separate         GL_EXT_blend_func_separate
  GL_EXT_blend_minmax                    GL_EXT_blend_subtract
  GL_EXT_compiled_vertex_array           GL_EXT_depth_bounds_test
  GL_EXT_direct_state_access             GL_EXT_draw_buffers2
  GL_EXT_draw_instanced                  GL_EXT_draw_range_elements
  GL_EXT_fog_coord                       GL_EXT_framebuffer_blit
  GL_EXT_framebuffer_multisample         GL_EXT_framebuffer_multisample_blit_scaled
  GL_EXT_framebuffer_object              GL_EXT_framebuffer_sRGB
  GL_EXT_geometry_shader4                GL_EXT_gpu_program_parameters
  GL_EXT_gpu_shader4                     GL_EXT_import_sync_object
  GL_EXT_memory_object                   GL_EXT_memory_object_win32
  GL_EXT_multi_draw_arrays               GL_EXT_packed_depth_stencil
  GL_EXT_packed_float                    GL_EXT_packed_pixels
  GL_EXT_pixel_buffer_object             GL_EXT_point_parameters
  GL_EXT_polygon_offset_clamp            GL_EXT_post_depth_coverage
  GL_EXT_provoking_vertex                GL_EXT_raster_multisample
  GL_EXT_rescale_normal                  GL_EXT_secondary_color
  GL_EXT_semaphore                       GL_EXT_semaphore_win32
  GL_EXT_separate_shader_objects         GL_EXT_separate_specular_color
  GL_EXT_shader_image_load_formatted     GL_EXT_shader_image_load_store
  GL_EXT_shader_integer_mix              GL_EXT_shadow_funcs
  GL_EXT_sparse_texture2                 GL_EXT_stencil_two_side
  GL_EXT_stencil_wrap                    GL_EXT_texture3D
  GL_EXT_texture_array                   GL_EXT_texture_buffer_object
  GL_EXT_texture_compression_dxt1        GL_EXT_texture_compression_latc
  GL_EXT_texture_compression_rgtc        GL_EXT_texture_compression_s3tc
  GL_EXT_texture_cube_map                GL_EXT_texture_edge_clamp
  GL_EXT_texture_env_add                 GL_EXT_texture_env_combine
  GL_EXT_texture_env_dot3                GL_EXT_texture_filter_anisotropic
  GL_EXT_texture_filter_minmax           GL_EXT_texture_integer
  GL_EXT_texture_lod                     GL_EXT_texture_lod_bias
  GL_EXT_texture_mirror_clamp            GL_EXT_texture_object
  GL_EXT_texture_sRGB                    GL_EXT_texture_sRGB_decode
  GL_EXT_texture_shared_exponent         GL_EXT_texture_storage
  GL_EXT_texture_swizzle                 GL_EXT_timer_query
  GL_EXT_transform_feedback2             GL_EXT_vertex_array
  GL_EXT_vertex_array_bgra               GL_EXT_vertex_attrib_64bit
  GL_EXT_win32_keyed_mutex               GL_EXT_window_rectangles
  GL_IBM_rasterpos_clip                  GL_IBM_texture_mirrored_repeat
  GL_KHR_blend_equation_advanced         GL_KHR_blend_equation_advanced_coherent
  GL_KHR_context_flush_control           GL_KHR_debug
  GL_KHR_no_error                        GL_KHR_parallel_shader_compile
  GL_KHR_robust_buffer_access_behavior   GL_KHR_robustness
  GL_KTX_buffer_region                   GL_NVX_blend_equation_advanced_multi_draw_buffers
  GL_NVX_conditional_render              GL_NVX_gpu_memory_info
  GL_NVX_multigpu_info                   GL_NVX_nvenc_interop
  GL_NV_ES1_1_compatibility              GL_NV_ES3_1_compatibility
  GL_NV_alpha_to_coverage_dither_control GL_NV_bindless_multi_draw_indirect
  GL_NV_bindless_multi_draw_indirect_count
  GL_NV_bindless_texture                 GL_NV_blend_equation_advanced
  GL_NV_blend_equation_advanced_coherent GL_NV_blend_minmax_factor
  GL_NV_blend_square                     GL_NV_clip_space_w_scaling
  GL_NV_command_list                     GL_NV_compute_program5
  GL_NV_conditional_render               GL_NV_conservative_raster
  GL_NV_conservative_raster_dilate       GL_NV_conservative_raster_pre_snap_triangles
  GL_NV_copy_depth_to_color              GL_NV_copy_image
  GL_NV_depth_buffer_float               GL_NV_depth_clamp
  GL_NV_draw_texture                     GL_NV_draw_vulkan_image
  GL_NV_explicit_multisample             GL_NV_feature_query
  GL_NV_fence                            GL_NV_fill_rectangle
  GL_NV_float_buffer                     GL_NV_fog_distance
  GL_NV_fragment_coverage_to_color       GL_NV_fragment_program
  GL_NV_fragment_program2                GL_NV_fragment_program_option
  GL_NV_fragment_shader_interlock        GL_NV_framebuffer_mixed_samples
  GL_NV_framebuffer_multisample_coverage GL_NV_geometry_shader4
  GL_NV_geometry_shader_passthrough      GL_NV_gpu_program4
  GL_NV_gpu_program4_1                   GL_NV_gpu_program5
  GL_NV_gpu_program5_mem_extended        GL_NV_gpu_program_fp64
  GL_NV_gpu_shader5                      GL_NV_half_float
  GL_NV_internalformat_sample_query      GL_NV_light_max_exponent
  GL_NV_multisample_coverage             GL_NV_multisample_filter_hint
  GL_NV_occlusion_query                  GL_NV_packed_depth_stencil
  GL_NV_parameter_buffer_object          GL_NV_parameter_buffer_object2
  GL_NV_path_rendering                   GL_NV_path_rendering_shared_edge
  GL_NV_pixel_data_range                 GL_NV_point_sprite
  GL_NV_primitive_restart                GL_NV_query_resource
  GL_NV_query_resource_tag               GL_NV_register_combiners
  GL_NV_register_combiners2              GL_NV_sample_locations
  GL_NV_sample_mask_override_coverage    GL_NV_shader_atomic_counters
  GL_NV_shader_atomic_float              GL_NV_shader_atomic_float64
  GL_NV_shader_atomic_fp16_vector        GL_NV_shader_atomic_int64
  GL_NV_shader_buffer_load               GL_NV_shader_storage_buffer_object
  GL_NV_shader_thread_group              GL_NV_shader_thread_shuffle
  GL_NV_stereo_view_rendering            GL_NV_texgen_reflection
  GL_NV_texture_barrier                  GL_NV_texture_compression_vtc
  GL_NV_texture_env_combine4             GL_NV_texture_multisample
  GL_NV_texture_rectangle                GL_NV_texture_rectangle_compressed
  GL_NV_texture_shader                   GL_NV_texture_shader2
  GL_NV_texture_shader3                  GL_NV_transform_feedback
  GL_NV_transform_feedback2              GL_NV_uniform_buffer_unified_memory
  GL_NV_vertex_array_range               GL_NV_vertex_array_range2
  GL_NV_vertex_attrib_integer_64bit      GL_NV_vertex_buffer_unified_memory
  GL_NV_vertex_program                   GL_NV_vertex_program1_1
  GL_NV_vertex_program2                  GL_NV_vertex_program2_option
  GL_NV_vertex_program3                  GL_NV_viewport_array2
  GL_NV_viewport_swizzle                 GL_S3_s3tc
  GL_SGIS_generate_mipmap                GL_SGIS_texture_lod
  GL_SGIX_depth_texture                  GL_SGIX_shadow
  GL_SUN_slice_accum                     GL_WIN_swap_hint
  WGL_ARB_buffer_region                  WGL_ARB_context_flush_control
  WGL_ARB_create_context                 WGL_ARB_create_context_no_error
  WGL_ARB_create_context_profile         WGL_ARB_create_context_robustness
  WGL_ARB_extensions_string              WGL_ARB_make_current_read
  WGL_ARB_multisample                    WGL_ARB_pbuffer
  WGL_ARB_pixel_format                   WGL_ARB_pixel_format_float
  WGL_ARB_render_texture                 WGL_ATI_pixel_format_float
  WGL_EXT_colorspace                     WGL_EXT_create_context_es2_profile
  WGL_EXT_create_context_es_profile      WGL_EXT_extensions_string
  WGL_EXT_framebuffer_sRGB               WGL_EXT_pixel_format_packed_float
  WGL_EXT_swap_control                   WGL_EXT_swap_control_tear
  WGL_NVX_DX_interop                     WGL_NV_DX_interop
  WGL_NV_DX_interop2                     WGL_NV_copy_image
  WGL_NV_delay_before_swap               WGL_NV_float_buffer
  WGL_NV_multisample_coverage            WGL_NV_render_depth_texture
  WGL_NV_render_texture_rectangle
:display:gsg:glgsg(debug): gl-debug supported, but NOT enabled.
:display:gsg:glgsg(debug): HAS EXT GL_ARB_vertex_program 1
:display:gsg:glgsg(debug): HAS EXT GL_ARB_fragment_program 1
:display:gsg:glgsg(debug): HAS EXT GL_NV_gpu_program5 1
:display:gsg:glgsg(debug): HAS EXT GL_NV_framebuffer_multisample_coverage 1
:display:gsg:glgsg(debug): Occlusion query counter provides 32 bits.
:display:gsg:glgsg(debug): HAS EXT GL_EXT_texture_mirror_clamp 1
:display:gsg:glgsg(debug): max texture dimension = 32768, max 3d texture = 16384, max 2d texture array = 2048, max cube map = 32768
:display:gsg:glgsg(debug): max_elements_vertices = 1048576, max_elements_indices = 1048576
:display:gsg:glgsg(debug): vertex buffer objects are supported.
:display:gsg:glgsg(debug): Supported compressed texture formats:
  GL_COMPRESSED_RGB_S3TC_DXT1_EXT
  GL_COMPRESSED_RGBA_S3TC_DXT3_EXT
  GL_COMPRESSED_RGBA_S3TC_DXT5_EXT
  GL_PALETTE4_RGB8_OES
  GL_PALETTE4_RGBA8_OES
  GL_PALETTE4_R5_G6_B5_OES
  GL_PALETTE4_RGBA4_OES
  GL_PALETTE4_RGB5_A1_OES
  GL_PALETTE8_RGB8_OES
  GL_PALETTE8_RGBA8_OES
  GL_PALETTE8_R5_G6_B5_OES
  GL_PALETTE8_RGBA4_OES
  GL_PALETTE8_RGB5_A1_OES
  GL_COMPRESSED_RGB8_ETC2
  GL_COMPRESSED_SRGB8_ETC2
  GL_COMPRESSED_RGB8_PUNCHTHROUGH_ALPHA1_ETC2
  GL_COMPRESSED_SRGB8_PUNCHTHROUGH_ALPHA1_ETC2
  GL_COMPRESSED_RGBA8_ETC2_EAC
  GL_COMPRESSED_SRGB8_ALPHA8_ETC2_EAC
  GL_COMPRESSED_R11_EAC
  GL_COMPRESSED_SIGNED_R11_EAC
  GL_COMPRESSED_RG11_EAC
  GL_COMPRESSED_SIGNED_RG11_EAC
  GL_COMPRESSED_RGBA_ASTC_4x4_KHR
  GL_COMPRESSED_RGBA_ASTC_5x4_KHR
  GL_COMPRESSED_RGBA_ASTC_5x5_KHR
  GL_COMPRESSED_RGBA_ASTC_6x5_KHR
  GL_COMPRESSED_RGBA_ASTC_6x6_KHR
  GL_COMPRESSED_RGBA_ASTC_8x5_KHR
  GL_COMPRESSED_RGBA_ASTC_8x6_KHR
  GL_COMPRESSED_RGBA_ASTC_8x8_KHR
  GL_COMPRESSED_RGBA_ASTC_10x5_KHR
  GL_COMPRESSED_RGBA_ASTC_10x6_KHR
  GL_COMPRESSED_RGBA_ASTC_10x8_KHR
  GL_COMPRESSED_RGBA_ASTC_10x10_KHR
  GL_COMPRESSED_RGBA_ASTC_12x10_KHR
  GL_COMPRESSED_RGBA_ASTC_12x12_KHR
  GL_COMPRESSED_SRGB8_ALPHA8_ASTC_4x4_KHR
  GL_COMPRESSED_SRGB8_ALPHA8_ASTC_5x4_KHR
  GL_COMPRESSED_SRGB8_ALPHA8_ASTC_5x5_KHR
  GL_COMPRESSED_SRGB8_ALPHA8_ASTC_6x5_KHR
  GL_COMPRESSED_SRGB8_ALPHA8_ASTC_6x6_KHR
  GL_COMPRESSED_SRGB8_ALPHA8_ASTC_8x5_KHR
  GL_COMPRESSED_SRGB8_ALPHA8_ASTC_8x6_KHR
  GL_COMPRESSED_SRGB8_ALPHA8_ASTC_8x8_KHR
  GL_COMPRESSED_SRGB8_ALPHA8_ASTC_10x5_KHR
  GL_COMPRESSED_SRGB8_ALPHA8_ASTC_10x6_KHR
  GL_COMPRESSED_SRGB8_ALPHA8_ASTC_10x8_KHR
  GL_COMPRESSED_SRGB8_ALPHA8_ASTC_10x10_KHR
  GL_COMPRESSED_SRGB8_ALPHA8_ASTC_12x10_KHR
  GL_COMPRESSED_SRGB8_ALPHA8_ASTC_12x12_KHR
:display:gsg:glgsg(debug): HAS EXT GL_EXT_texture_filter_anisotropic 1
:display:gsg:glgsg(debug): HAS EXT GL_ARB_bindless_texture 1
:display:gsg:glgsg(debug): HAS EXT GL_EXT_stencil_two_side 1
:display:gsg:glgsg(debug): max lights = 8
:display:gsg:glgsg(debug): max clip planes = 8
:display:gsg:glgsg(debug): max texture stages = 4
:display:gsg:glgsg(debug): Supported shader binary formats:
:display:gsg:glgsg(debug):   0x8E21
:display:gsg:glgsg(debug): Supported Cg profiles:
:display:gsg:glgsg(debug):   vp20
:display:gsg:glgsg(debug):   fp20
:display:gsg:glgsg(debug):   vp30
:display:gsg:glgsg(debug):   fp30
:display:gsg:glgsg(debug):   arbvp1
:display:gsg:glgsg(debug):   fp40
:display:gsg:glgsg(debug):   arbfp1
:display:gsg:glgsg(debug):   vp40
:display:gsg:glgsg(debug):   glslv
:display:gsg:glgsg(debug):   glslf
:display:gsg:glgsg(debug):   glslg
:display:gsg:glgsg(debug):   gp4fp
:display:gsg:glgsg(debug):   gp4vp
:display:gsg:glgsg(debug):   gp4gp
:display:gsg:glgsg(debug):   gp5fp
:display:gsg:glgsg(debug):   gp5vp
:display:gsg:glgsg(debug):   gp5gp
:display:gsg:glgsg(debug):   gp5tcp
:display:gsg:glgsg(debug):   gp5tep
:display:gsg:glgsg(debug): Cg GLSL version = CG_GL_GLSL_120
:display:gsg:glgsg(debug): Cg latest vertex profile = gp5vp
:display:gsg:glgsg(debug): Cg latest fragment profile = gp5fp
:display:gsg:glgsg(debug): Cg latest geometry profile = gp5gp
:display:gsg:glgsg(debug): basic-shaders-only #f
:display:gsg:glgsg(debug): Cg active vertex profile = gp5vp
:display:gsg:glgsg(debug): Cg active fragment profile = gp5fp
:display:gsg:glgsg(debug): Cg active geometry profile = gp5gp
:display:gsg:glgsg(debug): shader model = 5.0
:display:gsg:glgsg(debug): HAS EXT WGL_EXT_swap_control 1
:display:gsg:glgsg(debug): HAS EXT WGL_ARB_pbuffer 1
:display:gsg:glgsg(debug): HAS EXT WGL_ARB_pixel_format 1
:display:gsg:glgsg(debug): HAS EXT WGL_ARB_multisample 1
:display:gsg:glgsg(debug): HAS EXT WGL_ARB_render_texture 1


:pnmimage:png(warning): iCCP: known incorrect sRGB profile
:shader(error): Could not find shader file: skybox.vert.glsl
Traceback (most recent call last):
  File "main.py", line 83, in <module>
    ShaderTerrainDemo().run()
  File "main.py", line 81, in __init__
    skybox.set_shader(skybox_shader)
TypeError: NodePath.set_shader() argument 1 must be Shader, not NoneType
:display:gsg:glgsg(debug): GLGraphicsStateGuardian 0000019AA67A4ED0 destructing

#9

Oh, yes, there were no shader files. The problem is not a problem :grinning:


#10

My apologies – main.py is left over from the shader-terrain sample and should not be there.
The two relevant files are main_zfight.py and main_reversez.py


#11

I did some more experimenting. You can make the first argument pretty much as large as you want without problems, e.g. setNearFar(5e30,0.0001) still works fine.
This makes sense as its just like letting the standard depth buffer zFar-> Infinity which is allowed as rdb mentioned.

The real limitation is the second argument (the near cut off for reversez).
For example, problems occurs for setNearFar(anything, 0.00001) on this demo.
Given that far doesn’t matter, the important distance is from camera to the geometry in the scene which is about 1000 in the demo so this means the ratio when the problems start is about 1000/0.00001=1e9.

As I mentioned above, it isn’t failing gracefully. The scene just disappears. However, I don’t think it should fail and I can fiddle with the view a little (rotate it or scroll it) and get the scene to come back and there are no zfighting issues – the depth tests are still great. If you keep fiddling for some orientations it disappears again and occasionally just parts of it disappear.
To put it another way, if the depth buffer ranges from 0->1 and the first argument is really large then
the mapping is just z_NDC = 0.00001/z_eye which maps everything nicely into 0 to 1.
There’s no round-off issue for this expression if z_eye and z_NDC are 32 bit floats. This makes me think it isn’t a depth thing at all.

It make me wonder if the clipping or culling is acting up instead? Perhaps its not quite right when you reverse near and far. I know shader terrain does the cull itself every frame. It gets matrices from panda to do this. It also has its own shaders (though they don’t mess with the depth). However the water quad is just a standard piece of panda geometry with no special shader so it is just doing whatever panda would normally do.

It may be that a round off issue occurs in the culling or clipping. However the expressions there are also pretty straight forward (at least in eye -> NDC) so it isn’t clear how that would happen. It is possible that the overall MVP matrix has roundoff issues.

I haven’t actually found a near/far combination where reversez has zfighting issues on this test case – the clip/cull issue seems to occur first and just remove the scene from view.

Note that I am saying this just in the interests of making sure nothing was overlooked in case someone really needed to push the depth range to crazy extremes. Being able to use a near cut-off of 0.0001 on a scene extending roughly to 1000 is already perfect for what I am doing in practice.


#12

This is really cool. It is nice to see how effective it is.

I’m currently working on changing the culling in Panda to support infinite frusta (both near and far). The reason why 5e30 works is actually because Panda is limiting the depth based on the lens-far-limit setting, not because it actually supports a bounding volume that goes to infinity.

As for why a tiny near distance fails, I will also look into this; it might have something to do with the fact that Panda generally considers floats smaller than 1\times 10^{-6} to be zero.

You can set view-frustum-cull false in Config.prc to experiment with disabling culling altogether, or you can explicitly set an OmniBoundingVolume as the camera’s cull bounds.


#13

distance 12000 units, no flicker.


#14

I tried: view-frustrum-cull false but it didn’t seem to help.

I did try this: lens-far-limit 1e-10
I tried a range of values – smaller was not always better.
In many cases the water plane worked but the terrain disappeared!
This is tricky to disentangle. The terrain does its own cull and it therefore doing its own matrix operations in C++ on the CPU separate to the GPU. There are still special rotation angles I was able to find where the terrain would come back. even for self.camLens.set_near_far(5e30,0.00001) and lens-far-limit 1e-10.

I also tried to test the suggestion that 1e-6 was effectively zero. I did this by increasing all scales by a factor of 100, so that the scene is now ~ 100,000 wide and 100,000 from the camera. The ratio where problems occur stayed the same – it fails for 0.001 instead of 0.00001

This make me think it may be round-off somewhere. On this page:
https://developer.nvidia.com/content/depth-precision-visualized
the writer, Nathan Reed, cites a paper for two additional steps::

As mentioned earlier, Upchurch and Desbrun studied this and came up with two main recommendations to minimize roundoff error:

  • Use an infinite far plane.
  • Keep the projection matrix separate from other matrices, and apply it in a separate operation in the vertex shader, rather than composing it into the view matrix.

#15

With further experimentation, I see now that my initial approach was a bit naïve; I was just letting Panda continue to generate a GL-style projection matrix and only later convert it to a D3D-style matrix via a matrix multiplication. The problem is with one of the terms of the projection matrix, which approaches -1 as n goes to \infty:

\lim_{n \to \infty} \frac{f + n}{f - n} = -1

Precision of an infinitesimal far distance near -1 is nothing short of terrible, whereas when generating the D3D matrix, the values approach 0, allowing the far distance to enjoy the fullest range of floating-point precision:

\lim_{n \to \infty} \frac{f}{f - n} = 0

I have checked in a change that makes Panda take the limits directly when you explicitly set the near distance to float("inf"). This means it will now be better to set it to infinity than to a very large value. However, I will still want to let Panda generate the D3D-style matrix to begin with, rather than throwing away useful precision.

Your point about keeping the projection matrix separate is good as well. It is easy to change the terrain shader to keep the modelview and projection matrices separate, by adding this to the inputs:

uniform mat4 p3d_ProjectionMatrix;
uniform mat4 p3d_ModelViewMatrix;

…and changing this line:

  gl_Position = p3d_ModelViewProjectionMatrix * vec4(chunk_position, 1);

…to this:

  gl_Position = p3d_ProjectionMatrix * (p3d_ModelViewMatrix * vec4(chunk_position, 1));

I will continue to experiment and keep you updated.


#16

Addendum: regarding ShaderMeshTerrain, it is indeed a problem in the culling, because if I comment out this line in shaderTerrainMesh.cxx, I can make the far distance as small as I want and the terrain will still show up:

    if (intersection == BoundingVolume::IF_no_intersection) {
      // No intersection with frustum
      return;
    }

It’s not yet clear to me why the test is failing, I will have to investigate this further.