Specular maps and Normalmaps together

Report bugs, bug fixes and workarounds here.
Topic author
cartrite
Posts: 1778
Joined: 15.09.2005
With us: 14 years
Location: Pocono Mountains, Pennsylvania, USA

Specular maps and Normalmaps together

Post #1by cartrite » 14.03.2019, 20:30

This problem goes back to version 1.5.1. If I use a normalmap on earth with a specularmap the color gets all messed up. As far as I can tell, sometime after version 1.5.1 and before 1.6.1 a feature was added to normalmaps to use specular highlights. This allowed spec maps for a normalmap. But it conflicts with the specmap used for highlighting water on the earth texture. The only way I could find to deal with it was lowering the specular color. But that makes the spec map almost unusable.

This image shows a normalmap and specmap together running on version 1.5.1.
151.jpg


This image shows a normal and specmap running on version 1.6.1 and looks the same up to the current version.
161.jpg


I've looked though the code for the past couple of days and couldn't find what was done to cause this so ........
I think it probably has something to do with adding specular highlights on normalmaps. I'm not aware of any workaround.

cartrite
Toshiba Satellite P875=S7200 laptop, Intel i5 processor 2.5 ghz 6 gb ram, Graphics Intel(R) HD Graphics 4000 openSUSE Leap42.3
and Tumbleweed

Avatar
John Van Vliet
Posts: 2710
Joined: 28.08.2002
With us: 17 years

Post #2by John Van Vliet » 15.03.2019, 03:31

without even knowing what your hardware is i can not even guess
what is your 3d card and WHAT driver are you using - also WHAT is the Operating system ?
Linux
Apple
MS Windows XP,7,8,10, what ?

i have no such issue and have not had this issue in all my years


the only thing i can GUESS is your hardware is not using openGL or the Nvidia drivers

Topic author
cartrite
Posts: 1778
Joined: 15.09.2005
With us: 14 years
Location: Pocono Mountains, Pennsylvania, USA

Post #3by cartrite » 15.03.2019, 12:21

The hardware list is in my signature. OS is Tumbleweed. I took the first image using the windows version 1.5.1 running on wine. That one looks right. The other was running the gtk version 1.6.1. But it looks same on qt5 running 1.7.0 beta. Current source. Open GL 3.0. If nobody else has this problem it's probably a driver issue.
Toshiba Satellite P875=S7200 laptop, Intel i5 processor 2.5 ghz 6 gb ram, Graphics Intel(R) HD Graphics 4000 openSUSE Leap42.3
and Tumbleweed

onetwothree
Developer
Posts: 280
Joined: 22.09.2018
With us: 11 months 29 days

Post #4by onetwothree » 17.03.2019, 18:47

cartrite wrote:.

Hi, could you provide cel url and changed celestia data files if any so we can check on different hardware?

Topic author
cartrite
Posts: 1778
Joined: 15.09.2005
With us: 14 years
Location: Pocono Mountains, Pennsylvania, USA

Post #5by cartrite » 17.03.2019, 21:11

Code: Select all

cel://Follow/Sol:Earth/2019-03-17T20:52:37.95569?x=cHKSJfVkBAE&y=L9KNcHYx&z=WrEuiGiyMQ&ow=0.77054&ox=0.00189109&oy=-0.637387&oz=0.0017144&select=Sol:Earth&fov=29.8169&ts=1&ltd=0&p=0&rf=4164738951&lm=2048&tsrc=0&ver=4


I just got 1.6.1 gtk running normally. It depends on the shaders folder being in the main directory. /usr/local/share/celestia. Running from a terminal, it complains if there is no shader folder and gets an error loading ARB vertex program.

Code: Select all

Initializing ARB vertex programs . . .
Loading ARB vertex program: shaders/diffuse_arb.vp
Error loading ARB vertex program: shaders/diffuse_arb.vp
render path: 8

With the folder present, it loads it and all looks good. 1.6.1 still has the other render modes but it is running in opengl2.0.

Code: Select all

Initializing ARB vertex programs . . .
Loading ARB vertex program: shaders/diffuse_arb.vp
Loading ARB vertex program: shaders/specular_arb.vp
Loading ARB vertex program: shaders/haze_arb.vp
Loading ARB vertex program: shaders/bumpdiffuse_arb.vp
Loading ARB vertex program: shaders/bumphaze_arb.vp
Loading ARB vertex program: shaders/shadowtex_arb.vp
Loading ARB vertex program: shaders/diffuse_texoff_arb.vp
Loading ARB vertex program: shaders/rings_arb.vp
Loading ARB vertex program: shaders/ringshadow_arb.vp
Loading ARB vertex program: shaders/night_arb.vp
Loading ARB vertex program: shaders/glossmap_arb.vp
Loading ARB vertex program: shaders/diffuse2_arb.vp
Loading ARB vertex program: shaders/haze2_arb.vp
Loading ARB vertex program: shaders/diffuse_texoff2_arb.vp
Loading ARB vertex program: shaders/specular2_arb.vp
Loading ARB vertex program: shaders/night2_arb.vp
Loading ARB vertex program: shaders/star_arb.vp
Loading ARB vertex program: shaders/multishadow_arb.vp
Loading ARB vertex program: shaders/texphong_arb.vp
Loading ARB vertex program: shaders/texphong_alpha_arb.vp
Loading ARB vertex program: shaders/ell_galaxy_arb.vp
All ARB vertex programs loaded successfully.
render path: 8


When I look at the shaders log in 1.7 it seems to have the programs there but....... The 1.7.0 beta versions don't read from the shaders folder and they look the same. Version 1.6.1 is reading / loading files from /usr/local/share/celestia. The data files, solarsys.ssc and celestia.cfg do this out of the box. No changes. I did play with SpecularColor and it seems to color the ocean parts of earth with whatever color is present in the SpecularColor value in solarsys.ssc. The normal map was made with nmtools-2.0pre2. Tried both png and dxt5nm formats. Does it with BumpMaps too. The spec map is in the alpha channel but I did a rgb texture of earth and did a separate spec map. Didn't work.

cartrite
Toshiba Satellite P875=S7200 laptop, Intel i5 processor 2.5 ghz 6 gb ram, Graphics Intel(R) HD Graphics 4000 openSUSE Leap42.3
and Tumbleweed

Topic author
cartrite
Posts: 1778
Joined: 15.09.2005
With us: 14 years
Location: Pocono Mountains, Pennsylvania, USA

Post #6by cartrite » 18.03.2019, 18:22

Celestia-f48a72f9bc97d7e17da6ecd3952d0f9c9af36ed0
This builds and runs fine. Committed on Feb. 24 2018.Soon after this, and I'm not sure when exactly, normalmaps act strangely on my system. Gonna try and see what is the real cause but a lot has changed since then.
Toshiba Satellite P875=S7200 laptop, Intel i5 processor 2.5 ghz 6 gb ram, Graphics Intel(R) HD Graphics 4000 openSUSE Leap42.3
and Tumbleweed

onetwothree
Developer
Posts: 280
Joined: 22.09.2018
With us: 11 months 29 days

Post #7by onetwothree » 18.03.2019, 18:35

Let git bisect help you :)

Topic author
cartrite
Posts: 1778
Joined: 15.09.2005
With us: 14 years
Location: Pocono Mountains, Pennsylvania, USA

Post #8by cartrite » 19.03.2019, 03:02

Celestia-e290322b1c04cbbc75c9c1f354a7081dd8dd57a3
That was the last commit that worked normally. The next commit described as Finish eigenization
Celestia-85829063e0683e6de4c824f9f26121bc36f21536
was broken. Not what I was seeing above. It showed no earth. No png or dds files were loading. I'm not sure when they started being visible again. Looking..........

Added after 1 hour 24 minutes:
Somewhere between commits 73a869c and f478faa earth becomes visible again with too much specular.
Celestia-f478faa5a419f14aae4968a40db334468fbfee8b
Celestia-f478faa5a419f14aae4968a40db334468fbfee8b.jpg

Celestia-85829063e0683e6de4c824f9f26121bc36f21536
Celestia-85829063e0683e6de4c824f9f26121bc36f21536bad.jpg

Celestia-e290322b1c04cbbc75c9c1f354a7081dd8dd57a3
Celestia-e290322b1c04cbbc75c9c1f354a7081dd8dd57a3.jpg
Toshiba Satellite P875=S7200 laptop, Intel i5 processor 2.5 ghz 6 gb ram, Graphics Intel(R) HD Graphics 4000 openSUSE Leap42.3
and Tumbleweed

onetwothree
Developer
Posts: 280
Joined: 22.09.2018
With us: 11 months 29 days

Post #9by onetwothree » 19.03.2019, 07:15

Hi cartrite,

Thank you for investigations!

If you have time, could you test 85829063 with 8a894bfb cherry-picked? And then with 4ad277ed. (85829063 has an error in frustum computation and 8a894bfb provides an workaround for it (i still fill shame for publishing such terrible code) while 4ad277ed has a proper fix.)

Could you also send me files you used to reveal the error? So that I can reproduce the error.

Topic author
cartrite
Posts: 1778
Joined: 15.09.2005
With us: 14 years
Location: Pocono Mountains, Pennsylvania, USA

Post #10by cartrite » 19.03.2019, 11:05

OK, 8a894bfb fixed the problem, or worked around the problem completely. Looked normal. But 4ad277ed broke it again. The planet was visible but too much specular like above.
Toshiba Satellite P875=S7200 laptop, Intel i5 processor 2.5 ghz 6 gb ram, Graphics Intel(R) HD Graphics 4000 openSUSE Leap42.3
and Tumbleweed

onetwothree
Developer
Posts: 280
Joined: 22.09.2018
With us: 11 months 29 days

Post #11by onetwothree » 19.03.2019, 13:34

Hmm, i will try reproduce it locally. Basically 8a894bfb and 4ad277ed do the same thing (compute inverted planet view matrix), but 4ad277ed does it properly from vector math's point of view, i bet they produce they same results.

Topic author
cartrite
Posts: 1778
Joined: 15.09.2005
With us: 14 years
Location: Pocono Mountains, Pennsylvania, USA

Post #12by cartrite » 19.03.2019, 13:56

I'm sure they do. But something else may have been changed between the 2 commits. When I was trying current code, I was changing specular color in solarsys.ssc. The color of the oceans were the color I used. So somehow, by using a normalmap, the specular map was ignored and only the specular color was used?
Toshiba Satellite P875=S7200 laptop, Intel i5 processor 2.5 ghz 6 gb ram, Graphics Intel(R) HD Graphics 4000 openSUSE Leap42.3
and Tumbleweed

onetwothree
Developer
Posts: 280
Joined: 22.09.2018
With us: 11 months 29 days

Post #13by onetwothree » 19.03.2019, 14:16

So, you misunderstood me. My idea was that you use 85829063 as a base, then apply 8a894bfb and 4ad277ed only (using git cherry-pick). But nevermind, with your dataset I can reproduce the issue so I will do this myself.

The only possible change between 8a894bfb and 4ad277ed is vertex program removal, but in GL2 mode they were used for stars only. I need to check this better.

Topic author
cartrite
Posts: 1778
Joined: 15.09.2005
With us: 14 years
Location: Pocono Mountains, Pennsylvania, USA

Post #14by cartrite » 19.03.2019, 14:17

F478faa5 was just before 4ad277ed. The captured image from f478faa5 was posted above. 4ad277ed looked the same. In fact I downloaded 4ad277ed last night and was gonna try it this morning next. I was unaware that 8a89 had fixed the invisible planet. That seemed to be the last commit that rendered the planet correctly on my system. So my real problem was caused between that commit and 4ad277ed.

Added after 5 minutes 1 second:
I'm not too good at using git. I was unaware that cherry pick was a command. I couldn't even figure out how to use bisect. I just went on github and downloaded zips :weirdface: quite a challenge.
Toshiba Satellite P875=S7200 laptop, Intel i5 processor 2.5 ghz 6 gb ram, Graphics Intel(R) HD Graphics 4000 openSUSE Leap42.3
and Tumbleweed

onetwothree
Developer
Posts: 280
Joined: 22.09.2018
With us: 11 months 29 days

Post #15by onetwothree » 19.03.2019, 15:14

cartrite, are you sure that your e290322 is correct? With files you sent I have the following:

e290322:
Spoiler
Image

the same for 8a894bfb:
Spoiler
Image

f478faa5 and 4ad277ed look differently:
Spoiler
Image Image

I even compiled old commit a1043fb, it looks like e290322:
Spoiler
Image
Attachments
cel-8a894bfb.png
8a894bfb, workaround for missing planets
cel-f478faa5.png
f478faa5, before fixing planet view matrix properly
cel-e290322b.png
e290322b, before moving to eigen
cel-a1043fbf.png
a1043fbf, very old commit
cel-4ad277ed.png
4ad277ed, view matrix fixed

Topic author
cartrite
Posts: 1778
Joined: 15.09.2005
With us: 14 years
Location: Pocono Mountains, Pennsylvania, USA

Post #16by cartrite » 19.03.2019, 15:23

Although its possible that commit b9d532a90 caused the problem, I checked the next one
Celestia-6653929310fc6a6bcf7bb3151285031cd93b4252
and the problem was there. So maybe something was forgotten when removing vertexprog.cpp from the library or there was a bool statement in render.cpp removed that related to bumpMapping. Or maybe it was disabling done in the prior commit.

Added after 1 hour 10 minutes:
Yeah. Im sure. That long number was copied from the folder. I downloaded the zip file from github. Then extracted it. It put the name on the extraction folder.

I'm using libpng16. That's what created the textures. I think it did. I know the normalmap was cause I had to build nmtools. But the bmng files may have used other. I created that texture with f-tex-tools and they were probably built with a different png lib. Not sure if that would matter though. Strange that you get no good results. When I get back, I'll try to post the zip file for the textures here and maybe someone else can try.

Added after 1 hour 32 minutes:
If anyone else want to try and test this, this is a virtual texture created by FTex-Tools-2.0-pre2 and nmtools-2.0pre2. It's levels 0 and 1 . All the folders and ctx files included. It contains the July version of the Blue Marble (world.200407.3x86400x43200.bin )with a specular map in the alpha channel. It also has a normalmap that used the srtm_ramp2.world.86400x43200.bin file. There is also a data folder with solarsys.ssc. If you have a working version of Celestia 1.7.0 it will probably show problems as above.

Celestia_QT.zip
(11.43 MiB) Downloaded 9 times


Added after 24 minutes 8 seconds:
b9d532a is were I start getting the problem. 8a894bf and 717bf34 both work correctly.
Last edited by cartrite on 21.03.2019, 18:10, edited 1 time in total.
Toshiba Satellite P875=S7200 laptop, Intel i5 processor 2.5 ghz 6 gb ram, Graphics Intel(R) HD Graphics 4000 openSUSE Leap42.3
and Tumbleweed

Topic author
cartrite
Posts: 1778
Joined: 15.09.2005
With us: 14 years
Location: Pocono Mountains, Pennsylvania, USA

Post #17by cartrite » 20.03.2019, 14:24

I noticed something strange today. I was getting similar results that you were getting. All the commits I thought were working were showing results that you were getting.
This was after I renamed the shader folder to stop its use. To make a long story short, You seem to need the shader folder in Celestia_QT folder or your top celestia working folder. For these versions, to get results I was getting, the shader folder must be present. At least on my system.
Toshiba Satellite P875=S7200 laptop, Intel i5 processor 2.5 ghz 6 gb ram, Graphics Intel(R) HD Graphics 4000 openSUSE Leap42.3
and Tumbleweed

onetwothree
Developer
Posts: 280
Joined: 22.09.2018
With us: 11 months 29 days

Post #18by onetwothree » 21.03.2019, 21:19

cartrite, could you confirm that https://github.com/CelestiaProject/Celestia/tree/fix-specular works fine?

Topic author
cartrite
Posts: 1778
Joined: 15.09.2005
With us: 14 years
Location: Pocono Mountains, Pennsylvania, USA

Post #19by cartrite » 21.03.2019, 22:44

Thank You Much. Fixed!!!!!!!!!!!!! :smile: I had a feeling it had something to do with lodspheremesh.cpp. Had no idea how to fix it though.
Toshiba Satellite P875=S7200 laptop, Intel i5 processor 2.5 ghz 6 gb ram, Graphics Intel(R) HD Graphics 4000 openSUSE Leap42.3
and Tumbleweed

onetwothree
Developer
Posts: 280
Joined: 22.09.2018
With us: 11 months 29 days

Post #20by onetwothree » 22.03.2019, 07:15

That's was an example of a really bad code. A hidden, implicit dependency between VP and GLSL code paths. But thanks to your help it was quite easy to fix it.


Return to “Bugs”

Who is online

Users browsing this forum: 2 guests