Cool VL Viewer forum

View unanswered posts | View active topics It is currently 2024-03-28 19:37:41



Reply to topic  [ 9 posts ] 
Discard bias yo-yo (was: Perf regression in populated areas) 
Author Message

Joined: 2022-04-13 22:13:27
Posts: 8
Reply with quote
Performance regression after 1.30.0.34 in heavy populated areas, seemingly seems re-loading textures after initial loading

Description: I seen a performance regression in all viewers later than the mentioned version in heavy populated sites (eg Peak Club, best to see > 50 characters). The initial loading of texture (ctrl+shift+3) seems same. For me that is 2-3 minutes. After this the behavior is changing - while 1.30.0.34 is very "quiet" in textures loading, only new arrivals / particles seems, the later versions seems periodically (for me 1-3 minutes) also to reload already successful loaded avatars / textures. I can also visibly see characters quickly vanish partially and reappearing (because reloading its textures?).
I use cpu affinity settting and see all non dedicated core continue running 100% texture loading, instead 1.30.0.34 load is then according to new arrivals it seems.
Settings are the same for all versions tested. I use light & shadows.

I will investigate that further, to get a better descreption to reproduce, or helpful details.

Until now - How to reproduce: go into a heavy populated area like PEAK club, best 50+ characters, [EDIT] cam out so you see many characters [/EDIT] wait initial loading all textures etc. is finished. Wait more, and see a periodically reloading (ctrl+shift+3) and fps drop during reloading. Not seen that with 1.30.0.34.

I suspect this change, but my data basis is very weak: "Improved the texture discard bias change algorithm to minimize the occurrences of freezes in extreme bound GL textures usage conditions while turning your avatar/camera around."
[EDIT] And VRAM is always less than 90% as i set texture usage of memory to moderate 2,4 GB[/EDIT]

Cheers & sorry for the poor information, but i think better to report than wait more

CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz (4120.81 MHz)
Memory: 32038MB
OS version: Linux-x86_64 v5.14.21-150400.24.41-default #1 SMP PREEMPT_DYNAMIC Fri Jan 13 08:55:22 UTC 2023 (1d4442d)
Memory manager: jemalloc v5.3.1-20221122
Graphics card vendor: NVIDIA Corporation
Graphics card: NVIDIA GeForce GTX 1070/PCIe/SSE2
OpenGL version: 4.6.0 NVIDIA 525.78.01
Detected VRAM: 8192MB
J2C decoder: OpenJPEG: 1.4.0.635f
Audio driver: FMOD Studio v2.02.11 (ALSA)


2023-02-13 22:46:46
Profile

Joined: 2009-03-17 18:42:51
Posts: 5523
Reply with quote
This is not a performance regression, but on the contrary an improvement.

Go to the same club with v1.30.0.34. Wait for everything to load. Turn your avatar around and observe the HUGE hiccup with single-digit frame rates for several seconds. Do the same with the current release: no hiccup, or a moderate one only.
This issue is particularly annoying when walking around in highly textured environments (scenic sims, sims with a sh*tload of materials, etc), and it will only get worst with the future PBR renderer (up to 5 textures per face: diffuse texture (for compatibility with old renderers) + 4 PBR channels).

What happened ?... When the GL bound textures overshoots 3GB or so, NVIDIA cards choke big time, until the usage of this type of memory regresses below 2.5GB, and when turning your avatar around, you increase suddenly the bound textures number as new objects come into your camera FOV, while others that are no more in it have not yet been removed from memory and their bound textures freed.

What I did in v1.30.0.35 ?
I changed the texture discard bias algorithm to properly keep the bias at the same level as long as the bound texture memory usage is high (the "soft limit" is set to 2.5GB and can be adjusted via the "MaxBoundTexMem" debug setting), and I smoothed out the decrease in the bias so to avoid sudden changes, e.g. form 4.0 to 0.0 and then suddenly to 5.0 when you turn around, with large overshoots way above 3GB of bound textures.

In the texture console, observe the "Bias" number as well as the "Bound" figure, and you will see what I mean.

Now, the side effect is that the viewer takes more time to settle at an average bias that suits the textures usage, thus the reloading of textures after a few minutes or when people arrive/leave in your FOV. When the bias changes, the LOD of textures change as well, and the said textures must be re-decoded at the new LOD, thus the CPU usage spike.

Of course, the algorithm is not perfect, but I find it much smoother/better than what we previously had...


2023-02-13 23:33:39
Profile WWW

Joined: 2009-03-17 18:42:51
Posts: 5523
Reply with quote
I made some experiments, in crowded areas, zooming out like you did.

In these conditions, one of the causes for constant re-decoding of textures is the avatar "impostoring" feature: when zoomed out, the variation in the distance to the camera of the various avatars is much smaller, and causes "random" changes in avatars impostoring (i.e. for each avatar, the impostor status could change every few seconds), incurring attachments textures re-decoding at various LoD levels as well as the corresponding bound/unbound GL textures changes; since, nowadays, almost everything in an avatar is an attachment (and most of the time, with materials, i.e. with up to 3 textures per face), things can get messy...
One potential workaround for such cases is to turn impostors off (move the slider to the extreme left in the graphics settings); of course, this means more avatars to fully render and thus lower frame rates...

I also found out that it might be worth changing your "Texture memory" setting to lower it a bit, in such conditions (I know, it sounds counterintuitive)...

Finally, I further improved the discard bias algorithm to reduce spikes, and I am currently exploring a new, more reliable method for the bound textures accounting (the one used by the PBR project viewer), that seems also to improve things quite a bit... So, stay tuned for next release. :P


2023-02-15 10:48:45
Profile WWW

Joined: 2022-04-13 22:13:27
Posts: 8
Reply with quote
I did some more testing and if i understood you right, to simplify it - newer viewer versions just need more time to to balance all. I really stayed a long time in a heavy populated area,using camera carefully while looking at textures loads too. I still have the "better" behavior what you described as intended with 1.30.0.34. its less load on the cores used for texture loading, and stable fps after initial loading (ofc peaks dropping fps when start to cam around but normalizes soon). Even latest version (last Saturday) then after a while stays average circa 50% fps of 1.30.0.34. I will test more try to find out what influences it in my setup.
Btw - would it be possible maybe get the old code inside again and make it selectable to be used by a debug setting?
I am looking forward to see what you found out and will incooperate into next viewer version. Thank You so much for always keeping CoolVL Viewer ahead.


2023-02-15 23:40:14
Profile

Joined: 2009-03-17 18:42:51
Posts: 5523
Reply with quote
Today's release should largely solve the yo-yo effect (that, by the way, v1.30.0.34 was not immune against, by far: it simply happened under different conditions).

In this release, I reworked the texture bias change and related algorithms as follow:
  • Increased the maximum clamping value for the 10% minimum free VRAM from 500MB to 1000MB (meaning, for a 8GB VRAM card, the bias will start increasing when around 750MB of texture VRAM is left free, instead of 500MB): this helps avoiding to see the textures spill over the RAM (and the resulting second-long freezes in rendering) when the VRAM usage variates so fast that the bias algorithm cannot catch up in time.
  • Added a feature that automatically increases (by 1 only) the ALM textures discard level whenever the bound textures memory usage gets high (above 80% usage by default, adjustable via the new "BoundTexRatioToBiasALM" debug setting); note that the "ALM textures" are the normal and specular maps and the light projector textures, and the ensuing increase in their discard level got almost no visible impact (way less than the impact on the diffuse map textures that their "sacrifice" allows to preserve).
  • Added a "TexBiasDecreaseDelayFactor" debug setting, which allows to adjust the speed at which the texture discard bias is decreased (the delay in seconds between two decreases is obtained by multiplying this factor with the current discard bias). This greatly helps in avoiding the yo-yo effect with textures re-decoding in busy places where the texture memory gets full, while being a smarter algorithm than what was implemented in v1.30.0.35 (which was a constant delay). Note that bias changes occur at most once every 500ms, so the product TexBiasDecreaseDelayFactor*current_bias will be "rounded down" to the nearest 500ms (e.g. and for the default 0.8 value, it means that this factor got no effect at all for discard bias values of 0.5 or less).
  • Now automatically disable (for 30 seconds since the last time the bias reached 3.0) the texture fetching boost feature (when enabled). This also helps a lot against the yo-yo effect (loading textures faster and at a more accurate LoD is nice, but doing it while the bias algorithm was trying to find an average value was only destroying it). You may still force-boost with "CTRL B" ("Advanced" -> "Rendering" -> "Textures" -> "Boost textures fetches now"), possibly pressed twice or thrice in a single second, if (and only if) in urgent need to get the bias algorithm "un-stuck" (it may happen it stays stuck at a 5.0 bias, after abusing it violently, e.g. with Radar "Focus" actions repeated several times a second in a very busy or heavily textured place).
  • Do not any more force-release unused (non-GL) textures every frame when the discard bias reaches 4.0 (I implemented this in the 32 bits viewer time, to free some RAM as fast as possible when about to crash by lack of free virtual address space, but it is now largely useless); this algorithm now only kicks in when actually low on texture RAM (not VRAM, not bound GL textures memory), only at 4.5 discard bias or above, and only every 500ms. This old algorithm was one of the main reasons for the heavy slow downs in fps rates seen with former releases when reaching a discard bias of 4.0 or more.


2023-02-18 14:03:07
Profile WWW

Joined: 2022-04-13 22:13:27
Posts: 8
Reply with quote
I just make it short: great work done! It combines the performance gains already made also in heavy populated areas. I did test for some days, prefect. Even more potential to fine tune as parameters get available by debug settings (I won't do now its just working fine with the defaults). P.S. a short description what exactly the Bias setting does would be useful as i didn't found any good reference by now (just an idea). Big Thank's !


2023-02-22 22:30:42
Profile

Joined: 2009-03-17 18:42:51
Posts: 5523
Reply with quote
GismoZ wrote:
a short description what exactly the Bias setting does would be useful as i didn't found any good reference by now (just an idea).
The discard bias is not a setting, but a parameter computed (and updated every 500ms) by the viewer, which increases as needed to keep the texture memory usage within the allowed/configured limits (it takes into account your configured "texture memory" graphics setting, the maximum bound GL textures limit, and the free VRAM); when it increases the discard level of textures increases with it, meaning textures are re-decoded at a lower LoD level (lower resolution), so to consume less memory. When less texture memory is used, the discard bias decreases as well, allowing more detailed textures to be decoded and used again.


2023-02-22 23:07:24
Profile WWW

Joined: 2009-03-17 18:42:51
Posts: 5523
Reply with quote
In today's v1.30.2.4 release I added:
Quote:
Slightly lowered the limit (from 90% to 87.5% of the available texture GL and bound memory pools) under which the textures discard bias is allowed to decrease, and added a "TexMemLowerBoundScale" debug setting to allow adjusting it; this reduces the likeliness to enter a textures reloading yo-yo in textures-heavy places.


2023-03-11 10:54:22
Profile WWW

Joined: 2009-03-17 18:42:51
Posts: 5523
Reply with quote
In today's release, I improved again the texture discard bias algorithm by using the last memory usage to decide whether the bias should be increased or decreased, in excess of the current memory usage, i.e. extrapolating figures for next update. This helps a great deal to avoid entering a yo-yo by avoiding (sometimes large) overshoots while decreasing or increasing the discard bias.

I also added a new "TextureUpdateBoostRatioPerDiscard" debug setting to adjust how much more textures we update per frame proportionally to the discard bias value (this setting is used to re-decode more textures at the new bias and free memory faster), but reduced its former (hard-coded) value from 1.0 to 0.25 (*) (the higher the value, the larger the impact on the frame rate), since with last weeks' changes, we also now force-free memory on discard bias changes, reducing the need for this feature.

I also added an "Upd/frame" (texture decode priority updates per frame) stat to the texture console.


(*) I.e. at discard 5.0 we decode 2.25 times more textures instead of 6.0 times more (the multiplier equals to 1.0 + discard_bias * TextureUpdateBoostRatioPerDiscard).


2023-04-01 09:55:28
Profile WWW
Display posts from previous:  Sort by  
Reply to topic   [ 9 posts ] 

Who is online

Users browsing this forum: No registered users and 26 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group
Designed by ST Software.