Cool VL Viewer forum

Oh darn, just when I had a nice crash with the gdb thingy!

I'm attaching it anyway, maybe it's still useful. And I highly appreciate your work on this to the point of actually hanging out there!

Interesting... it's apparently a crash within the tcmalloc library (i.e. not really a viewer crash)... Will see if there is a new version with that bug fixed.

Uhm, I'll take your word for it!

Incidentally, I got exactly the same backtrace with the 1.26.1.6 (obviously with different addresses).

So, let's see, what's outlandish with my environment? 8 GB on 32 Bits using PAE, possibly? If there's any way to drill this down any further, I'd be glad to help.

Love,
Lia

Just take your own backtrace words for it:

It's no surprise, since the same tcmalloc library is used by all v1.26 versions (and also by LL's mesh viewers, though I just did a quick 'cmp' between the tcmalloc I'm using (the one v2.6-mesh used) with the one bundled with LL's v3.0.0 viewer: they are different although they got the same version number: you could try and copy the lib/libtcmalloc.so.0.1.0 library from v3.0.0 viewer into the Cool VL Viewer lib/ directory and see what happens...).

Nothing extraordinary (4Gb, 32bits with PAE, here).

However, after googling for your CPU model and "bug", I found this article (sorry, it's in French and I didn't find an equivalent in English): your CPU is perhaps (if it's actually an AMD Phenom II X4 965 (revision C2) and not a 965s (revision C3): see the table at the end of the article for the models, revisions and serial numbers) suffering from a bug that could show if you are using 4 DDR3 DRAM slots at 1333MHz. Not sure how the bug would show, but an exception during a memory allocation is not impossible. This said, a tcmalloc bug (that would show only in a specific memory environment, which would depend on the free memory, running processes, etc) is quite possible and your system may be perfectly fine.

Will do, have to download my first Linden viewer in over a year for that first!


Well, I checked the invoice and from the parts number given there and the 125W rating it should be a C3; the /proc/cpuinfo thingy says it's 'stepping 3', which I suspect would corroborate that story.

Again, thank you very much for looking into this. And now I'm off to download a (cough) Linden viewer.

Love,
Lia

Ok, I have a theory, after watching the issue a little closer. It may indeed be an out of memory issue. Not physical, but rather virtual memory. Due to my configuration and the fact that I'm pushing the poor viewer with my 2048x1152 main screen with loads of shinies enabled (I love taking pictures), it may actually run out of address space.

My graphics adapter wants 1024 MByte of virtual memory for indirect memory, which would leave 3 GB. And the last few times when I watched the processes on the second screen, the viewer always crashed once it came up to the 3 GB VIRT mark, every time without the stack trace file.

If that theory solidifies, then I guess that there is no solution to the problem except for changing my settings to less memory hungry ones (I guess lowering draw distance [256m] would probably a prime candidate, as allowing for less non-impostors [20]). Come to think of it, doing so may help pinning the issue down in the first place. I'll have a look and report.

Love,
Lia

Yes, I could also reproduce the issue by enabling the (very dangerous) "Full Res Textures" setting in a crowded place.
Even v3 viewers end up crashing with this setting on and in the same conditions, but they do more "elegantly" (if a crash can be elegant at all) by being OOM-killed (v3.0.0) or by detecting the out of memory issue and voluntarily committing suicide (v3.0.5 with private memory pools on).

With the Cool VL Viewer, the crash occurs during memory allocation for textures by tcmalloc, but I also noticed that it occurred while the texture cache was being cleaned-up with the anti-hiccup "split" cleaning algorithm (instead of hanging the viewer for a couple of seconds or more while the excess of cached textures is wiped out of the disk, like LL's viewers do, the Cool VL Viewer wipes them out in small batches, with a maximum 0.1s "hiccup" each time). This could be the sign of a conflict between APR pools memory allocations (which happen each time the viewer reads, writes or removes files) and tcmalloc memory allocations... The newest v3 viewers have better APR pool code, so I'll port this and see what happens.

There is however an annoying thing with tcmalloc: it never releases back "free" memory to the system, i.e. if the viewer once needed 2Gb of memory to render a complex/crowded scene, even after you TP in a desert sim with few textures, it selfishly keeps the 2Gb allocation for itself (note that this is not a leak, since the 2Gb will be enough for the viewer during the whole session, even when you return to a crowded place). This is a known drawback of tcmalloc, but it's very annoying...

Also annoying is the large memory overhead tcmalloc induces: for v1.26.0.20, I removed tcmalloc, and the viewer suddenly needs half the memory v1.26.0.19 needs to render the same scene !... I'll make v1.26.0.20 tcmalloc-free for this very reason but, alas, I can't do this with v1.26.1 since the mesh code needs tcmalloc under Linux and Windows because the system malloc() and new() calls do not provide the 16-bytes auto alignment required for SSE2 operations, and using them results in segfaults as soon as a miss-aligned variable is used in a SSE2 operation...

Finally, I find it quite abnormal, even with v3 viewers' "elegant" way of bailing out, that we get out of virtual memory while the system didn't even start to swap !

There's definitely something fishy with tcmalloc, but for now it's a necessary evil. I'll yet try to investigate further and will report my findings on the sldev mailing list for LL to pick up (Bao Linden seems to be working hard on better memory allocations, but he still uses tcmalloc and the true solution would be to replace it entirely).

Hello again Henri,

thank you for sharing your experiences there, which does indeed give me (and hopefully others) a bit of insight.

Definitely not on, as much as I would love it from a visual aspect at times.

Hmm, I was pondering trying with 3.x, but after reading this something tells me flat out it would be a waste of time. OOM kills are never really an issue for me, because the remainder of the system has plenty of RAM even if the viewer goes Berzerk. The viewer process simply seems to exhaust its entire address space, upon which it then decides to throw in the towel with the abort in tcmalloc as you have figured out.

I have been testing this for a bit now, and I can definitely say that once the process reaches a VIRT of 3072 MB in top, it's gone. The excessive (?) memory use seems connected to the number of AVs and can be accelerated by camming around. I can make the viewer crash this way at my favourite (aforementioned) testing spot in under two minutes now, dependably. I've desperately played with settings, and so far the obvious ones (lowering texture memory to 256 MBytes, halving draw distance to 128m and even sliding the avatar impostor slider down to just render 1 AV!) have failed to make a dent. I will try to figure out what other options I have and try those, in hope of finding something that will make a difference -- not that I want to hang out there, but it will be nice to know what can be done if need be.

Not that I understand half of it, but it touches on something that I've wondered for a long time. In this day and age of Terabyte hard drives, why are we limited to a comparably joking 1GB of cache? I wouldn't mind having the viewer cleaning up less and let the cache grow to a much higher size. Heck, if need be I'd put on my maid costume and clean it up myself!

Thank goodness, that one never really hurts me due to my 8 GB of memory.

Wow! I'll keep that in mind for recommendations, although I fear the time for non-mesh viewers may be running out soon at least on SL.

Well, I can add a little salt to that wound: with a 64 bit build, my problem would not exist (although I am sure I would have lots of lovely new ones ). Not to mention that the memory of my graphics card (it came with 2GB, the smaller one was no longer available at the shop, and backordering it would have saved me a measly 20 €) isn't even used more than a quarter or so at best.

But those are obviously issues not directed at you but rather at LL, who seems to try hard to keep us in the 20th century at times.

Love,
Lia

News from the memory front -- not really a solution as such, however an interesting observation: seems the viewer is (either consciously or through an odd usage of the system) actually loading all installed fonts available via fontconfig into memory.

If you have a large number of fonts installed, this will take a sizeable chunk out of memory available for other purposes (dare I say it was easily 700 MByte in my case? ).

I thought it might be a good idea to put this here, if only for documentation purposes.

Of course the viewer will still happily crash when reaching the 3072 MByte virtual mark, but it obviously does take notably longer to get there (if I'm intent on it, a few minutes now instead of just one or two at best).

Love,
Lia

Yes, all fonts get loaded... I'll see if v3 viewers do better, but I doubt it.

Anyway, on the memory front, I found out that the "Boost Attachment Textures" advanced option (in Advanced -> Rendering) could cause massive memory usage in very crowed areas (about half a Gigabyte for 50+ avatars around in Bondage Ranch, where avatars wear a lot of BDSM related attachments (collar, cuffs, belt, you name it...) with many detailed textures.

In v1.26.0.20 and v1.26.1.8 I changed the algorithm so that it's now much gentler on memory (easily halving the overhead). I also made it easier to configure (the "boosting" was already automatically shutting itself down when the "texture discard bias" was reaching 1.5, which means the texture buffer is filling up; you can now change this threshold value (which was lowered to 0.5 by default) with TextureBoostBiasThreshold to anything between 0.0 and 1.5) and changed the default minimum virtual size for attachments textures (TextureMinAttachmentsVSize) from 192x192 to 64x64.