Cool VL Viewer forum - View topic - Memory Usage Safety Check

View unanswered posts | View active topics

It is currently 2024-04-24 22:15:31

Memory Usage Safety Check

Page 2 of 3

[ 25 posts ]

Go to page Previous 1, 2, 3 Next

Print view

Previous topic | Next topic

Memory Usage Safety Check

Author

Message

Henri Beauchamp

Joined: 2009-03-17 18:42:51
Posts: 5550

Re: Memory Usage Safety Check


	Raish wrote: It does in a way. I watched my system work with its memory while running Cool VL. It is swaping parts of Cool VL's memory into the pagefile even while it's active and on top. None the less WebGearbox's system actually used only 0.5 GB of it's physical memory.

That's exactly what I said... The 1.5Gb "free memory" was actually an illusion (because part of the actually used memory was swapped out). Windoze is known to swap even when it doesn't need it... which only makes it slower when compared to other OSes.

Raish wrote:


	Henri Beauchamp wrote: do you REALLY want to see your computer swapping while the viewer runs ?... It would cause a MASSIVE slow down !

I'd rather have a slow down instead of a crash or a view distance of max 100m. Also its running smoothly, don't really notice problems with the swaping.

There *is* a slow down each time the viewer needs to access textures or objects data that were swapped and be sure this data is regularly needed: the viewer got its own algorithms to discard the textures and objects it doesn't need any more (those which were not "seen" in the last two minutes or so), so whatever Windows is swapping to the hard drive is actually needed data and not just old data that is rarely used...

But who am I to deny you the "pleasure" of slow swapping ?... The next release will add a setting (off by default) that will allow to ignore the amount of virtual memory reported by the OS and will only watch for the 3Gb high mem "wall" (4Gb when running on 64bits OSes) so to avoid crashes; this will effectively allow swapping... I also added a small bar graph (placed on the left of the packet loss bar graph in the status bar) that reports the percentage of the usable memory currently in use by the viewer.

2011-12-22 15:39:32

WebGearbox

Joined: 2011-07-27 13:02:35
Posts: 78
Location: Ukraine

Re: Memory Usage Safety Check

I bought 1 more GB of RAM

Thank you, Henri

2011-12-22 16:15:39

Raish

Joined: 2011-12-04 15:00:58
Posts: 22

Re: Memory Usage Safety Check


	Henri Beauchamp wrote: The next release will add a setting (off by default) that will allow to ignore the amount of virtual memory reported by the OS and will only watch for the 3Gb high mem "wall" (4Gb when running on 64bits OSes) so to avoid crashes; this will effectively allow swapping...

Actually virtual memory should be exactly the right value to determine if we are in trouble. In my case it should report as (3.3 GB) - (memory used by Cool VL), no matter how much other resource hoging stuff is running. Without swaping it should be the actual physical free RAM.
Something is fishy about the value you get there.

Not a fan of many new options if it can be avoided, as it's not newbie friendly.


	Henri Beauchamp wrote: I also added a small bar graph (placed on the left of the packet loss bar graph in the status bar) that reports the percentage of the usable memory currently in use by the viewer.

Nice!

2011-12-22 20:29:49

Henri Beauchamp

Joined: 2009-03-17 18:42:51
Posts: 5550

Re: Memory Usage Safety Check

Raish wrote:


	Henri Beauchamp wrote: The next release will add a setting (off by default) that will allow to ignore the amount of virtual memory reported by the OS and will only watch for the 3Gb high mem "wall" (4Gb when running on 64bits OSes) so to avoid crashes; this will effectively allow swapping...


	Quote: Nothing fishy at all...

Just weird, because of the stupid design of x86 processors. Even while with modern x86 32bits processors it is possible to address up to 4Gb, you still have to split it because the OS and the user space programs must be kept in separate areas (and I'm not even counting here the memory fragments that are used for I/O mapping, etc). Under 32bits Linux, you can configure the kernel at compile time to use 1Gb/3Gb, 2Gb/2Gb or 3Gb/1Gb splits (the default configuration being 1Gb/3Gb). Under Windows it depends on the version. With 32bits XP, the processes are normally allocated a maximum of 2Gb unless they were compiled (with a special compiler flag) to use 3Gb (this is the case for the Cool VL Viewer). So, even if you got 3.3Gb of free memory seen by the OS, only a maximum of 3Gb can be used by a single program and attempting to use more causes a crash.

Now, when running a 32bits program on a 64bits OS, the 4Gb addressing space barrier of the latter is lifted and the 32bits processes can be allocated up to 4Gb (since the program is itself 32bits, it can't address more than that).

2011-12-22 20:59:25

Raish

Joined: 2011-12-04 15:00:58
Posts: 22

Re: Memory Usage Safety Check

Not that good with C++, but i took a look at your code and you seem to be using a LL function to get the available memory which must be failing.

Maybe try with this:


	Code: #include <windows.h> long getAvailVirtualMemoryKB() { MEMORYSTATUSEX status; status.dwLength = sizeof(status); GlobalMemoryStatusEx(&status); return status.ullAvailVirtual/1024; }

Correctly returns a tad less than 2 GB for me.
Wanted to test it with the 3 GB as well but got trouble with seting it.

2011-12-23 11:23:37

Henri Beauchamp

Joined: 2009-03-17 18:42:51
Posts: 5550

Re: Memory Usage Safety Check

You didn't look at this "LL function" then (LLMemoryInfo::getAvailableMemoryKB()), because it does use the same code as you did. Please, do not insult my programming skills...

2011-12-23 11:43:05

Raish

Joined: 2011-12-04 15:00:58
Posts: 22

Re: Memory Usage Safety Check

Just trying to help, because something isn't working right.

2011-12-23 12:59:09

Henri Beauchamp

Joined: 2009-03-17 18:42:51
Posts: 5550

Re: Memory Usage Safety Check


	Raish wrote: Just trying to help, because something isn't working right.

It's working just fine on the contrary... FYI, I just tested the new code with the new "Allow Swapping" advanced setting, and guess what... Windows actually starts swapping (and pretty badly) when this setting is enabled.

2011-12-23 14:46:14

Raish

Joined: 2011-12-04 15:00:58
Posts: 22

Re: Memory Usage Safety Check

Finally found the function and here is the problem:


	Code: avail_physical_mem_kb = (U32)(state.ullAvailPhys / 1024); avail_virtual_mem_kb = (U32)(state.ullAvailVirtual / 1024); //return now and it should be fine .. but after #ifndef: U32 max_virtual_mem_kb = avail_physical_mem_kb; //that's the error of thought U32 virtual_size_kb = (U32)(counters.WorkingSetSize / 1024); virtual_size_kb += virtual_size_kb / 10; // 10% penalty for fragmentation avail_virtual_mem_kb = max_virtual_mem_kb - virtual_size_kb;

So essentially:


	Code: avail_virtual_mem_kb = avail_physical_mem_kb - virtual_size_kb * 1.1;

Not how it works on windows.
Maximum virtual memory is the maximum physical memory, or the 2 or 3 GB. As long as the pagefile/swaping is enabled anyways, which it is by default.

2011-12-23 14:55:54

Henri Beauchamp

Joined: 2009-03-17 18:42:51
Posts: 5550

Re: Memory Usage Safety Check

Raish wrote:

Finally found the function and here is the problem:


	Code: avail_physical_mem_kb = (U32)(state.ullAvailPhys / 1024); avail_virtual_mem_kb = (U32)(state.ullAvailVirtual / 1024); //return now and it should be fine .. but after #ifndef: U32 max_virtual_mem_kb = avail_physical_mem_kb; //that's the error of thought U32 virtual_size_kb = (U32)(counters.WorkingSetSize / 1024); virtual_size_kb += virtual_size_kb / 10; // 10% penalty for fragmentation avail_virtual_mem_kb = max_virtual_mem_kb - virtual_size_kb;

So essentially:


	Code: avail_virtual_mem_kb = avail_physical_mem_kb - virtual_size_kb * 1.1;

Not how it works on windows.
Maximum virtual memory is the maximum physical memory, or the 2 or 3 GB. As long as the pagefile/swaping is enabled anyways, which it is by default.

When you will stop stating the obvious... This is EXACTLY what I said and what I coded !!! The algorithm in v1.26.2.10 is coded so that swapping doesn't happen. in v1.26.2.11, there is an additional setting that allows swapping to happen, and then the available virtual memory is considered to be 3Gb (for 32bits OSes) or 4Gb (for 64bits OSes).

2011-12-23 15:10:11

Page 2 of 3

[ 25 posts ]

Go to page Previous 1, 2, 3 Next

Who is online

Users browsing this forum: No registered users and 52 guests

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum