Cool VL Viewer forum

Register • FAQ • Search • Login
View unanswered posts | View active topics It is currently 2025-08-25 07:55:13

Board index » Bug reports

All times are UTC




Reply to topic  Page 3 of 3
  [ 29 posts ]  Go to page Previous  1, 2, 3
Print view Previous topic | Next topic 
1.26.1.5 Crash out of nowhere 
Author Message
Alexandra.Rucker

Joined: 2011-03-09 13:18:17
Posts: 40
Reply with quote
Post Re: 1.26.1.5 Crash out of nowhere
I also have a very VERY large number of fonts - over 600 - on my system. Didn't realize that could be an issue. :/
2011-09-23 20:48:15
Profile
Amalia Illios

Joined: 2010-04-07 08:23:18
Posts: 215
Reply with quote
Post Re: 1.26.1.5 Crash out of nowhere
Henri Beauchamp wrote:
Yes, all fonts get loaded...
:o

That's kinda insane. Is that a Linux only thing or does it do that on other platforms as well?

I mean, I fixed it now by giving the viewer its own font.conf via setting an environment variable in the start script. But if this is happening on all platforms ... talk about oodles of RAM going to waste for people! :roll:

Henri Beauchamp wrote:
Anyway, on the memory front, I found out that the "Boost Attachment Textures" advanced option (in Advanced -> Rendering) could cause massive memory usage in very crowed areas (about half a Gigabyte for 50+ avatars around in Bondage Ranch, where avatars wear a lot of BDSM related attachments (collar, cuffs, belt, you name it...) with many detailed textures.
That makes sense. I don't think I have ever touched that one, but I'll check next time I'm on.

Henri Beauchamp wrote:
In v1.26.0.20 and v1.26.1.8 I changed the algorithm so that it's now much gentler on memory (easily halving the overhead). I also made it easier to configure (the "boosting" was already automatically shutting itself down when the "texture discard bias" was reaching 1.5, which means the texture buffer is filling up; you can now change this threshold value (which was lowered to 0.5 by default) with TextureBoostBiasThreshold to anything between 0.0 and 1.5) and changed the default minimum virtual size for attachments textures (TextureMinAttachmentsVSize) from 192x192 to 64x64.

Thank you for pointing that out and explaining it. I did a brief test run over at Damsel earlier, and I have to say I needed to cam around a lot more than usual to finally reach the 3 GB threshold this time, without having touched the values at all so far. I'll experiment some more and will report my findings.

I presume the reason this is hitting me may actually be a combination of the lacking 1GB in address space that my graphics adapter steals with the fairly extensive screen real estate. If I understand correctly, all LOD calculations and whatnot take the current size of the window/screen into account, so this may well result in a higher memory usage right from the start?

Love,
Lia
2011-09-24 06:54:09
Profile
Henri Beauchamp

Joined: 2009-03-17 18:42:51
Posts: 6043
Reply with quote
Post Re: 1.26.1.5 Crash out of nowhere
Amalia Illios wrote:
Henri Beauchamp wrote:
Yes, all fonts get loaded...
:o

That's kinda insane. Is that a Linux only thing or does it do that on other platforms as well?
It should affect all platforms since this is not Linux-specific code.

Quote:
I mean, I fixed it now by giving the viewer its own font.conf via setting an environment variable in the start script. But if this is happening on all platforms ...
Care to share your trick with us ? :)

Quote:
Henri Beauchamp wrote:
In v1.26.0.20 and v1.26.1.8 I changed the algorithm so that it's now much gentler on memory (easily halving the overhead). I also made it easier to configure (the "boosting" was already automatically shutting itself down when the "texture discard bias" was reaching 1.5, which means the texture buffer is filling up; you can now change this threshold value (which was lowered to 0.5 by default) with TextureBoostBiasThreshold to anything between 0.0 and 1.5) and changed the default minimum virtual size for attachments textures (TextureMinAttachmentsVSize) from 192x192 to 64x64.

Thank you for pointing that out and explaining it. I did a brief test run over at Damsel earlier, and I have to say I needed to cam around a lot more than usual to finally reach the 3 GB threshold this time, without having touched the values at all so far.
Yes, caming on avatars triggers the loading of their attachments textures at low discard level, so the memory usage climbs rapidly when you do. Nothing that can be done to prevent this to happen, alas.

Quote:
I presume the reason this is hitting me may actually be a combination of the lacking 1GB in address space that my graphics adapter steals
Actually, the viewer will never use over 512Mb of *texture* memory in your graphic card, but the rest of the memory is used for other purposes (VBO, FBO and stuff). The problem is that for each texture loaded in your video card memory, there is a copy of it in the PC main memory and the viewer uses a buffer to store these copies that is up to 150% larger than the texture memory used in the video card (meaning that if you set the video texture buffer size to 512Mb, the viewer will use up to 768Mb of RAM on the CPU side). You can significantly lower the memory usage by reducing the texture memory setting in the Hardware Graphic settings (reducing it to 256Mb should bring almost half a Gb of gain in crowded or texture-heavy sims), at the cost of more half-blurry textures in your field of view.

Quote:
with the fairly extensive screen real estate. If I understand correctly, all LOD calculations and whatnot take the current size of the window/screen into account, so this may well result in a higher memory usage right from the start?
This is true for HUD attachments (but this would make only a few megabytes of difference on larger screens), but I'm not convinced it's the case for the rest (since the field of view wideness should not depend on the screen size but only on the virtual camera zoom level and focal length)... You could see by yourself, by reducing the size of the viewer window.

In any case, I found a way to convince tcmalloc to release freed memory to the system and posted a new overlay package containing a patched tcmalloc version. See this message for details.
2011-09-24 14:57:11
Profile WWW
Amalia Illios

Joined: 2010-04-07 08:23:18
Posts: 215
Reply with quote
Post Re: 1.26.1.5 Crash out of nowhere
Henri Beauchamp wrote:
Care to share your trick with us ? :)
It's not a particularly clever or practical one. I copied fonts.conf to stupid.conf in /etc/fonts, removed all the <dir>...</dir> entries and added the viewer's fonts directory as sole directory containing fonts.

Then I added this to the startup script:
Code:
## - Avoid loading all 3195 million fonts available on the system
export FONTCONFIG_FILE=/etc/fonts/stupid.conf
This works for me, and reduces the startup memory usage (i.e. what is used when it sits waiting for login) by just about 700something MByte (for 609 fonts, or so fc-list claims).
Henri Beauchamp wrote:
You can significantly lower the memory usage by reducing the texture memory setting in the Hardware Graphic settings (reducing it to 256Mb should bring almost half a Gb of gain in crowded or texture-heavy sims), at the cost of more half-blurry textures in your field of view.
Blurry textures! Can't have that! :o ;)
Seriously though, I played with that in my experiments, but it felt like it didn't have a major impact on stability. I obviously couldn't see it due to tcmalloc, but since ...
Henri Beauchamp wrote:
In any case, I found a way to convince tcmalloc to release freed memory to the system and posted a new overlay package containing a patched tcmalloc version.
... I will revisit that once again.

Henri Beauchamp wrote:
You could see by yourself, by reducing the size of the viewer window.
I will, at some point. I'm not as "desperate" anymore as I was, since with the fonts avoided and your tweaks in 1.26.1.8, things are no longer in a really critical area. I watch the memory use now, and when I see it reaching dangerous levels, I either let people know I'm about to crash and/or I relog.

But I will certainly do a bit more experimentation here and there, if only to satisfy my own curiousity.

Love,
Lia
2011-09-25 04:47:58
Profile
Henri Beauchamp

Joined: 2009-03-17 18:42:51
Posts: 6043
Reply with quote
Post Re: 1.26.1.5 Crash out of nowhere
Quote:
I will, at some point. I'm not as "desperate" anymore as I was, since with the fonts avoided and your tweaks in 1.26.1.8, things are no longer in a really critical area. I watch the memory use now, and when I see it reaching dangerous levels, I either let people know I'm about to crash and/or I relog.
Well, now all you have to do is to TP away in a quieter area (a skybox is perfect), wait a minute or two for the textures to get unloaded so that the memory is freed, then TP back to your friends.
2011-09-25 08:37:19
Profile WWW
Henri Beauchamp

Joined: 2009-03-17 18:42:51
Posts: 6043
Reply with quote
Post Re: 1.26.1.5 Crash out of nowhere
Amalia Illios wrote:
Henri Beauchamp wrote:
Care to share your trick with us ? :)
It's not a particularly clever or practical one. I copied fonts.conf to stupid.conf in /etc/fonts, removed all the <dir>...</dir> entries and added the viewer's fonts directory as sole directory containing fonts.

Then I added this to the startup script:
Code:
## - Avoid loading all 3195 million fonts available on the system
export FONTCONFIG_FILE=/etc/fonts/stupid.conf
This works for me, and reduces the startup memory usage (i.e. what is used when it sits waiting for login) by just about 700something MByte (for 609 fonts, or so fc-list claims).
Hmm... I tried but for me it still tells me "35 system fonts loaded", with or without your trick... Perhaps it's because you have got a .fonts/ directory full of fonts in your home directory (~/.fonts/) ?... I don't have one, so only the basic system fonts are loaded by fontconfig.

Also, I used a custom font.conf (see below), local to the viewer installation (I put it in the fonts/ sub-directory of the viewer installation directory), removing *all* the <dir> ... </dir> entries (you don't need to add the viewer fonts dir either: LL's patched fontconfig, which is statically linked to the viewer, will automatically add them), and using the following lines in the "snowglobe" wrapper script:

Code:
## Custom font.conf to avoid loading a gazillion fonts from the system:
export FONTCONFIG_FILE=`pwd`"/fonts/font.conf"
Could you please try and tell me if it works for you ?... If it does, I could make this work around part of the next releases. ;)


fonts/font.conf file:

Code:
<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
   <match target="pattern">
      <test qual="any" name="family">
         <string>mono</string>
      </test>
      <edit name="family" mode="assign">
         <string>monospace</string>
      </edit>
   </match>

<!--
  Accept alternate 'sans serif' spelling, replacing it with 'sans-serif'
-->
   <match target="pattern">
      <test qual="any" name="family">
         <string>sans serif</string>
      </test>
      <edit name="family" mode="assign">
         <string>sans-serif</string>
      </edit>
   </match>

<!--
  Accept deprecated 'sans' alias, replacing it with 'sans-serif'
-->
   <match target="pattern">
      <test qual="any" name="family">
         <string>sans</string>
      </test>
      <edit name="family" mode="assign">
         <string>sans-serif</string>
      </edit>
   </match>

<!--
  These are the default Unicode chars that are expected to be blank
  in fonts.  All other blank chars are assumed to be broken and
  won't appear in the resulting charsets
 -->
      <blank>
         <int>0x0020</int>   <!-- SPACE -->
         <int>0x00A0</int>   <!-- NO-BREAK SPACE -->
         <int>0x00AD</int>   <!-- SOFT HYPHEN -->
         <int>0x034F</int>   <!-- COMBINING GRAPHEME JOINER -->
         <int>0x0600</int>   <!-- ARABIC NUMBER SIGN -->
         <int>0x0601</int>   <!-- ARABIC SIGN SANAH -->
         <int>0x0602</int>   <!-- ARABIC FOOTNOTE MARKER -->
         <int>0x0603</int>   <!-- ARABIC SIGN SAFHA -->
         <int>0x06DD</int>   <!-- ARABIC END OF AYAH -->
         <int>0x070F</int>   <!-- SYRIAC ABBREVIATION MARK -->
         <int>0x115F</int>   <!-- HANGUL CHOSEONG FILLER -->
         <int>0x1160</int>   <!-- HANGUL JUNGSEONG FILLER -->
         <int>0x1680</int>   <!-- OGHAM SPACE MARK -->
         <int>0x17B4</int>   <!-- KHMER VOWEL INHERENT AQ -->
         <int>0x17B5</int>   <!-- KHMER VOWEL INHERENT AA -->
         <int>0x180E</int>   <!-- MONGOLIAN VOWEL SEPARATOR -->
         <int>0x2000</int>   <!-- EN QUAD -->
         <int>0x2001</int>   <!-- EM QUAD -->
         <int>0x2002</int>   <!-- EN SPACE -->
         <int>0x2003</int>   <!-- EM SPACE -->
         <int>0x2004</int>   <!-- THREE-PER-EM SPACE -->
         <int>0x2005</int>   <!-- FOUR-PER-EM SPACE -->
         <int>0x2006</int>   <!-- SIX-PER-EM SPACE -->
         <int>0x2007</int>   <!-- FIGURE SPACE -->
         <int>0x2008</int>   <!-- PUNCTUATION SPACE -->
         <int>0x2009</int>   <!-- THIN SPACE -->
         <int>0x200A</int>   <!-- HAIR SPACE -->
         <int>0x200B</int>   <!-- ZERO WIDTH SPACE -->
         <int>0x200C</int>   <!-- ZERO WIDTH NON-JOINER -->
         <int>0x200D</int>   <!-- ZERO WIDTH JOINER -->
         <int>0x200E</int>   <!-- LEFT-TO-RIGHT MARK -->
         <int>0x200F</int>   <!-- RIGHT-TO-LEFT MARK -->
         <int>0x2028</int>   <!-- LINE SEPARATOR -->
         <int>0x2029</int>   <!-- PARAGRAPH SEPARATOR -->
         <int>0x202A</int>   <!-- LEFT-TO-RIGHT EMBEDDING -->
         <int>0x202B</int>   <!-- RIGHT-TO-LEFT EMBEDDING -->
         <int>0x202C</int>   <!-- POP DIRECTIONAL FORMATTING -->
         <int>0x202D</int>   <!-- LEFT-TO-RIGHT OVERRIDE -->
         <int>0x202E</int>   <!-- RIGHT-TO-LEFT OVERRIDE -->
         <int>0x202F</int>   <!-- NARROW NO-BREAK SPACE -->
         <int>0x205F</int>   <!-- MEDIUM MATHEMATICAL SPACE -->
         <int>0x2060</int>   <!-- WORD JOINER -->
         <int>0x2061</int>   <!-- FUNCTION APPLICATION -->
         <int>0x2062</int>   <!-- INVISIBLE TIMES -->
         <int>0x2063</int>   <!-- INVISIBLE SEPARATOR -->
         <int>0x206A</int>   <!-- INHIBIT SYMMETRIC SWAPPING -->
         <int>0x206B</int>   <!-- ACTIVATE SYMMETRIC SWAPPING -->
         <int>0x206C</int>   <!-- INHIBIT ARABIC FORM SHAPING -->
         <int>0x206D</int>   <!-- ACTIVATE ARABIC FORM SHAPING -->
         <int>0x206E</int>   <!-- NATIONAL DIGIT SHAPES -->
         <int>0x206F</int>   <!-- NOMINAL DIGIT SHAPES -->
         <int>0x2800</int>   <!-- BRAILLE PATTERN BLANK -->
         <int>0x3000</int>   <!-- IDEOGRAPHIC SPACE -->
         <int>0x3164</int>   <!-- HANGUL FILLER -->
         <int>0xFEFF</int>   <!-- ZERO WIDTH NO-BREAK SPACE -->
         <int>0xFFA0</int>   <!-- HALFWIDTH HANGUL FILLER -->
         <int>0xFFF9</int>   <!-- INTERLINEAR ANNOTATION ANCHOR -->
         <int>0xFFFA</int>   <!-- INTERLINEAR ANNOTATION SEPARATOR -->
         <int>0xFFFB</int>   <!-- INTERLINEAR ANNOTATION TERMINATOR -->
      </blank>
<!--
  Rescan configuration every 30 seconds when FcFontSetList is called
 -->
      <rescan>
         <int>30</int>
      </rescan>
   </config>

</fontconfig>
2011-09-26 09:56:31
Profile WWW
Amalia Illios

Joined: 2010-04-07 08:23:18
Posts: 215
Reply with quote
Post Re: 1.26.1.5 Crash out of nowhere
Henri Beauchamp wrote:
Hmm... I tried but for me it still tells me "35 system fonts loaded", with or without your trick... Perhaps it's because you have got a .fonts/ directory full of fonts in your home directory (~/.fonts/) ?... I don't have one, so only the basic system fonts are loaded by fontconfig.
No, no ~/.fonts, all my stuff is installed in the system font directories. Apart from that, the log probably isn't telling the whole truth. It tells me something about 76 fonts. /proc/<pid>/maps begs to differ. :roll:
Henri Beauchamp wrote:
Also, I used a custom font.conf (see below), local to the viewer installation (I put it in the fonts/ sub-directory of the viewer installation directory), removing *all* the <dir> ... </dir> entries (you don't need to add the viewer fonts dir either: LL's patched fontconfig, which is statically linked to the viewer, will automatically add them), and using the following lines in the "snowglobe" wrapper script:

Code:
## Custom font.conf to avoid loading a gazillion fonts from the system:
export FONTCONFIG_FILE=`pwd`"/fonts/font.conf"
Could you please try and tell me if it works for you ?... If it does, I could make this work around part of the next releases. ;)

Not working. See below why. ;)

Depending on where you want to put this, using pwd might of course yield the wrong result, since the cd is way below all the variable declarations. Then again, who am I to tell you that? :oops:

As I mentioned, it didn't work. Took me a while to figure it out, but you deleted too much stuff, resulting in an invalid XML, which is subsequently ignored. Inserting a <config> above the comment for the Unicode character declarations fixed that. After that, same result as with my first try, with no directories included. Less virtual used, but only little boxes on the HTML start page. Rest working fine. With viewer font directory added, all is peachy!

This might be because I've been very bad and messed with the font settings of the viewer anyway to have it use my preferred font (I know, I probably deserve a spanking for that :oops: ), then again maybe not. It's late, I'm cranky, and so that experiment will have to wait. :twisted:

Nonetheless, thank you for having another look into this.

Love,
Lia
2011-09-26 20:30:07
Profile
Henri Beauchamp

Joined: 2009-03-17 18:42:51
Posts: 6043
Reply with quote
Post Re: 1.26.1.5 Crash out of nowhere
Amalia Illios wrote:
Depending on where you want to put this, using pwd might of course yield the wrong result, since the cd is way below all the variable declarations. Then again, who am I to tell you that? :oops:
I thought it was rather obvious that the export should be placed after the 'cd "${RUN_PATH}"' line... But I should have specified it.

Quote:
As I mentioned, it didn't work. Took me a while to figure it out, but you deleted too much stuff, resulting in an invalid XML, which is subsequently ignored. Inserting a <config> above the comment for the Unicode character declarations fixed that.
You are right... If only the guy who wrote the original XML had any clue about indenting... Anyway, even once the file is fixed, I see no difference here (still 35 system fonts loaded, plus the viewer fonts).

Quote:
After that, same result as with my first try, with no directories included. Less virtual used, but only little boxes on the HTML start page.
Not seeing that here...

Quote:
Rest working fine. With viewer font directory added, all is peachy!
Well, in any case it doesn't look like a very portable trick (probably because of distro-patched fontconfig), so I guess I'll not use it... Thanks for your contribution nonetheless.
2011-09-26 21:16:48
Profile WWW
Amalia Illios

Joined: 2010-04-07 08:23:18
Posts: 215
Reply with quote
Post Re: 1.26.1.5 Crash out of nowhere
Henri Beauchamp wrote:
Anyway, even once the file is fixed, I see no difference here (still 35 system fonts loaded, plus the viewer fonts).
Now that is bizarre. With no directories specified and the setup proposed by you (or mine, doesn't make a difference -- as it should), I get
Code:
2011-09-27T16:05:25Z INFO: getDynamicFallbackFontList: Getting system font list from FontConfig...
2011-09-27T16:05:25Z INFO: getDynamicFallbackFontList: Language de
2011-09-27T16:05:25Z INFO: getDynamicFallbackFontList: Location DE
2011-09-27T16:05:25Z INFO: getDynamicFallbackFontList: Variant UTF-8
2011-09-27T16:05:25Z INFO: getDynamicFallbackFontList: Preferring fonts of language: de
2011-09-27T16:05:25Z INFO: getDynamicFallbackFontList: Using 0 system font(s).
2011-09-27T16:05:25Z INFO: dump: LLFontRegistry dump:
2011-09-27T16:05:25Z INFO: dump: Size: Huge => 16
2011-09-27T16:05:25Z INFO: dump: Size: Large => 12
2011-09-27T16:05:25Z INFO: dump: Size: Medium => 10
2011-09-27T16:05:25Z INFO: dump: Size: Monospace => 9
2011-09-27T16:05:25Z INFO: dump: Size: Small => 9
2011-09-27T16:05:25Z INFO: dump: Font: name=DejaVu style=[0] size=[TEMPLATE] fileNames=
2011-09-27T16:05:25Z INFO: dump:   file: DejaVuSans.ttf
2011-09-27T16:05:25Z INFO: dump: Font: name=DejaVu style=[1] size=[TEMPLATE] fileNames=
2011-09-27T16:05:25Z INFO: dump:   file: DejaVuSansBold.ttf
2011-09-27T16:05:25Z INFO: dump: Font: name=DejaVu style=[2] size=[TEMPLATE] fileNames=
2011-09-27T16:05:25Z INFO: dump:   file: DejaVuSansOblique.ttf
2011-09-27T16:05:25Z INFO: dump: Font: name=DejaVu style=[3] size=[TEMPLATE] fileNames=
2011-09-27T16:05:25Z INFO: dump:   file: DejaVuSansBoldOblique.ttf
2011-09-27T16:05:25Z INFO: dump: Font: name=Helvetica style=[0] size=[TEMPLATE] fileNames=
2011-09-27T16:05:25Z INFO: dump:   file: arial.ttf
2011-09-27T16:05:25Z INFO: dump: Font: name=Helvetica style=[1] size=[TEMPLATE] fileNames=
2011-09-27T16:05:25Z INFO: dump:   file: arialbd.ttf
2011-09-27T16:05:25Z INFO: dump: Font: name=Helvetica style=[2] size=[TEMPLATE] fileNames=
2011-09-27T16:05:25Z INFO: dump:   file: ariali.ttf
2011-09-27T16:05:25Z INFO: dump: Font: name=Helvetica style=[3] size=[TEMPLATE] fileNames=
2011-09-27T16:05:25Z INFO: dump:   file: arialbi.ttf
2011-09-27T16:05:25Z INFO: dump: Font: name=Monospace style=[0] size=[TEMPLATE] fileNames=
2011-09-27T16:05:25Z INFO: dump:   file: DejaVuSansMono.ttf
2011-09-27T16:05:25Z INFO: dump: Font: name=OverrideTest style=[0] size=[TEMPLATE] fileNames=
2011-09-27T16:05:25Z INFO: dump:   file: times.ttf
2011-09-27T16:05:25Z INFO: dump: Font: name=SansSerif style=[0] size=[TEMPLATE] fileNames=
2011-09-27T16:05:25Z INFO: dump:   file: /usr/share/fonts/TTF/Arialn.ttf
2011-09-27T16:05:25Z INFO: dump: Font: name=SansSerif style=[1] size=[TEMPLATE] fileNames=
2011-09-27T16:05:25Z INFO: dump:   file: /usr/share/fonts/TTF/Arialnb.ttf
2011-09-27T16:05:25Z INFO: dump: Font: name=default style=[0] size=[TEMPLATE] fileNames=
2011-09-27T16:05:25Z INFO: dump:   file: /usr/share/fonts/TTF/Arialn.ttf
2011-09-27T16:05:25Z INFO: createFont: createFont SansSerif size Small style 0
2011-09-27T16:05:25Z INFO: createFont: createFont SansSerif size Medium style 0
2011-09-27T16:05:25Z INFO: createFont: createFont SansSerif size Large style 0
2011-09-27T16:05:25Z INFO: createFont: createFont SansSerif size Huge style 0
2011-09-27T16:05:25Z INFO: createFont: createFont SansSerif size Medium style 1
2011-09-27T16:05:25Z INFO: createFont: createFont Monospace size Monospace style 0
DejaVu and so on obviously not found, since there's no font paths -- the Arial Narrow is found due to the fact that I put it into skins/default/xui/en-us/fonts.xml with a full path. So understandable that the HTML content only gets the little blocks -- 0 system fonts for me.

Henri Beauchamp wrote:
Well, in any case it doesn't look like a very portable trick (probably because of distro-patched fontconfig), so I guess I'll not use it...
Certainly seems that way. Well, as I initially said, at least it's now documented and maybe someone else with a similar issue on Ubuntu 10.04 can profit from it.

Henri Beauchamp wrote:
Thanks for your contribution nonetheless.
I'm always up for help with what little I can achieve. I've essentially stuck with Cool almost since I joined SL, and it has served me so well over the years, it's the least I can do!

Love,
Lia
2011-09-27 16:29:44
Profile
Display posts from previous:  Sort by  
Reply to topic   Page 3 of 3
  [ 29 posts ]  Go to page Previous  1, 2, 3

Board index » Bug reports

All times are UTC


Who is online

Users browsing this forum: No registered users and 2 guests

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group
Designed by ST Software.