Cool VL Viewer forum

View unanswered posts | View active topics It is currently 2021-11-30 00:32:26



Reply to topic  [ 46 posts ]  Go to page 1, 2, 3, 4, 5  Next
IO Completion Port based LLLFSThread for windows & Linux 
Author Message

Joined: 2011-10-07 10:39:20
Posts: 141
Reply with quote
Similar transformation as the texture processing thread patch.

Pushing the actual file reads to a thread pool (Kernel pool) via the Windows IOCP functions and fetching the completions later.
A very similar patch can be done for Linux io_uring, just needs switching a few functions and structures around (see https://unixism.net/loti/tutorial/register_eventfd.html for the basic idea).
(on my list of todos, when i get my Linux build environ going again).

This makes a bit better use of the huge command queues on modern SSDs/systems. Just let the OS figure out the best I/O sequence and do not bother to block the thread with a blocking read()/write() call.

It seems to work nicely, been running the viewer with the patch for a few days now without issues, and it feels a little more snappy when having a large draw distance and a lot of texture loading.


Attachments:
iocp.zip [2.61 KiB]
Downloaded 11 times


Last edited by kathrine on 2021-09-29 23:44:51, edited 1 time in total.

2021-09-19 15:46:39
Profile

Joined: 2009-03-17 18:42:51
Posts: 4773
Reply with quote
Many thanks !

I'll have a look, though probably not before a couple of weeks, due to RL constraints next week... With some luck, by that time, you'll have a Linux version ready... :D


2021-09-19 17:53:37
Profile WWW

Joined: 2011-10-07 10:39:20
Posts: 141
Reply with quote
I added a Linux io_uring based version in addition to the Windows IOCP based one.

The Linux version needs at least Kernel version 5.6 and liburing lib and headers installed, the Windows Version should work with any NT based version.

The patch is pretty basic, but seems to work without crashes or bigger oddities i noticed. It felt a little snappier, but thats probably subjective.
Currently it handles results right away, it might be better to first queue all pending events before processing the results in batches, but that would have needed a bit more work with the QueuedThread baseclass that i didn't want to do (yet).


Attachments:
async_io_lllfsthread.patch.gz [4.72 KiB]
Downloaded 9 times
2021-09-29 23:43:26
Profile

Joined: 2016-06-19 21:33:37
Posts: 238
Location: San Francisco bay area, CA, USA
Reply with quote
kathrine wrote:
It felt a little snappier, but thats probably subjective.
I have applied and been testing this patch. For me, it does seem to improve scene loading immediately on login/teleport/region crossing, even though I'm also using a ramdisk for the viewer cache. Thanks for continuing to investigate improvements for modern hardware/OSes, Kathrine.


2021-09-30 13:46:49
Profile

Joined: 2009-03-17 18:42:51
Posts: 4773
Reply with quote
I'm back and could have a try at io_uring under Linux (the Windows build compiles fine but is untested so far).

Beside adding an UseIOUring debug setting to allow (de)activating it on demand (after a restart), I also added a runtime Linux kernel version detection (so to deactivate it automatically when running on kernels older than v5.6).

However, as is, the code crashed almost immediately on login for me.

I already spotted and fixed the following issues/bugs:
  • The processNextRequest() method should be made virtual in the LLQueuedThread class and declared as an override in the LLLFSThread class.
  • For liburing/Linux, the opening for writing should also have the O_CREAT flag added ("int outfile = open(mFileName.c_str(), O_CREAT | O_WRONLY);"), else the viewer cannot create cache files.
  • For liburing/Linux, in LLLFSThread::processNextRequest(), cqe and req must be checked against NULL return values and cause the while() loop exit when NULL.
  • For both Linux and Windows, when hitting the "Unable to do async read/write" warning, an exit from the method should be done with a failed read/write condition (mBytesRead = 0; return true;).

Despite the above fixes (which seem to have solved the crashes), I'm still getting issues with sound file decoding ("cannot write file", despite the O_CREAT in the open() call)... Perhaps because I am using links to a ramdisk directory for cache directories...


2021-09-30 16:14:35
Profile WWW

Joined: 2011-10-07 10:39:20
Posts: 141
Reply with quote
Hi Henri,

thx for taking a look. Never had a crash with it when trying (but only merged the linux/windows parts with ifdefs pretty late, so might have missed some parts there), but well, your fixes are obviously necessary and my oversights. Probably worked because i had a populated cache.

I noticed some sound issues after looking closer, yes. The background sounds like wind etc. are there, but other sounds and UI sounds seem to fail.

I'll see if i can spot the reason for the sound problems.

Kathrine


2021-09-30 16:50:31
Profile

Joined: 2009-03-17 18:42:51
Posts: 4773
Reply with quote
kathrine wrote:
Hi Henri,I noticed some sound issues after looking closer, yes.
This might be due to the fact you are using low level file operations with open() while LLLFSThread is normally using APR files... If the file gets opened for read via APR before LLLFSThread attempts to write to it and opens it with open(), there might be a file lock preventing it to do so. Just guess work from my part, since I did not investigate properly so far.

EDIT: in the Linux version, when mOffset is negative, seeking to the end of file is missing for reads and writes. Here is a fixed patch (hopefully, I did not break the Windows code when simplifying/factorizing it), that sadly still not resolves the sound issue:
Attachment:
io_uring-v3.patch.gz [8.64 KiB]
Downloaded 8 times


2021-09-30 17:55:10
Profile WWW

Joined: 2011-10-07 10:39:20
Posts: 141
Reply with quote
Thanks, will give it a try.

I think i might have an idea what happens with the Sound.

The llAudioDecodeMgr also uses the global LLFSThread to write its decoded files.

It seems to manage to write one file (the click sound when opening a floater...), and than hangs and the LFS Thread never finishes the request for some reason (it gets a warning on exit that requests are pending in the LFS QueuedThread during shutdown). Will try to debug into it, to see what happens.


2021-09-30 21:31:08
Profile

Joined: 2011-10-07 10:39:20
Posts: 141
Reply with quote
Oh,

the mOffset stuff is kind of weird for READ...

The old code does an llassert(mOffsett >= 0) and then goes on to handle the case that mOffset <0 for seeking from the end. So that should basically never happen.

Code:
           llassert(mOffset >= 0);
      LLAPRFile infile;   // auto-closes
      infile.open(mFileName, LL_APR_RB, mThread->getLocalAPRFilePool());
      if (!infile.getFileHandle())
      {
         llwarns << "Unable to read file: " << mFileName << llendl;
         mBytesRead = 0; // fail
         return true;
      }
      S32 off;
      if (mOffset < 0)
      {
         off = infile.seek(APR_END, 0);
      }
      else
      {
         off = infile.seek(APR_SET, mOffset);
      }


2021-09-30 21:35:26
Profile

Joined: 2009-03-17 18:42:51
Posts: 4773
Reply with quote
I just noticed a bug in my patch: I inverted the QueuedRequest flags for sUseIoUring. It should read:
Code:
LLLFSThread::Request::Request(LLLFSThread* thread,
                       handle_t handle, U32 priority,
                       operation_t op, const std::string& filename,
                       U8* buffer, S32 offset, S32 numbytes,
                       Responder* responder)
:   QueuedRequest(handle, priority,
              sUseIoUring ? FLAG_AUTO_COMPLETE | FLAG_ASYNC
                       : FLAG_AUTO_COMPLETE),


2021-09-30 21:37:33
Profile WWW
Display posts from previous:  Sort by  
Reply to topic   [ 46 posts ]  Go to page 1, 2, 3, 4, 5  Next

Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group
Designed by ST Software.