Cool VL Viewer forum

View unanswered posts | View active topics It is currently 2021-12-01 20:56:18



Reply to topic  [ 46 posts ]  Go to page Previous  1, 2, 3, 4, 5
IO Completion Port based LLLFSThread for windows & Linux 
Author Message

Joined: 2011-10-07 10:39:20
Posts: 141
Reply with quote
Hi,

as you were worried about the idle spinning of the thread, i added blocking on the completions, when the request queue is empty:

The code adds some override for the LLThread::wake() method (which needs to be made virtual and added to the lllfsthread.h).

Now a wake() first calls the base implementation and wakes up the LLCondition of the thread and then (if the io_uring is initialized) queues a NOP/dummy completion with key = 1 on the blocking io_uring call (io_uring_wait_cqe() instead of peek) to wake it up. The threadedUpdate() blocks on the completion port, if no more requests are queued and just peeks (as before) when new requests are incoming.

I did not test the Linux part yet, but the Windows part seems to run nicely, as far as i could tell.

Code:
#if LL_IOURING
// Lets have some fun with overlapped I/O and io_uring ;-)
// (c)2021 Kathrine Jansma
// Requests map to OVERLAPPED ReadFile / WriteFile calls basically
// Responder => The OVERLAPPED Result gets delivered to the responder
// We pick up the completions later and have the OVERLAPPED result delivered to
// the responder. Completions MAY happen out of order !

//virtual
bool LLLFSThread::runCondition()
{
   return !mIdleThread || mPendingResults || !mRequestQueue.empty();
}

//virtual
void LLLFSThread::threadedUpdate()
{
   if (!sRingInitialized)
   {
      return;
   }

# if LL_LINUX
   struct io_uring_cqe* cqe;
# elif LL_WINDOWS
   DWORD bytes_processed = 0;
   LPOVERLAPPED result;
   DWORD_PTR key;
# endif

   // Dequeue finished results
   bool have_work = true;
   while (have_work)
   {
# if LL_LINUX
      // Check for finished cqe's without blocking; it might be more
      // efficient to use the batch variant.
      if (!mRequestQueue.empty()) {
         have_work = io_uring_peek_cqe(&mRing, &cqe) == 0;
      }
      else {
         // block, as we have no further requests queued
         have_work = io_uring_wait_cqe(&mRing, &cqe) == 0;
      }
      if (have_work)
      {
         llassert_always(mPendingResults > 0);
         --mPendingResults;
         if (!cqe)   // Paranoia ?
         {
            break;
         }
         S32 bytes_handled = cqe->res;

         LLLFSThread::Request* req =
            (LLLFSThread::Request*)io_uring_cqe_get_data(cqe);
         if (!req || req == 1)   // Paranoia or magic wakeup cqe
         {
            io_uring_cqe_seen(&mRing, cqe);
            break;
         }
         if (bytes_handled >= 0)
         {
            req->setBytesRead(bytes_handled);
         }
         else   // An I/O error occurred: mark as such.
         {
            req->setBytesRead(0);
         }
         // true = resquest is over and must auto-complete (self-destruct)
         setRequestResult(req, true);
         io_uring_cqe_seen(&mRing, cqe);
      }
# elif LL_WINDOWS
      DWORD timeout = INFINITE;
      if (!mRequestQueue.empty()) {
         timeout = 0;
      }
         
      have_work = GetQueuedCompletionStatus(mIOCPPort, &bytes_processed,
                                   &key, &result, timeout);
      if (key == 1) {
         // our magic wakeup package
         --mPendingResults;
         break;
      }
      LLLFSThread::Request* req = (LLLFSThread::Request*)key;
      if (have_work)
      {
         llassert_always(mPendingResults > 0);
         --mPendingResults;
         if (req)
         {
            req->setBytesRead(bytes_processed);
            setRequestResult(req, true);
         }
      }
      else if (GetLastError() == WAIT_TIMEOUT) {
         break;
      }
      else if (!result)
      {
         if (!req || GetLastError() != ERROR_ABANDONED_WAIT_0)
         {
            llwarns << "IOCP GetQueuedCompletionStatus failed with code: "
                  << GetLastError() << llendl;
         }
         else
         {
            llwarns << "IOCP for '" << req->getFilename()
                  << "' closed while waiting for completion status !"
                  << llendl;
         }
         break;
      }
      else
      {
         llassert_always(mPendingResults > 0);
         --mPendingResults;
         if (req)
         {
            llwarns << "IOCP operation for " << req->getFilename()
                  << " failed with code: " << GetLastError() << llendl;
            req->setBytesRead(0);   // An error occurred: mark as such.
            // true = resquest is over and must auto-complete
            // (self-destruct)
            setRequestResult(req, true);
            // We may have further IOCP results pending
            have_work = true;
         }
         
      }
# endif
   }
}

/* wake the thread and signal to waiting IOCP/io_uring*/
void LLLFSThread::wake()
{
   LLQueuedThread::wake();
   if (!sRingInitialized)
   {
      return;
   }
   // wakeup the completion ports if those are blocked
   ++mPendingResults;
#if   LL_WINDOWS
   PostQueuedCompletionStatus(mIOCPPort, 0, (ULONG_PTR)1, NULL);
#elif LL_LINUX
   struct io_uring_sqe* sqe = io_uring_get_sqe(&(mThread->mRing));
   io_uring_sqe_set_data(sqe, (void*)1);
   io_uring_prep_nop(sqe);
   io_uring_submit(&(mThread->mRing));
#endif
}
#endif   // LL_IOURING


2021-10-07 21:21:56
Profile

Joined: 2009-03-17 18:42:51
Posts: 4774
Reply with quote
Well, for Linux you need to change the code (beside "(intptr_1)req == 1" in the if clause, and using &mRing directly instead of &(mThread->mRing) in wake()) so that a "++mPendingResults;" is inserted before the "io_uring_sqe_set_data(sqe, (void*)1);" line in wake(), else you hit the llassert_always(mPendingResults > 0) and crash.

But even once the code fixed, and while it seems to be working while logged in, the viewer never shuts down completely on exit and I must kill it manually...

So, I'll stick with the "spinning thread version" for now... :P


2021-10-07 22:19:05
Profile WWW

Joined: 2016-06-19 21:33:37
Posts: 238
Location: San Francisco bay area, CA, USA
Reply with quote
While testing out the latest build (1.28.2.44), I noticed that the .dsf files do not get created with the proper file permissions in cache. I swear this was working before with the various builds and patches, but perhaps I was mistaken and/or did not have the feature properly enabled.

The cache entry yields entries like:
Code:
----------.  1 zanezimer zanezimer 574640 Oct 10 06:02 3643f41e-fc7b-d44c-120a-35a9aecb4de7.dsf
-rw-r--r--.  1 zanezimer zanezimer   1109 Oct 10 05:20 64e1d876-1854-4214-9c5d-753d7f1391f9.cached_mute
-rw-r--r--.  1 zanezimer zanezimer 943215 Oct 10 05:20 64e1d876-1854-4214-9c5d-753d7f1391f9_inv.llsd.gz
drwx------. 18 zanezimer zanezimer    360 Mar 13  2021 assets
-rw-r--r--.  1 zanezimer zanezimer     76 Oct 10 05:26 avatar_name_cache.xml
-rw-r--r--.  1 zanezimer zanezimer     81 Oct 10 05:20 experience_cache.xml
-rw-r--r--.  1 zanezimer zanezimer   8265 Oct 10 05:20 name.cache
drwx------.  2 zanezimer zanezimer    300 Oct  9 19:26 objectcache
drwx------. 18 zanezimer zanezimer    400 Oct  9 19:18 texturecache


Note the .dsf file has no permissions, which leads to these errors in the log:
Code:
2021-10-10 13:02:06Z WARNING: LLAudioBufferOpenAL::loadWAV: Error loading: /home/zanezimer/.secondlife/cache_coolvlviewer/3643f41e-fc7b-d44c-120a-35a9aecb4de7.dsf - I/O error


This is the case with the official or self-built versions. I have not tried this without a ram disc configuration, however, but reverting UseIOURing to false, allows .dsf files to be created as:
Code:
-rw-rw-r--.  1 zanezimer zanezimer 574640 Oct 10 06:13 3643f41e-fc7b-d44c-120a-35a9aecb4de7.dsf

My viewer info is:
Code:
Cool VL Viewer v1.28.2.44, 64 bits, Oct  9 2021 10:52:06
Release notes

You are at 308937.4, 235498.0, 62.2  in Bugle located at
simhost-06708fc7472a0215b.agni.secondlife.io (52.43.159.126:13025)
Alias: ec2-52-43-159-126.us-west-2.compute.amazonaws.com
Second Life Server 2021-10-01.564394
Release notes

CPU: AMD Ryzen 7 3700X 8-Core Processor (4087.95 MHz)
Memory: 64220MB
OS version: Linux-x86_64 v5.14.9-200.fc34.x86_64 #1 SMP Thu Sep 30 11:55:35 UTC 2021
Memory manager: jemalloc v5.2.1-20211008
Graphics card vendor: NVIDIA Corporation
Graphics card: NVIDIA GeForce GTX 1080 Ti/PCIe/SSE2
OpenGL version: 4.6.0 NVIDIA 470.74
Detected VRAM: 11264MB
J2C decoder: OpenJPEG: 1.4.0.635f
Audio driver: OpenAL v1.1 ALSOFT 1.19.1 (OpenAL Soft: OpenAL Soft)
Networking backend: libcurl/7.47.0 OpenSSL/1.0.2u zlib/1.2.11.zlib-ng
Embedded browser: Dullahan 1.12.3/CEF 94.4.9/Chromium 94.0.4606.71
Packets lost: 0/3154 (0.0%)

Built with: GCC v5.5.0
Compiler-generated maths: SSE2.

Compile flags used for this build:
-O3 -fno-delete-null-pointer-checks -fno-ipa-cp-clone -fno-align-labels -fno-align-loops -fsched-pressure -frename-registers -fweb -fira-hoist-pressure -DNDEBUG -std=c++14 -fno-stack-protector -fno-threadsafe-statics -fPIC -pipe -g -gdwarf-2 -gstrict-dwarf -fno-var-tracking-assignments -fexceptions -fno-strict-aliasing -fvisibility=hidden -fsigned-char -m64 -mfpmath=sse -fno-math-errno -fno-trapping-math -pthread -Wall -Wno-reorder -Werror -DLL_LINUX=1 -D_REENTRANT -DXML_STATIC -DLL_JEMALLOC=1 -DLL_PHMAP=1 -DLL_IOURING=1 -DLL_ELFBIN=1 -DOV_EXCLUDE_STATIC_CALLBACKS -DLL_FMOD=1 -DLL_OPENAL=1 -DLL_SDL2=1 -DLL_X11=1 -DLL_NDOF=1


2021-10-10 13:19:40
Profile

Joined: 2009-03-17 18:42:51
Posts: 4774
Reply with quote
ZaneZimer wrote:
I noticed that the .dsf files do not get created with the proper file permissions in cache.
Yep, that's obviously a bug (though, the fact io_uring does not set the default file permissions on creation under Linux is rather surprising). I did not notice it, because I'm always 'root'... This will definitely need fixing.


2021-10-10 14:33:18
Profile WWW

Joined: 2009-03-17 18:42:51
Posts: 4774
Reply with quote
In fact, the problem is not with io_uring but with the open() call for write operations: since O_CREAT is among the flags, it needs a supplementary parameter specifying the file permissions.

Simply replace line 512 in linden/indra/llfilesystem/lllfsthread.cpp with:
Code:
         S32 outfile = open(mFileName.c_str(), O_CREAT | O_WRONLY, S_IRUSR | S_IWUSR);


2021-10-10 17:17:26
Profile WWW

Joined: 2016-06-19 21:33:37
Posts: 238
Location: San Francisco bay area, CA, USA
Reply with quote
Henri Beauchamp wrote:
Simply replace line 512 in linden/indra/llfilesystem/lllfsthread.cpp with:
Code:
         S32 outfile = open(mFileName.c_str(), O_CREAT | O_WRONLY, S_IRUSR | S_IWUSR);
Thanks Henri. That's the line I was looking into but had no idea what parameters it might take or might be missing. I applied that change, rebuilt and now I get '-rw-------.' for the file perms. There are no, as expected, WARNINGS about I/O in the log now.


2021-10-10 18:23:52
Profile
Display posts from previous:  Sort by  
Reply to topic   [ 46 posts ]  Go to page Previous  1, 2, 3, 4, 5

Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group
Designed by ST Software.