Cool VL Viewer forum http://sldev.free.fr/forum/ |
|
IO Completion Port based LLLFSThread for windows & Linux http://sldev.free.fr/forum/viewtopic.php?f=10&t=2222 |
Page 1 of 5 |
Author: | kathrine [ 2021-09-19 15:46:39 ] | ||
Post subject: | IO Completion Port based LLLFSThread for windows & Linux | ||
Similar transformation as the texture processing thread patch. Pushing the actual file reads to a thread pool (Kernel pool) via the Windows IOCP functions and fetching the completions later. A very similar patch can be done for Linux io_uring, just needs switching a few functions and structures around (see https://unixism.net/loti/tutorial/register_eventfd.html for the basic idea). (on my list of todos, when i get my Linux build environ going again). This makes a bit better use of the huge command queues on modern SSDs/systems. Just let the OS figure out the best I/O sequence and do not bother to block the thread with a blocking read()/write() call. It seems to work nicely, been running the viewer with the patch for a few days now without issues, and it feels a little more snappy when having a large draw distance and a lot of texture loading.
|
Author: | Henri Beauchamp [ 2021-09-19 17:53:37 ] |
Post subject: | Re: IO Completion Port based LLLFSThread for windows |
Many thanks ! I'll have a look, though probably not before a couple of weeks, due to RL constraints next week... With some luck, by that time, you'll have a Linux version ready... |
Author: | kathrine [ 2021-09-29 23:43:26 ] | ||
Post subject: | Re: IO Completion Port based LLLFSThread | ||
I added a Linux io_uring based version in addition to the Windows IOCP based one. The Linux version needs at least Kernel version 5.6 and liburing lib and headers installed, the Windows Version should work with any NT based version. The patch is pretty basic, but seems to work without crashes or bigger oddities i noticed. It felt a little snappier, but thats probably subjective. Currently it handles results right away, it might be better to first queue all pending events before processing the results in batches, but that would have needed a bit more work with the QueuedThread baseclass that i didn't want to do (yet).
|
Author: | ZaneZimer [ 2021-09-30 13:46:49 ] | |||||||||
Post subject: | Re: IO Completion Port based LLLFSThread | |||||||||
|
Author: | Henri Beauchamp [ 2021-09-30 16:14:35 ] |
Post subject: | Re: IO Completion Port based LLLFSThread for windows & Linux |
I'm back and could have a try at io_uring under Linux (the Windows build compiles fine but is untested so far). Beside adding an UseIOUring debug setting to allow (de)activating it on demand (after a restart), I also added a runtime Linux kernel version detection (so to deactivate it automatically when running on kernels older than v5.6). However, as is, the code crashed almost immediately on login for me. I already spotted and fixed the following issues/bugs:
Despite the above fixes (which seem to have solved the crashes), I'm still getting issues with sound file decoding ("cannot write file", despite the O_CREAT in the open() call)... Perhaps because I am using links to a ramdisk directory for cache directories... |
Author: | kathrine [ 2021-09-30 16:50:31 ] |
Post subject: | Re: IO Completion Port based LLLFSThread for windows & Linux |
Hi Henri, thx for taking a look. Never had a crash with it when trying (but only merged the linux/windows parts with ifdefs pretty late, so might have missed some parts there), but well, your fixes are obviously necessary and my oversights. Probably worked because i had a populated cache. I noticed some sound issues after looking closer, yes. The background sounds like wind etc. are there, but other sounds and UI sounds seem to fail. I'll see if i can spot the reason for the sound problems. Kathrine |
Author: | Henri Beauchamp [ 2021-09-30 17:55:10 ] | ||||||||||||||||||
Post subject: | Re: IO Completion Port based LLLFSThread for windows & Linux | ||||||||||||||||||
EDIT: in the Linux version, when mOffset is negative, seeking to the end of file is missing for reads and writes. Here is a fixed patch (hopefully, I did not break the Windows code when simplifying/factorizing it), that sadly still not resolves the sound issue:
|
Author: | kathrine [ 2021-09-30 21:31:08 ] |
Post subject: | Re: IO Completion Port based LLLFSThread for windows & Linux |
Thanks, will give it a try. I think i might have an idea what happens with the Sound. The llAudioDecodeMgr also uses the global LLFSThread to write its decoded files. It seems to manage to write one file (the click sound when opening a floater...), and than hangs and the LFS Thread never finishes the request for some reason (it gets a warning on exit that requests are pending in the LFS QueuedThread during shutdown). Will try to debug into it, to see what happens. |
Author: | kathrine [ 2021-09-30 21:35:26 ] | |||||||||
Post subject: | Re: IO Completion Port based LLLFSThread for windows & Linux | |||||||||
Oh, the mOffset stuff is kind of weird for READ... The old code does an llassert(mOffsett >= 0) and then goes on to handle the case that mOffset <0 for seeking from the end. So that should basically never happen.
|
Author: | Henri Beauchamp [ 2021-09-30 21:37:33 ] | |||||||||
Post subject: | Re: IO Completion Port based LLLFSThread for windows & Linux | |||||||||
I just noticed a bug in my patch: I inverted the QueuedRequest flags for sUseIoUring. It should read:
|
Page 1 of 5 | All times are UTC |
Powered by phpBB® Forum Software © phpBB Group https://www.phpbb.com/ |