Cool VL Viewer forum

Any idea why it's just Cool VL Viewer that has (or hopefully "had") this issue?

I doubt very much it's just the Cool VL Viewer. Any viewer using FMOD Ex is affected by FMOD Ex bugs (and this *is* a FMOD Ex bug, like proved the log messages I added and the logs that were posted here, the messages showing without possible doubt that the freeze occurs inside the release call to FMOD Ex).

This said, I always adopted and updated to the very latest FMOD Ex version for the Cool VL Viewer, while other viewers are still using old versions (that have bugs too, but perhaps not that one), so there may be different behaviours with different viewers.

Also, I *never* encountered neither could reproduce this bug myself, and only a couple of users seem affected, so it's a pretty rare occurrence which might have not been spotted (or might have been mistaken for another issue, such as a graphics driver freeze) with other viewers so far.

Finally, it only affects Linux viewers, which constitute a very small proportion (a few percents) of the viewers in use.

The problem has been two-fold, both a bug in FMOD Ex itself (which was fixed in 4.44.53 after i filed a bug report to FMOD) as well as the viewer stopping and releasing streams improperly according to the netstreaming example and documentation that comes with the API.

The explaination of how and why the viewers have been stopping and releasing streams the wrong way and how it is fixed can be viewed at https://bitbucket.org/GibsonFirehawk/op ... a1d2fa9430 . Although that work is originally part of my FMOD Studio contribution to LL, the principals are the same for Ex.

The stream release problem with viewers was not just on Linux, even though it was more commonly reported there. There was a deadlock that would happen on Mac as well that i saw many times myself that would randomly hit on releasing a stream. My changes to how streams are stopped and released by the viewer has eliminated that bug. The same kind of hang would probably also hit Linux users along with the other bug inside of FMOD Ex itself that was just fixed for the particular case of calling release on a starving netstream. A two-fold problem needing fixes in both FMOD Ex as well as in the viewer code itself to completely eliminate stream release deadlocks.

Firestorm has just integrated my viewer-side fix for how streams are stopped and released with FMOD Ex in their development repository along with updating to 4.44.53. Feel free to use that viewer-side code for FMOD Ex in Cool VL if you like, Henri.

I will not be using FMOD Studio, because OSS support was dropped in it, and I'm using OSS on all my Linux systems. Your contribution can't therefore be applied to the Cool VL Viewer.

See http://hg.phoenixviewer.com/phoenix-fir ... 8bbab1eb2a for FMOD Ex. That is what i was referring to, not suggesting you change to FMOD Studio.

I will have a closer look at your code (it's close to midnight now, here), but the Cool VL Viewer's code already got quite a few fixes added when compared to Firestorm's (or LL's) one, so most of your code changes seem moot to me, apart from that "wait for release" loop on viewer exit (but the Cool VL Viewer was never reported as suffering issues with Internet streams on viewer closure).

At first glance, I see quite a few bugs and potential issues in your code (for example, in LLStreamingAudio_FMODEX::delayedRelease(), where the "streams" boolean should only be set to true only when streamp->stopStream() failed, and not at every iteration of the for loop). Also, you should move the stream releasing loop on exit inside the destructor of LLStreamingAudio_FMODEX instead of having it in llappviewer.cpp... Technically, the destruction of the class could potentially happen anywhere else than in llappviewer.cpp, and should still cause the delayed release of the streams.

There is something I disagree about too: in your code for LLAudioStreamManagerFMODEX::stopStream(), in which you release the stream only for the FMOD_OPENSTATE_READY and FMOD_OPENSTATE_ERROR (what about FMOD_OPENSTATE_[LOADING/BUFFERING/SEEKING/PLAYING ?) and don't even test for the actual result of release() (which is FMOD_OK on success) in those cases, assuming instead that release() always succeeds.

Finally, in your previous message, you said:
But the bug which this very thread is dealing with was exclusively seen in *some* Linux systems, and the changes log for the fixed FMOD Ex v4.44.53 does say the bug was indeed only a Linux related one.

Cool VL, like the other viewers, just calls pause, not stop on the stream, and then calls release right after if the stream is not in connecting state. Just pausing the stream does not make the stream exit playing state in most cases, and releasing a stream in playing state is exactly what FMOD warns against doing in their documentation in regards to stalling the main thread. Just changing pause to stop isn't enough to get the stream to exit playing state in most cases when calling release right after. I explained this problem thoroughly in my commit message at https://bitbucket.org/GibsonFirehawk/op ... a1d2fa9430 . For why there is a check for release on shutdown, see below.



In the end, it doesn't matter. True or false is irrelevant when delayedRelease is called during each update and the viewer is not coded to react differently to true or false during calls to delayedRelease via update. The only thing that happens differently ultimately with stream stop and release by having the boolean for every iteration for the loop instead of only if streamp->stopStream() failed is that on shutdown the viewer checks to see if there are any streams to release only one extra time.



The stream states other than FMOD_OPENSTATE_READY and FMOD_OPENSTATE_ERROR are covered by the default to false for close. When stop (not pause) is called on a stream, it will return very quickly to FMOD_OPENSTATE_READY if it was not a stream that was in error state. With my code, the first time openstate is checked before release, if the stream is not in ready or error state, the stream is pushed to the delayedRelease list and then is continuously checked for when ready to be released similar to how things were done with the dead streams list previously. This is how FMOD stops and releases streams in their netstream example program that comes with their API. They call stop on the stream and wait for the stream to return to FMOD_OPENSTATE_READY before calling release. The stream, if not in error state, will eventually leave whatever state it was in like buffering, loading, seeking, etc after stop was called and return on its own to ready state before release is called. That is also why i have the check for release during viewer shutdown. If a stream hasn't reached ready state yet and been released, the viewer waits for that to happen before calling release on the stream and shutting down.

As far as whether or not failure or success occurs during release is obvious by the viewer deadlocking or crashing or not when release is called. Just the fact that the viewer does not deadlock or crash and the viewer makes it to the command to write in the log "Successfully released internet stream" after release was called is enough to know that release succeeded. That said, in my opinion, checking for FMOD_OK on release is useless.

Maybe only in regards to just Cool VL and the reports you have seen in this forum makes you believe it is just a Linux only problem and just one specific hang bug that is completely fixed by the new FMOD Ex update. Just because you haven't been aware of stream stopping issues on other platforms and viewers too, or a different variant of a release hang, doesn't mean the problem doesn't exist. My code deals with the wrong way the viewers have been stopping and releasing streams that has been causing at least one variant of a stream release deadlock on release. With my code plus the FMOD Ex update, all of the Linux deadlocks on stream release should be gone for good. I brought these things to your attention because you had a long thread here with users that have experienced deadlocks on stream release on Linux. If you don't see the problem in how FMOD Ex viewers, including Cool VL, have been stopping and releasing streams, or for whatever else reason don't want my code, then fine, your choice.

Cool VL Viewer, please... "Viewer" is part of the name, just like "Tower" in "Eiffel Tower"...

Yes, this is the second difference that I didn't spot yesterday evening, but like I wrote, "I will have a closer look at your code (it's close to midnight now, here)"... Note that the pausing was probably initially done by choice and on-purpose by LL, because if you look at the code for LLStreamingAudio_FMODEX::pause(), it calls stop()... But in the same method, instead of un-pausing the stream on restart, they actually restart it from scratch... Plus, pausing a stream means it'd fill up the buffer quickly and end up stuttering on un-pausing anyway... so, yes, your method of stopping it altogether seems appropriate here.

Excepted that the "main thread" should never be at risk for FMOD_NONBLOCKING streams, which is what the viewer uses... any call done to stop/play/release/do-wahetever on non-blocking streams should return immediately, with either a success code (FMOD_OK), or an error, indicating the operation could not be achieved. Apparently however, FMOD Ex non-blocking calls have been plagued (till v4.44.53 was released) with dead lock bugs under Linux (and also under some other OSes with yet other bugs fixed in previous releases: see the FMOD Ex change log).

It matters for the sake of cleanliness of the code (and the removal of an unnecessary while() loop in your code, at the force-release stage).
But what matters most, is that the class properly implements its own cleanup code (at worst, with a specific cleanup() method, when the cleanup cannot happen in the class destructor itself) and that the latter doesn't get scattered though the consumer code of the application using the library the class pertains to, with undue access to low level implementation-dependent routines, pointers and structures... FYI, your cleanup code was brought down to only 5 lines of code in the destructor of the LLStreamingAudio_FMODEX class, in my implementation, with no change whatsoever on how the class is shut down in llappviewer.cpp.

Yes, with the stop() against pause difference, it makes sense now.

Again, the deadlock should never occur on a non-blocking stream... This was a bug in FMOD Ex. However you should still test for the returned error code of release() and keep delaying it when/should an error occur, till force-releasing time is reached: doing otherwise is no better than what was done before, on viewer quitting... Thinking release() will always succeed in the cases you allow for the stream states (FMOD_OPENSTATE_READY and FMOD_OPENSTATE_ERROR) is making assumptions on the FMOD Ex implementation and code correctness, which neither you or me can tell anything about, since the said code is closed source...

This very bug, dealt with in this very forum thread has been solved by that Linux-specific fix in FMOD Ex:
Period.

You indeed pin-pointed two potential issues, one of which (namely, the stream shut down on session quitting) is apparently not a problem for the Cool VL Viewer (but is still good to address, for code cleanliness and extra safety measure), and the other triggered a bug in FMOD Ex (the latter being supposedly fixed now, in v4.44.53). Your help is appreciated and your contribution is accepted (even if re-worked by me) and will be duly credited.
However, you should not get upset when someone (who, incidentally, has been programming for almost 4 decades and therefore got a little experience in coding) points out bugs in your code: the whole point of Open Source is to exchange code and learn from each others' code (and even mistakes in that code) so to produce even better code next time...

If you are curious about what your fixes look like for the Cool VL Viewer after being revisited by me, here's the diff:

FYI ...according to the FMOD developer i dealt with to fix the bug mentioned the release notes of 4.44.53, the problem was a socket recv hanging forever in the Linux version preventing release on a starving stream. When 4.44.53 was released, i tested for a while without my fixes and was able to reproduce a deadlock on release on a non-starving stream. Non-blocking calls aren't completely plague free yet in 4.44.53.

What you pointed out were not bugs in my code, but rather the way you would do things instead or improve. I wasn't upset about it, but maybe you should consider being a little more careful about what you call bugs in another person's work.

Glad i was able to be of help.

A function (delayedRelease()) returning true when it should return false (your code makes it return true as soon as it releases a stream, instead of only when there are still streams left to release) is a bug (which even caused you to use a pointless while() loop on force-release). Not checking for a failed release() (in stopStream()) and thus not retrying it when it should be, is also a bug.
They may not be fatal bugs, but they are bugs nonetheless, and calling them otherwise is... a very "buggy" reasoning... One thing is certain: I will never let such bugs slip into the Cool VL Viewer code.