No auto-reply does not work like this. You must first un-check "Open new IM session as soon as someone starts typing to you" (and keep it disabled), close any existing IM session, then have them IMing you. The auto-reply is sent when a new message is received
and no session (i.e. no corresponding IM tab) currently exists with the person IMing you.
I don't use this early session opening feature myself, but could confirm here that when enabled it breaks auto-reply (since the session gets opened, and when the first IM arrives, the code sees a session already exists, and consequently does not trigger auto-reply). A fix is ready for next release.