Cool VL Viewer forum

View unanswered posts | View active topics It is currently 2020-02-17 23:59:55



Reply to topic  [ 2 posts ] 
The built-in sources preprocessor 
Author Message

Joined: 2009-03-17 18:42:51
Posts: 4160
Reply with quote
Starting with v1.26.22.36, the Cool VL Viewer implements sources preprocessing (akin to what can be done by a C preprocessor such as 'cpp').

The preprocessor is usable both with Lua script files loaded by the Cool VL Viewer and with LSL script assets (either in your inventory or in one of your in-world rezzed objects).

While not as complete (and complex) as a genuine C preprocessor (in particular, macros expansion is not supported), it provides a good range of features that should cover pretty much all common use cases.
Instead of using the (enormous) boost::wave library like some other TPVs have done, I wrote it from scratch, resulting in a compiled code that is about 20 times smaller and much faster. It uses Lua to evaluate expressions following #if directives, but with (mostly) C-compatible operators.

Just like for cpp, the directives are recognized by their '#' first character, that must also be the first non-spacing character in the line (i.e. spaces and tabs are ignored). The first word that follows the '#' (again, spacing is ignored between '#' and that word) is the preprocessor directive name.
Example of valid preprocessor directives:
Code:
#define SWITCH 0
#if SWITCH
    #   include   "somefile.lua"
#else
    #   warning   "SWITCH is 0"
#endif


Here are the available preprocessor directives:

  • #include "include_name" (or #include <include_name>): causes the preprocessor to load and include/merge the contents of 'include_name' in the sources.
    For Lua, the 'include_name' must be a file present in either users_settings/include/ or avatar_name[@grid]/include/ (the latter taking the precedence over the former, if both contain a file named 'include_name'). However, you may choose another default include directory via a special #pragma directive (see below); if the file is not found in that directory, the preprocessor will still fallback to the per-account or user settings sub-directories.
    For LSL and by default (i.e. unless a special #pragma is used), the 'include_name' represents an inventory script asset, that must be present in either the same inventory folder as the script asset being preprocessed (this, of course, cannot apply to in-world rezzed objects scripts) or in the 'Scripts' folder of your inventory. However, you may choose another default include inventory folder or even to include a file from your OS file system instead via a special #pragma directive (see below).
    Note that, to avoid infinite loops, a file/asset can be included only once (the preprocessor keeps track of what files/assets have been included and would ignore #include directives that would attempt to include those files a second time); this also means you do not need to add include guard directives (such as #ifndef INCLUDE_NAME/#define INCLUDE_NAME/<sources proper>/#endif) in your include files/assets. Included files/assets can themselves #include other files/assets (there is no limit on the number of nested #includes).

  • #define TOKEN [value]: allows to replace all further occurrences of 'TOKEN' in the sources with 'value' (or an empty string when value is omitted). 'value' may be anything, from a number, to a string litteral (enclosed in any string delimiter valid for the language of the sources), to a function name, or a complex expression. 'value' is not evaluated by the preprocessor itself, unless its associated TOKEN is used later in a #if clause. A valid TOKEN must be a single word, starting with a (non-accented) letter (a-z and A-Z) or an underline ('_') character, and containing only letters, digits (0-9) and underline characters.
    Note: you cannot redefine an already #defined token: you must first #undef it or you will get an error.

  • #undef TOKEN: forgets any previous #definition of 'TOKEN'. If 'TOKEN' was not already defined, this is ignored.

  • #ifdef TOKEN|#ifndef TOKEN|#if <expression>/[#elif <expression2>/[#elif <expression3>]...]/[#else]/#endif: this construct is fully supported, with also the special 'defined()' "function" accepted in #[el]if expressions (so that you can both test for a token definition and for some other condition in the same #[el]if). Any defined tokens present in <expression> are both substitued and their value evaluated when the #[el]if condition is tested by the preprocessor. #if* clauses can be nested (there is no limit on the number of nested clauses).

  • #pragma: this directive accepts arguments that allow to affect how the preprocessor works (in C-preprocessors, it is usually affecting options passed to the compiler). Since the built-in preprocessor cannot be passed options (like you could do for a stand-alone preprocessor program, on its command line), #pragma offers that opportunity instead, with the following recognized arguments:
    • '#pragma preprocessor-off' and '#pragma preprocessor-on' allow to (respectively) suspend and resume the preprocessing within a file/asset being preprocessed (or included). See in the limitations chapter below for usage.
    • '#pragma include-from: some_folder' allows to specify a folder to look into for the #include directives that follow it (important: 'some_folder' shall appear without any quotes).
      For Lua sources, 'some_folder' may be a sub-folder of one of the user settings or settings per account (users_settings/ or avatar_name[@grid]/) directories or, when 'some_folder' starts with "~/", of a sub-folder of the "home" directory of your OS user account (same directory as the one set by the viewer's file selector, when you use the "Home directory" entry in the pull down list of the "Upper level" button). You may use the '/' separator in 'some_folder' to point deeper in the directory tree. E.g.: #pragma include-from: ~/My Project/include/lua
      For LSL scripts, this directive allows to either specify a given inventory folder, or a file system folder (in the viewer user settings or per account settings directory, or in your "home" directory). For inventory, use the inventory folder names, separated from each others with the pipe ('|') character, such as '#pragma include-from: |Scripts|include|My Project' (this is an absolute inventory path since it starts with a pipe character) or '#pragma include-from: include|draft' (which is a path relative to the folder containing the script being edited in the script editor floater). When you wish to include a file from the user or per account directories instead, use "./" as the first characters of 'some_folder' and '/' as the path separator, such as '#pragma include-from: ./include/lsl'. For a sub-folder of your home directory, use the same convention as for Lua (see above), such as: '#pragma include-from: ~/My Project/include/lsl'.
      NOTE: for obvious security reasons, the folder and included file names are "sanitized" so that you cannot use illegal characters neither "climb up" the directory tree of your file system with "../" path elements (those are removed).

  • #warning some warning message: sends a warning message to the interpreter/compiler.
    The Lua interpreter implemented in the Cool VL Viewer will print such a warning (with references to the preprocessed file name and line number) in the chat console, as a system message prefixed with "Lua preprocessor warning:".
    The LSL script editor will print such messages into its compile messages window, below the editor window.

  • #error some error message: same as for #warning above, but with a "Lua preprocessor error:" prefix in the chat console, and with the interruption of the preprocessing (which will of course also lead to an error reported by the interpreter/compiler).

  • #! is ignored when appearing as the first two characters of the first line of a script (or included) file. This is to cope with shebangs.


Here are the available predefined special (read-only) tokens:

  • __FILE__: expands to a quoted string reflecting the full path of the file being preprocessed.

  • __LINE__: expands to a number which is the line number where __LINE__ appears in the preprocessed file.

  • __DATE__: expands to a quoted string reflecting the current (computer clock) date in the format defined by the "ShortDateFormat" debug setting, which is the international YYYY-MM-DD format by default. E.g.: 2019-02-23

  • __TIME__: expands to a quoted string reflecting the current (computer clock) time in the format defined by the "LongTimeFormat" debug setting, which is the 24 hours HH:MM:SS format by default. E.g.: 14:15:32

  • __AGENT_ID__: expands to a quoted string reflecting your avatar's UUID.

  • __AGENT_NAME__: expands to a quoted string reflecting your avatar's (legacy) name.


Beside not implementing macros expansion, the preprocessor got the following limitations (when compared to a full-fledged C-preprocessor):

  • The preprocessor can properly deal with quoted text when substituting #defined tokens with their values in the sources (i.e. the tokens are not substituted when they are part of a quoted string). However, only the ' and " quotes are taken into account, while you may also use long quotes in Lua, for example. To prevent the preprocessor from doing wild things with such quoted texts in your sources, I therefore implemented a way for you to tell it to stop preprocessing the sources (thus stopping substitutions, but also ignoring anything resembling a preprocessor directive and considering it just normal sources) via two special #pragma directives:
    Code:
    #define TOKEN 1

    #if TOKEN
    # pragma preprocessor-off
    print([[
    Some text that is quoted with Lua
    long-quotes and for which we do not
    want to see TOKEN substituted with
    its value by the preprocessor.
    And we do not want either to see this
    #-bulleted list to be interpreted as
    preprocessor directives:
      # Some listed item
      # Another item
    ]])
    # pragma preprocessor-on
    print("TOKEN is now substitued with: " .. TOKEN)
    #endif

  • defined() does not accept spacing characters between 'defined' and its opening parenthesis, neither the omission of the said parenthesis (but you may use spacing characters around the tested token). I.e. '#if defined( TOKEN )' is valid, while '#if defined (TOKEN)' or '#if defined TOKEN' are not.

  • In #if (and #elif) expressions, care must be taken that it is in fact a Lua evaluator under the hood. While the "!=", "||", "&&", "!" and "^" operators are usable in the expressions and automatically replaced for you by the preprocessor into their Lua equivalents (respectively "~=", " or ", " and ", " not " and "~"), the "!" (not) operator behaves differently than with C-preprocessors: under Lua, 0 == true (ugh !), so !0 is false (doh !). The preprocessor has been coded with care, so that when the result of an expression is a number (rather than a boolean value), it is retrieved from the Lua stack as a number (instead of letting Lua convert it into a boolean) and automatically compared to 0 (which is then properly considered false). But when the expression contains a mix of numbers and booleans, you will get weird results if you do not take care youself of comparing numbers with 0 before using the result with booleans or boolean operators. Example:
    Code:
    #define TOKEN1 12
    #if TOKEN1 % 2 && TOKEN1 >= 10
    print("TOKEN1 is uneven and greater or equal to 10")
    #else
    print("TOKEN1 is even or smaller than 10")
    #endif
    will give you the wrong result, because 12 % 2 == 0 and 0 is 'true' under Lua, which, once combined (via '&&') with another boolean, gives the wrong boolean result.

    The proper way to write it is:
    Code:
    #define TOKEN1 12
    #if TOKEN1 % 2 != 0 && TOKEN1 >= 10
    print("TOKEN1 is uneven and greater or equal to 10")
    #else
    print("TOKEN1 is even or smaller than 10")
    #endif

    Note that:
    Code:
    #define TOKEN1 12
    #if TOKEN1 % 2
    print("TOKEN1 is uneven")
    #else
    print("TOKEN1 is even")
    #endif
    does work as intended without the need for the '!= 0' comparison, because the result of the 'TOKEN1 % 2' expression is a number, not a boolean, and the preprocessor properly considers the zero value as 'false'.

  • The preprocessor got no concept about what is a comment (which is language-dependent), and would not accept comments at the end of its directives (or more exactly, the comment would be considered a directive argument), so:
    Code:
    #define SWITCH1 0 -- Some Lua comment
    #define SWITCH2 0 /* Some LSL comment */
    #define SWITCH3 0 // Some LSL comment
    are illegal.

    Note that since it ignores what is a comment, the preprocessor will also substitute #defined tokens in them, but it is of no consequence whatsoever since the comments are ignored by the language interpreter/compiler and the preprocessed sources are not meant to be human-readable.

Debugging:

To help you finding out what mistake you did in sources with preprocessor directives (or to find a possible bug in my code), you may enable the "Preprocessor" debug tag ("Advanced" -> "Consoles" -> "Debug tags") and watch the messages in the debug console or viewer log.


2019-02-23 14:01:50
Profile WWW

Joined: 2009-03-17 18:42:51
Posts: 4160
Reply with quote
In today's release, I added the possibility to save "escaped, unprocessed" script assets to your inventory via the script editor floater. Here is how it works:

When you want to split your scripts into #include assets and main script parts, you do not want the parts that are to be used as #includes later on, to be preprocessed at the moment you save them, and saving them without a 'default' state would cause the server-side script compiler to report an error anyway...

So, I implemented a way to escape such scripts sources and add a dummy 'default' state in order to:
  • Prevent preprocessing when the #include asset script is saved (thanks to the escaping part).
  • Make the server-side compiler happy (by hiding all the sources and adding the dummy state).

When you need to save such an asset, simply use the "Save include" pull-down entry of the "Save" button, and the script will automatically be converted.

Then, just use this include script asset in your other scripts, with the #include preprocessor directive, and the script floater editor code will automatically and properly unescape and preprocess the #included code. 8-)

Here is an example:
  • "Test.inc" asset:
    As typed and seen in the "Edited script" tab of the script floater:
    Code:
    // Test include

    #if TEST
    string MyName = __AGENT_NAME__;
    #else
    key MyId = __AGENT_ID__;
    #endif

    // Compiled on __DATE__ __TIME__


    And once "Save include" has been used, in the "Saved script" tab (that's what the server sees and stores):
    Code:
    //********** Non-preprocessed include sources **********//
    //* // Test include
    //*
    //* #if TEST
    //* string MyName = __AGENT_NAME__;
    //* #else
    //* key MyId = __AGENT_ID__;
    //* #endif
    //*
    //* // Compiled on __DATE__ __TIME__
    //********* End of non-preprocessed include sources *********//

    default { state_entry() { llOwnerSay("This is an #include script."); } }

  • "Test script" asset, which makes use of "Test.inc":
    As typed and seen in the "Edited script" tab of the script floater:
    Code:
    #define TEST 1

    #include "Test.inc"

    default {
        state_entry() {
    #if TEST
            llOwnerSay("My avatar's name is: " + MyName);
    #else
            llOwnerSay("My avatar's UUID is: " + (string)MyId);
    #endif
        }
    }


    And once the "Save" button has been pressed, in the "Saved script" tab (that's what the server sees and stores):
    Code:

    // Test include

    string MyName = "Henri Beauchamp";

    // Compiled on "09/03/2019" "12:23:45"

    default {
        state_entry() {
            llOwnerSay("My avatar's name is: " + MyName);
        }
    }

    //********** Escaped, original, non-preprocessed sources **********//
    //* #define TEST 1
    //*
    //* #include "Test.inc"
    //*
    //* default {
    //*     state_entry() {
    //* #if TEST
    //*         llOwnerSay("My avatar's name is: " + MyName);
    //* #else
    //*         llOwnerSay("My avatar's UUID is: " + (string)MyId);
    //* #endif
    //*     }
    //* }

    Then you can change "#define TEST 1" with "#define TEST 0" and "Save" again to see the resulting script changing...


2019-03-09 11:20:21
Profile WWW
Display posts from previous:  Sort by  
Reply to topic   [ 2 posts ] 

Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group
Designed by ST Software.