Case Study in Customising Syntax Highlighting and Folding in Vim for a Niche Scripting Language Called Inquisit

This post presents my efforts to setup a productive environment for editing Inquisit scripts in Vim. In addition to being relevant to people who write Inquisit scripts, the post is designed as a general case study in customising Vim for a niche programming language. Specifically, the post discusses how to setup in Vim: (a) code folding using a custom expression, (b) custom syntax highlighting, and (c) interactions with the command-line.

Overview

Inquisit is proprietary software typically used for running computerised experiments in the behavioural sciences. The Inquisit Website has further explanation and includes a free limited trial. I have previously discussed the benefits of the software for running online psychological experiments.

Inquisit includes desktop software that runs on Windows. It can be used to write, debug, and run Inquisit scripts that control the flow of the experiment. I have provided a simple example of a script here as a gist on github. It contains a simple RT task, four choice RT task, and a typing test. Further example scripts can be seen in the Task Library on the Inquisit Website.

More recently I have started using Vim to do my text editing (see my previous post on this conversion. The conversion was motivated by an aim to increase the power and consistency of my text editing environment. Thus, instead of using a separate text editor for each programming language that I code in, I wanted, as much as possible to be able to use Vim for as many tasks as possible.

Vim has great built-in support for many languages, however, Inquisit is a niche scripting language. Thus, Vim lacked built-in support for the language. Nonetheless, to true power of Vim is its customisability.

Below I explain the customisations I developed to work more effectively with Inquisit. It is still a work in progress, and imperfect. However, even at this stage, I find writing Inquisit scripts far more enjoyable than using the editor built-in to the Inquisit software.

I present this example as a case study in customising Vim for a new programming language.

If you want to follow any updates, check out:

• the git repository with the entire contents of my vimfiles and plugins
• inside the repository is my vimrc which includes various customisations for Inquisit file types.
• my syntax highlighting file for Inquisit files

Below I describe a few of these customisations. The sections of greatest general interest are likely to be those on syntax highlighting and code folding.

Overview of an Inquisit script

Before describing my customisations, below is a short snippet from an Inquisit script:

# general stimuli
<text cross>
/position=(50%, 50%)
/items=("+")
/fontstyle = ("Arial", 60pt, false)
</text>

<text error.toosoon>
/position=(50%, 50%)
/items=("PRESSED TOO SOON")
/fontstyle = ("Arial", 30pt, false)
/txcolor =(255,0,0)
</text>

• The line beginning with a hash is my own internal section marker. All text outside element tags <...>...</...> is not interpreted by Inquisit.
• The code is made up of elements which have a keyword name (e.g., text) and an identifier (e.g., cross or error.toosoon).
• In the above example, these elements are used to represent objects displayed on a screen such as a plus symbol or some text.
• Elements have attributes, which have a name and a value. Attributes are contained between opening (e.g., <text ...>) and closing element tags (e.g., </text>). Attributes begin with a slash, then the name of the attribute (e.g., position, then equals sign, and then the value of the attribute.
• Values of attributes have various syntactical forms typically involving either parentheses or square brackets.

My scripts often range between 500 and 2000 lines of code. For a more complete example, see the gist mentioned earlier. You can even download a free trial of the software if you'd like to see how it runs. The remainder of this post describes the customisations that I developed to make writing Inquisit scripts in Vim more productive.

Launching an Inquisit Script from Vim

Question

• How can Inquisit Scripts be tested directly from Vim command-line?

Discussion

• Inquisit Help lists command line options

I adopted the following strategy. First, I added the Inquisit executable to the path; E.g., I added C:\Program Files\Millisecond Software\Inquisit 3 to my path. This means that entering inquisit on the command-line starts Inquisit.

Then I added the following code to my vimrc file.

function! RunInquisit()
    let cmdstring = "!start inquisit " . '"%:p"'
    exe cmdstring
endfunction

• Thus, when I have an Inquisit script open in Vim, I just have to type :call RunInquisit() and this will start Inquisit and send the script in the active buffer to Inquisit for running. Once run, this enables debugging of any errors (Inquisit has a good built-in debugger). If no errors are obtained, the experiment will start, which facilitates the identification of errors in the actual design of the experiment.
• function! declares a new function overriding any previous function by the same name.
• RunInquisit() is the arbitrary name of the function, although it does need to start with a capital letter. In this case, it takes no arguments.
• let cmdstring... assigns a string to the variable cmdstring. The period (.) is the concatenation operator. !start is means of starting a program from the command line asynchronously. inquisit is the name of the Inquisit executable. %:p is a file modifier pattern where % represents the active buffer and :p indicates that the pattern should be replaced with the file name with complete path for the active buffer. See :h filename-mod for more information.
• exe executes the string on the Vim command-line.

Code Folding

Question

• What is a useful strategy for code folding for Inquisit Scripts?

General discussion of code folding

I'm a huge fan of code folding in general, and in Vim specifically. When done appropriately, code folding provides many benefits. In particular, it makes large files easier to navigate, and it makes it easier to get an overview of the contents of a file.

After some reflection I have a few principles that I use to guide the design of a folding scheme.

• Use the language
  • Folds should where possible take advantage of natural markers in the programming language.
  • Using built-in markers means that time does not need to be allocated to the task of managing the creation of folds.
• Use unobtrusive markers
  • When markers are used that are not part of the language, they should be simple an unobtrusive.
  • Markers should not accidentally occur.
  • This typically involves taking advantage of the commenting system in the language.
• Two or three fold levels is typically sufficient
  • Too many folds creates navigational issues whereby excessive time is spent navigating between folds.
  • As a general rule I like it when the lowest level of folding contains at least 5 or more lines and not much more than a page or page and a half of text.
• Avoid Fold-1 - text - Fold-2 pattern
  • I.e., when a fold is expanded, there should not be a passage of expanded text and then a lower level fold.

Inquisit scripts and code folding

Inquisit scripts are made up of elements that look somewhat like HTML. Good coding practice suggests grouping related elements into sections. This lead me to adopt the following folding scheme.

• Level 1 folds
  • These are lines that begin with a single hash #.
  • As such they are not defined by the language, but the language permits putting almost any text outside elements.
  • They are designed to represent logical groups of Inquisit elements.
  • I typically have sections for:
    • Experiment: a section that sets out any experiment level information such as global variables, data format, random allocation of subjects, and so forth.
    • Blocks: a section that sets out each block in the experiment .
    • One for each block: a section that contains all the trials, stimuli, and item code for a given block.
• Level 2 folds
  • These represent each element.
  • Thus, they are automatically created by the language.

My code implementing folding for Inquisit scripts

The code below achieves this design. It is placed in my vimrc file.

function! InquisitLevel(elements)
    " elements: 0 or 1
    if getline(v:lnum) =~ '^# .*$'
        return ">1"
    endif
    if a:elements && getline(v:lnum) =~ '^<[a-zA-Z].*$'
        return ">2"
    endif
    return "=" 
endfunction                  
au BufEnter *.exp setlocal foldexpr=InquisitLevel(1)
au BufEnter *.exp setlocal foldmethod=expr

• It contains a function that takes an argument called elements. If it is 0, then only section headings are folded and not elements.
• getline is a Vim function that returns the content of a line. In this case v:lnum is used by expr fold method to indicate a particular line number.
• =~ is a logical operator that sees whether the text on the left is matched by the regular expression on the right.
• '^# .*$' means match from the start of the line (^) where the hash character is in the first column, followed by a space, and then any number (*) of characters (.) followed by the end of the line ( $).
• If such a match does occur, then return ">1" which means that heading 1 starts on that line. See h fold-expr for more information.
• a:elements is a variable that contains the value of the argument elements passed to the function. The a: is a necessary prefix for arguments. In Vim 0 is false and 1 is true. The double ampersand (&&) is the logical AND operator.
• The second match matches lines that begin (^) with the less than sign followed by at least one alphabetical character and then any number of additional arbitrary characters. This provides a simple way that adequately matches for my purposes the start of Inquisit elements.
• Thus, if elements is non-zero and there is a match, this line is the start of a level 2 heading (">2").
• If no matches are obtained, a value of = is returned, which tells expr to use the fold level of the previous line.
• The code is set up automatically using the two au commands.
• BufEnter *.exp is an event that occurs when the buffer is entered through file opening or buffer switching.
• *.exp is a file name filter. Thus, the command only works once the file has a file names.
• foldexpr is a Vim setting that takes a custom function that determines fold levels.
• foldmethod is a Vim setting that indicates the type of folding to use.

Using code folding

The main keys that I use for code folding are as follows.

• zj an zk
  • to move the cursor down and up between fold headings.
  • This is not needed when text is already folded, but is useful when the cursor is expanded text and I want to get the cursor to the previous or next fold.
  • I often use zjzx to move down and show only the next fold.
• zc
  • to close a fold.
  • I often follow this up with j and k to navigate over folds.
• zx
  • to show only the current fold.
  • It is an awesome command that I use all the time. It functions like magnifying glass zooming in on the code of interest while at the same time providing hierarchical context.
• zM
  • to close all folds to level 1.
  • I often use zMzx when folds have been expanded. This then gives the effect of only showing the active fold.
• zR
  • to expand all folds.
• zO
  • to expand a single fold and its sub-folds.
  • I often use this when a section of text has more folds than is desirable.

Syntax highlighting

Question

• What is a basic syntax highlighting system for Inquisit scripts?

Basic installation instructions

If all you want to do is make use of my syntax highlighting for Inquisit files, follow these steps:

• Copy my syntax highlighting file for Inquisit files into your syntax directory in your custom vimfiles directory.
• Add the following command to your vimrc file au BufEnter *.exp set ft=inquisit
  • It creates a file type called inquisit that then lets Vim know that it should apply the Inquisit syntax highlighting.
  • Note that it assumes that you are using the standard *.exp file extension.

Screenshots

The following are some screenshots showing what the folding and syntax highlighting look like with a molokai colour scheme.

Image with sections folded

Image of folding with element expanded

Basic image of the syntax highlighting

Overview of writing the syntax highlighting file

Before describing the syntax file in more detail, I should present a few caveats. I am new to syntax files in Vim, and I am far from a master in regular expressions. I hacked out a syntax highlighting scheme that works for me, adequately. It is far from perfect. But I guess that's a lesson in itself when it comes to using Vim. Vim allows for near infinite customisation, but you have to consider the return on investment of any customisation efforts.

In general, when developing the syntax file I had a look through a few example syntax files for what I thought might be similar file types. For every command that I did not understand I did Vim help searches. And in general I just fiddled around with a sample Inquisit file open until I got something that basically worked. For people needing a little more regular expression training, there's a great Vim-oriented tutorial here.

Discussion of the syntax highlighting file

At the top of the file was the following:

syntax case ignore
syn clear

• This tells Vim that syntax highlighting should be case insensitive
• syn clear clears any existing syntax highlighting applied to the file, although this is often not required.

The syntax file then contains many syn keyword commands, such as the following:

syn keyword inquisitElementKeyword contained survey 
syn keyword inquisitElementKeyword contained surveypage 
syn keyword inquisitElementKeyword contained textbox 
...

• Words like survey, surveypage, etc. are all element names and keywords within the language.
• keyword is a simple form of matching that matches the keyword exactly without resorting to patterns.
• inquisitElementKeyword is a grouping name which I use later to assign colours and show where such keywords are contained.
• The contained keyword allows the scope of keywords to be limited to only situations where they are contained within some other syntactical structure.

The syntax file then contains code for matching operators, an example of which is shown below:

syn match inquisitOperator contained "||"
syn match inquisitOperator contained "&&"

• This involved using the match approach. This was required because the characters in operators are not alphabetical characters.

I then added code to syntax highlight numbers.

syn match inquisitNumber contained "\<[0-9%][0-9%]*"

• The code shows the diverse options that match provides. It matches one or more numbers or the percent sign. The combination of backslash and less-than-sign means that numbers must be at the start of what Vim sees as a word.

Then came some code to represent regions of text

syn region inquisitString start=+"+ end=+"+

• The above code defined a region based on quotation marks.

• The plus sign is used as an alternate delimiter in order to indicate that the double quotes are the actual text used to match the start and end of the region.

  syn region inquisitElementBlock start="<..*>" end="" transparent

• The above code represents an Inquisit element.

• The similarity with HTML tags may be apparent.

• The transparent keyword is used to indicate that this merely represents a logical unit and does not impose any actual syntax highlighting.

  syn region inquisitAttributeKeyword contained start="\<" end=">" syn region inquisitAttributeRegion start="\/" end="=" oneline contains=inquisitAttributeKeyword

• The above code creates a quick and dirty way of highlighting attribute names in elements.

• Such attribute names always appear as a word contained between a slash and an equals sign.
• \< and \> match the start and end of a word, but this is contained.
• The second line indicates that the first line is contained in it.
• The oneline keyword ensures that the match occurs entirely on one line.

After some additional code comes the assignment of colours to groups. The following are a few examples:

hi link inquisitElementKeyword Keyword
hi link inquisitOperator Operator
hi link inquisitDelimiter Delimiter  
hi link inquisitString Character

• Each command creates a highlighting link between the custom groups created in the script and a common logical highlighting object used by many colour schemes.
• In case you were interested, I use molokai as my default colour scheme. It is a popular dark scheme available from vim.org, and from Thomas Restrepo's blog.

Getting Inquisit Help while in Vim

Question

Inquisit provides clear and well-organised documentation. In Desktop Inquisit, pressing F1 in a script brings up the help for the element under the cursor.

• How can I make it easy to get help on Inquisit commands while continuing to work within Vim?

Discussion

The simplest strategy is just to keep a copy of the Inquisit help file open while coding and type in search terms directly. In the end, I have found this to be quite adequate. However, I record some thoughts and ideas I explored for more automated approaches below:

• HH "C:\Program Files\Millisecond Software\Inquisit 3\inquisit.chm" will open Inquisit help (at least on my system given my installation path and version).
• Alternatively there is a online version of help: < http://www.millisecond.com/support/docs/v1/index.htm >
• I asked a question on StackOverflow for more information.
• I installed keyhh. The following command basically did the trick: keyhh -#klink "data element" c:\Program Files\Millisecond Software\Inquisit 3\inquisit.chm where "data element" is a hypothetical search term.
• I also found this vimscript designed to assist interaction with chm help files.

Future refinements

In the future I may:

• Improve the syntax highlighting
• Introduce an Inquisit code formatting customisation
• Improve integration with the Inquisit language particularly with regard to attributes
• Develop code that facilitates writing Inquisit scripts, such as sequences of numbers for trial and block sequences

General lessons Learnt

• Tinkering with the syntax highlighting, folding, and command-line interaction in Vim encouraged me to think more deeply about each feature and how the feature could be optimised for my workflow.
• The ability to customise Vim as I have done here validates my decision to adopt Vim. Vim encourages the development of amazing text editing shortcut keys; the more they can be applied to the languages that I write in, the better.
• Investment in customisation pays off both in terms of (a) making the immediate task of programming in the given language easier, and (b) making it easier to customise other languages in the future.

Additional Resources

• My Conversion to Vim
• Conducting an online experiment with Inquisit