Algorithms

Introduction

A nonterminal can be processed as follows. One selects an alternative and processes it from left to right.

If at the current position inside the alternative there is a token, one compares this with the current input token. If they match the input token is eaten and the working point inside the rule is advanced to the next member.

If at the current position there is a nonterminal, this nonterminal is processed and then the working point is advanced after the symbol.

When at a given point during passing more than one alternative can be applied, there are several approaches:

• Exhaustive parsing follows all possibilities in parallel. (This approach can deal with all grammars)
• Predictive parsing inspects the next input token and requires that this uniquely discriminates one alternative. (This approach can deal with a restrictive class of grammars, LL(1) grammars)

Our approach combines exhaustive and predictive passing. Exhaustive parsing is used to achieve generality. Predictive parsing is used to improve efficency.

Exhaustive Parsing

Whereas in Accent a nonterminal is defined by one rule with several alternatives

   N : A_1 | ... | A_n

   N : A_1
   ...
   N : A_n

   N : M_1 ... M_i ... M_n

   N : M_1 ... * M_i ... M_n

An item has also a "back-pointer" to find items that triggered the actual one (I do not discuss this here).

Earley [1] proposes to attach a dynamically computed look ahead strings to items. Since we use static look ahead set computation we do not use this component.

The algorithm constructs an item list for each input token.

The kernel of the item list for a particular input token is constructed by a step called the scanner.

• Scanner

  If 't' is the current input token then all items of the previous list that have the form

     M : ... * 't' ...
  

  are placed into the next item list where the dot is advanced behind the token

     M : ... 't' * ...
  

  This indicates that 't' has been recognized and the symbol following it has to be processed.

• Predictor

  If the dot appears in front of a nonterm, the "predictor" is invoked. It inserts items that start the processing of the member.

  If the item has the form

     M : ... * N ...
  

  and there is a rule

     N : Alpha
  

  then an item

     N : * Alpha
  

  is inserted.

• Completer

  If the dot appears at the end of a rule, the "completer" is invoked. It takes the item that caused the processing of this rule and puts the dot after the corresponding nonterminal.

  If the item has the form

     N : ... *
  

  and this item was initially triggered by an item

     M : ... * N ...
  

  then an item

     M : ... N * ...
  

  is added, indicating that the member N has been processed.

Processing starts with the item

  YYSTART : * S YYEOF

Predictive Parsing

In this approach, for each alternative of a nonterminal a set of director tokens is computed at compiler generation time. This set contains all tokens that are legal tokens when we start to process the nonterminal. These are given by (1) those tokens that can start the alternative and (2), if the alternative can produce the empty string, the tokens that can follow a phrase for the nonterminal.

For example, consider this simple grammar

   S : N 'x' ;
   N : A | B ;
   A : 'a' ;
   B : ;

   N : A ;

For the alternative

   N : B ;

When parsing a text we begin with the start symbol S and hence have to recognize an N. Assume that we are confronted with an 'a'. In this case we would choose the first alternative for N, because 'a' is in its director set. If we are confronted with an 'x', we would choose the second alternative, because 'x' is in its director set.

In general, if a choice must be made which alternative has to be used to parse a phrase for a particular nonterminal, the first token of the rest of the input is inspected. If it appears in the director set the corresponding alternative is selected.

In order that this works one has to postulate that the director sets of the alternatives of a nonterminal are mutually disjoint (otherwise it would not be clear what alternative should be selected).

This restricts the class of grammars that can be processed with this approach.

Predictive parsers are often implemented by recursive descent.

Here one writes a procedure for each nonterminal that inspects the current input token and uses it to select an alternative. It then processes the members of this alternative.

For example, the nonterminal N from the above grammar could be implemented in this way:

   procedure N()

      if current_token in { 'a' } then
	 A();
      else if current_token in { 'x' } then
	 B();
      else
	 Error();

Combined Parsing

Exhaustive parsing is implemented as described above. We cannot use predictive parsing directly because it would narrow the class of grammars that we want to process.

We cannot use director sets to deterministically select an alternative, but we can use them to exclude an alternative that would not be viable.

For example, if the input

   a x

   S : * N 'x'

   N : * A

   N : * B

   B : *

   N : B *

   S : N * 'x'

The director set of the alternative

   N : B

Predictive parsing requires that the director sets uniquely determines one alternative. In Accent, if the current token is in the director set of more than one alternative, all these alternatives are processed. Alternatives with a director set that does not contain the current token are excluded.

Accent also uses recursive descent to execute semantic actions. This cannot be done during parsing because several rules can be processed in parallel. Hence there is a second pass. The generated procedures look like those presented above, they do not inspect the director set but use the result of the parsing to select an alternative.

Structure Information

There is a "sub-pointer" that refers to the "subtree item" of the current item. If the item has the form

   M : alpha N * beta

   N : gamma *

A "left-pointer" is used as a reference to "preceding items". If the item has the form

   M : alpha N * beta

   M : alpha * N beta

This information can be used to detect and resolve ambiguities at the earliest point.

References