282 |
* Parser States:: The parser is a finite-state-machine with stack. |
* Parser States:: The parser is a finite-state-machine with stack. |
283 |
* Reduce/Reduce:: When two rules are applicable in the same situation. |
* Reduce/Reduce:: When two rules are applicable in the same situation. |
284 |
* Mystery Conflicts:: Reduce/reduce conflicts that look unjustified. |
* Mystery Conflicts:: Reduce/reduce conflicts that look unjustified. |
285 |
|
* Generalized LR Parsing:: Parsing arbitrary context-free grammars. |
286 |
* Stack Overflow:: What happens when stack gets full. How to avoid it. |
* Stack Overflow:: What happens when stack gets full. How to avoid it. |
287 |
|
|
288 |
Operator Precedence |
Operator Precedence |
389 |
a semantic value (the value of an integer, |
a semantic value (the value of an integer, |
390 |
the name of an identifier, etc.). |
the name of an identifier, etc.). |
391 |
* Semantic Actions:: Each rule can have an action containing C code. |
* Semantic Actions:: Each rule can have an action containing C code. |
392 |
|
* GLR Parsers:: Writing parsers for general context-free languages |
393 |
* Locations Overview:: Tracking Locations. |
* Locations Overview:: Tracking Locations. |
394 |
* Bison Parser:: What are Bison's input and output, |
* Bison Parser:: What are Bison's input and output, |
395 |
how is the output used? |
how is the output used? |
420 |
context-free grammar. The input to Bison is essentially machine-readable |
context-free grammar. The input to Bison is essentially machine-readable |
421 |
BNF. |
BNF. |
422 |
|
|
423 |
Not all context-free languages can be handled by Bison, only those |
@cindex LALR(1) grammars |
424 |
that are LALR(1). In brief, this means that it must be possible to |
@cindex LR(1) grammars |
425 |
|
There are various important subclasses of context-free grammar. Although it |
426 |
|
can handle almost all context-free grammars, Bison is optimized for what |
427 |
|
are called LALR(1) grammars. |
428 |
|
In brief, in these grammars, it must be possible to |
429 |
tell how to parse any portion of an input string with just a single |
tell how to parse any portion of an input string with just a single |
430 |
token of look-ahead. Strictly speaking, that is a description of an |
token of look-ahead. Strictly speaking, that is a description of an |
431 |
LR(1) grammar, and LALR(1) involves additional restrictions that are |
LR(1) grammar, and LALR(1) involves additional restrictions that are |
433 |
LR(1) grammar that fails to be LALR(1). @xref{Mystery Conflicts, , |
LR(1) grammar that fails to be LALR(1). @xref{Mystery Conflicts, , |
434 |
Mysterious Reduce/Reduce Conflicts}, for more information on this. |
Mysterious Reduce/Reduce Conflicts}, for more information on this. |
435 |
|
|
436 |
|
@cindex GLR parsing |
437 |
|
@cindex generalized LR (GLR) parsing |
438 |
|
@cindex ambiguous grammars |
439 |
|
@cindex non-deterministic parsing |
440 |
|
Parsers for LALR(1) grammars are @dfn{deterministic}, meaning roughly that |
441 |
|
the next grammar rule to apply at any point in the input is uniquely |
442 |
|
determined by the preceding input and a fixed, finite portion (called |
443 |
|
a @dfn{look-ahead}) of the remaining input. |
444 |
|
A context-free grammar can be @dfn{ambiguous}, meaning that |
445 |
|
there are multiple ways to apply the grammar rules to get the some inputs. |
446 |
|
Even unambiguous grammars can be @dfn{non-deterministic}, meaning that no |
447 |
|
fixed look-ahead always suffices to determine the next grammar rule to apply. |
448 |
|
With the proper declarations, Bison is also able to parse these more general |
449 |
|
context-free grammars, using a technique known as GLR parsing (for |
450 |
|
Generalized LR). Bison's GLR parsers are able to handle any context-free |
451 |
|
grammar for which the number of possible parses of any given string |
452 |
|
is finite. |
453 |
|
|
454 |
@cindex symbols (abstract) |
@cindex symbols (abstract) |
455 |
@cindex token |
@cindex token |
456 |
@cindex syntactic grouping |
@cindex syntactic grouping |
656 |
The action says how to produce the semantic value of the sum expression |
The action says how to produce the semantic value of the sum expression |
657 |
from the values of the two subexpressions. |
from the values of the two subexpressions. |
658 |
|
|
659 |
|
@node GLR Parsers |
660 |
|
@section Writing GLR Parsers |
661 |
|
@cindex GLR parsing |
662 |
|
@cindex generalized LR (GLR) parsing |
663 |
|
@findex %glr-parser |
664 |
|
@cindex conflicts |
665 |
|
@cindex shift/reduce conflicts |
666 |
|
|
667 |
|
In some grammars, there will be cases where Bison's standard LALR(1) |
668 |
|
parsing algorithm cannot decide whether to apply a certain grammar rule |
669 |
|
at a given point. That is, it may not be able to decide (on the basis |
670 |
|
of the input read so far) which of two possible reductions (applications |
671 |
|
of a grammar rule) applies, or whether to apply a reduction or read more |
672 |
|
of the input and apply a reduction later in the input. These are known |
673 |
|
respectively as @dfn{reduce/reduce} conflicts (@pxref{Reduce/Reduce}), |
674 |
|
and @dfn{shift/reduce} conflicts (@pxref{Shift/Reduce}). |
675 |
|
|
676 |
|
To use a grammar that is not easily modified to be LALR(1), a more |
677 |
|
general parsing algorithm is sometimes necessary. If you include |
678 |
|
@code{%glr-parser} among the Bison declarations in your file |
679 |
|
(@pxref{Grammar Outline}), the result will be a Generalized LR (GLR) |
680 |
|
parser. These parsers handle Bison grammars that contain no unresolved |
681 |
|
conflicts (i.e., after applying precedence declarations) identically to |
682 |
|
LALR(1) parsers. However, when faced with unresolved shift/reduce and |
683 |
|
reduce/reduce conflicts, GLR parsers use the simple expedient of doing |
684 |
|
both, effectively cloning the parser to follow both possibilities. Each |
685 |
|
of the resulting parsers can again split, so that at any given time, |
686 |
|
there can be any number of possible parses being explored. The parsers |
687 |
|
proceed in lockstep; that is, all of them consume (shift) a given input |
688 |
|
symbol before any of them proceed to the next. Each of the cloned |
689 |
|
parsers eventually meets one of two possible fates: either it runs into |
690 |
|
a parsing error, in which case it simply vanishes, or it merges with |
691 |
|
another parser, because the two of them have reduced the input to an |
692 |
|
identical set of symbols. |
693 |
|
|
694 |
|
During the time that there are multiple parsers, semantic actions are |
695 |
|
recorded, but not performed. When a parser disappears, its recorded |
696 |
|
semantic actions disappear as well, and are never performed. When a |
697 |
|
reduction makes two parsers identical, causing them to merge, Bison |
698 |
|
records both sets of semantic actions. Whenever the last two parsers |
699 |
|
merge, reverting to the single-parser case, Bison resolves all the |
700 |
|
outstanding actions either by precedences given to the grammar rules |
701 |
|
involved, or by performing both actions, and then calling a designated |
702 |
|
user-defined function on the resulting values to produce an arbitrary |
703 |
|
merged result. |
704 |
|
|
705 |
|
Let's consider an example, vastly simplified from C++. |
706 |
|
|
707 |
|
@example |
708 |
|
%@{ |
709 |
|
#define YYSTYPE const char* |
710 |
|
%@} |
711 |
|
|
712 |
|
%token TYPENAME ID |
713 |
|
|
714 |
|
%right '=' |
715 |
|
%left '+' |
716 |
|
|
717 |
|
%glr-parser |
718 |
|
|
719 |
|
%% |
720 |
|
|
721 |
|
prog : |
722 |
|
| prog stmt @{ printf ("\n"); @} |
723 |
|
; |
724 |
|
|
725 |
|
stmt : expr ';' %dprec 1 |
726 |
|
| decl %dprec 2 |
727 |
|
; |
728 |
|
|
729 |
|
expr : ID @{ printf ("%s ", $$); @} |
730 |
|
| TYPENAME '(' expr ')' |
731 |
|
@{ printf ("%s <cast> ", $1); @} |
732 |
|
| expr '+' expr @{ printf ("+ "); @} |
733 |
|
| expr '=' expr @{ printf ("= "); @} |
734 |
|
; |
735 |
|
|
736 |
|
decl : TYPENAME declarator ';' |
737 |
|
@{ printf ("%s <declare> ", $1); @} |
738 |
|
| TYPENAME declarator '=' expr ';' |
739 |
|
@{ printf ("%s <init-declare> ", $1); @} |
740 |
|
; |
741 |
|
|
742 |
|
declarator : ID @{ printf ("\"%s\" ", $1); @} |
743 |
|
| '(' declarator ')' |
744 |
|
; |
745 |
|
@end example |
746 |
|
|
747 |
|
@noindent |
748 |
|
This models a problematic part of the C++ grammar---the ambiguity between |
749 |
|
certain declarations and statements. For example, |
750 |
|
|
751 |
|
@example |
752 |
|
T (x) = y+z; |
753 |
|
@end example |
754 |
|
|
755 |
|
@noindent |
756 |
|
parses as either an @code{expr} or a @code{stmt} |
757 |
|
(assuming that @samp{T} is recognized as a TYPENAME and @samp{x} as an ID). |
758 |
|
Bison detects this as a reduce/reduce conflict between the rules |
759 |
|
@code{expr : ID} and @code{declarator : ID}, which it cannot resolve at the |
760 |
|
time it encounters @code{x} in the example above. The two @code{%dprec} |
761 |
|
declarations, however, give precedence to interpreting the example as a |
762 |
|
@code{decl}, which implies that @code{x} is a declarator. |
763 |
|
The parser therefore prints |
764 |
|
|
765 |
|
@example |
766 |
|
"x" y z + T <init-declare> |
767 |
|
@end example |
768 |
|
|
769 |
|
Consider a different input string for this parser: |
770 |
|
|
771 |
|
@example |
772 |
|
T (x) + y; |
773 |
|
@end example |
774 |
|
|
775 |
|
@noindent |
776 |
|
Here, there is no ambiguity (this cannot be parsed as a declaration). |
777 |
|
However, at the time the Bison parser encounters @code{x}, it does not |
778 |
|
have enough information to resolve the reduce/reduce conflict (again, |
779 |
|
between @code{x} as an @code{expr} or a @code{declarator}). In this |
780 |
|
case, no precedence declaration is used. Instead, the parser splits |
781 |
|
into two, one assuming that @code{x} is an @code{expr}, and the other |
782 |
|
assuming @code{x} is a @code{declarator}. The second of these parsers |
783 |
|
then vanishes when it sees @code{+}, and the parser prints |
784 |
|
|
785 |
|
@example |
786 |
|
x T <cast> y + |
787 |
|
@end example |
788 |
|
|
789 |
|
Suppose that instead of resolving the ambiguity, you wanted to see all |
790 |
|
the possibilities. For this purpose, we must @dfn{merge} the semantic |
791 |
|
actions of the two possible parsers, rather than choosing one over the |
792 |
|
other. To do so, you could change the declaration of @code{stmt} as |
793 |
|
follows: |
794 |
|
|
795 |
|
@example |
796 |
|
stmt : expr ';' %merge <stmtMerge> |
797 |
|
| decl %merge <stmtMerge> |
798 |
|
; |
799 |
|
@end example |
800 |
|
|
801 |
|
@noindent |
802 |
|
|
803 |
|
and define the @code{stmtMerge} function as: |
804 |
|
|
805 |
|
@example |
806 |
|
static YYSTYPE stmtMerge (YYSTYPE x0, YYSTYPE x1) |
807 |
|
@{ |
808 |
|
printf ("<OR> "); |
809 |
|
return ""; |
810 |
|
@} |
811 |
|
@end example |
812 |
|
|
813 |
|
@noindent |
814 |
|
with an accompanying forward declaration |
815 |
|
in the C declarations at the beginning of the file: |
816 |
|
|
817 |
|
@example |
818 |
|
%@{ |
819 |
|
#define YYSTYPE const char* |
820 |
|
static YYSTYPE stmtMerge (YYSTYPE x0, YYSTYPE x1); |
821 |
|
%@} |
822 |
|
@end example |
823 |
|
|
824 |
|
@noindent |
825 |
|
With these declarations, the resulting parser will parse the first example |
826 |
|
as both an @code{expr} and a @code{decl}, and print |
827 |
|
|
828 |
|
@example |
829 |
|
"x" y z + T <init-declare> x T <cast> y z + = <OR> |
830 |
|
@end example |
831 |
|
|
832 |
|
|
833 |
@node Locations Overview |
@node Locations Overview |
834 |
@section Locations |
@section Locations |
835 |
@cindex location |
@cindex location |
3111 |
is an array holding locations of all right hand side elements of the rule |
is an array holding locations of all right hand side elements of the rule |
3112 |
being matched. The last one is the size of the right hand side rule. |
being matched. The last one is the size of the right hand side rule. |
3113 |
|
|
3114 |
By default, it is defined this way: |
By default, it is defined this way for simple LALR(1) parsers: |
3115 |
|
|
3116 |
@example |
@example |
3117 |
@group |
@group |
3123 |
@end group |
@end group |
3124 |
@end example |
@end example |
3125 |
|
|
3126 |
|
@noindent |
3127 |
|
and like this for GLR parsers: |
3128 |
|
|
3129 |
|
@example |
3130 |
|
@group |
3131 |
|
#define YYLLOC_DEFAULT(Current, Rhs, N) \ |
3132 |
|
Current.first_line = YYRHSLOC(Rhs,1).first_line; \ |
3133 |
|
Current.first_column = YYRHSLOC(Rhs,1).first_column; \ |
3134 |
|
Current.last_line = YYRHSLOC(Rhs,N).last_line; \ |
3135 |
|
Current.last_column = YYRHSLOC(Rhs,N).last_column; |
3136 |
|
@end group |
3137 |
|
@end example |
3138 |
|
|
3139 |
When defining @code{YYLLOC_DEFAULT}, you should consider that: |
When defining @code{YYLLOC_DEFAULT}, you should consider that: |
3140 |
|
|
3141 |
@itemize @bullet |
@itemize @bullet |
4101 |
@findex YYBACKUP |
@findex YYBACKUP |
4102 |
Unshift a token. This macro is allowed only for rules that reduce |
Unshift a token. This macro is allowed only for rules that reduce |
4103 |
a single value, and only when there is no look-ahead token. |
a single value, and only when there is no look-ahead token. |
4104 |
|
It is also disallowed in GLR parsers. |
4105 |
It installs a look-ahead token with token type @var{token} and |
It installs a look-ahead token with token type @var{token} and |
4106 |
semantic value @var{value}; then it discards the value that was |
semantic value @var{value}; then it discards the value that was |
4107 |
going to be reduced by this rule. |
going to be reduced by this rule. |
4242 |
* Parser States:: The parser is a finite-state-machine with stack. |
* Parser States:: The parser is a finite-state-machine with stack. |
4243 |
* Reduce/Reduce:: When two rules are applicable in the same situation. |
* Reduce/Reduce:: When two rules are applicable in the same situation. |
4244 |
* Mystery Conflicts:: Reduce/reduce conflicts that look unjustified. |
* Mystery Conflicts:: Reduce/reduce conflicts that look unjustified. |
4245 |
|
* Generalized LR Parsing:: Parsing arbitrary context-free grammars. |
4246 |
* Stack Overflow:: What happens when stack gets full. How to avoid it. |
* Stack Overflow:: What happens when stack gets full. How to avoid it. |
4247 |
@end menu |
@end menu |
4248 |
|
|
4837 |
; |
; |
4838 |
@end example |
@end example |
4839 |
|
|
4840 |
|
@node Generalized LR Parsing |
4841 |
|
@section Generalized LR (GLR) Parsing |
4842 |
|
@cindex GLR parsing |
4843 |
|
@cindex generalized LR (GLR) parsing |
4844 |
|
@cindex ambiguous grammars |
4845 |
|
@cindex non-deterministic parsing |
4846 |
|
|
4847 |
|
Bison produces @emph{deterministic} parsers that choose uniquely |
4848 |
|
when to reduce and which reduction to apply |
4849 |
|
based on a summary of the preceding input and on one extra token of lookahead. |
4850 |
|
As a result, normal Bison handles a proper subset of the family of |
4851 |
|
context-free languages. |
4852 |
|
Ambiguous grammars, since they have strings with more than one possible |
4853 |
|
sequence of reductions cannot have deterministic parsers in this sense. |
4854 |
|
The same is true of languages that require more than one symbol of |
4855 |
|
lookahead, since the parser lacks the information necessary to make a |
4856 |
|
decision at the point it must be made in a shift-reduce parser. |
4857 |
|
Finally, as previously mentioned (@pxref{Mystery Conflicts}), |
4858 |
|
there are languages where Bison's particular choice of how to |
4859 |
|
summarize the input seen so far loses necessary information. |
4860 |
|
|
4861 |
|
When you use the @samp{%glr-parser} declaration in your grammar file, |
4862 |
|
Bison generates a parser that uses a different algorithm, called |
4863 |
|
Generalized LR (or GLR). A Bison GLR parser uses the same basic |
4864 |
|
algorithm for parsing as an ordinary Bison parser, but behaves |
4865 |
|
differently in cases where there is a shift-reduce conflict that has not |
4866 |
|
been resolved by precedence rules (@pxref{Precedence}) or a |
4867 |
|
reduce-reduce conflict. When a GLR parser encounters such a situation, it |
4868 |
|
effectively @emph{splits} into a several parsers, one for each possible |
4869 |
|
shift or reduction. These parsers then proceed as usual, consuming |
4870 |
|
tokens in lock-step. Some of the stacks may encounter other conflicts |
4871 |
|
and split further, with the result that instead of a sequence of states, |
4872 |
|
a Bison GLR parsing stack is what is in effect a tree of states. |
4873 |
|
|
4874 |
|
In effect, each stack represents a guess as to what the proper parse |
4875 |
|
is. Additional input may indicate that a guess was wrong, in which case |
4876 |
|
the appropriate stack silently disappears. Otherwise, the semantics |
4877 |
|
actions generated in each stack are saved, rather than being executed |
4878 |
|
immediately. When a stack disappears, its saved semantic actions never |
4879 |
|
get executed. When a reduction causes two stacks to become equivalent, |
4880 |
|
their sets of semantic actions are both saved with the state that |
4881 |
|
results from the reduction. We say that two stacks are equivalent |
4882 |
|
when they both represent the same sequence of states, |
4883 |
|
and each pair of corresponding states represents a |
4884 |
|
grammar symbol that produces the same segment of the input token |
4885 |
|
stream. |
4886 |
|
|
4887 |
|
Whenever the parser makes a transition from having multiple |
4888 |
|
states to having one, it reverts to the normal LALR(1) parsing |
4889 |
|
algorithm, after resolving and executing the saved-up actions. |
4890 |
|
At this transition, some of the states on the stack will have semantic |
4891 |
|
values that are sets (actually multisets) of possible actions. The |
4892 |
|
parser tries to pick one of the actions by first finding one whose rule |
4893 |
|
has the highest dynamic precedence, as set by the @samp{%dprec} |
4894 |
|
declaration. Otherwise, if the alternative actions are not ordered by |
4895 |
|
precedence, but there the same merging function is declared for both |
4896 |
|
rules by the @samp{%merge} declaration, |
4897 |
|
Bison resolves and evaluates both and then calls the merge function on |
4898 |
|
the result. Otherwise, it reports an ambiguity. |
4899 |
|
|
4900 |
|
It is possible to use a data structure for the GLR parsing tree that |
4901 |
|
permits the processing of any LALR(1) grammar in linear time (in the |
4902 |
|
size of the input), any unambiguous (not necessarily LALR(1)) grammar in |
4903 |
|
quadratic worst-case time, and any general (possibly ambiguous) |
4904 |
|
context-free grammar in cubic worst-case time. However, Bison currently |
4905 |
|
uses a simpler data structure that requires time proportional to the |
4906 |
|
length of the input times the maximum number of stacks required for any |
4907 |
|
prefix of the input. Thus, really ambiguous or non-deterministic |
4908 |
|
grammars can require exponential time and space to process. Such badly |
4909 |
|
behaving examples, however, are not generally of practical interest. |
4910 |
|
Usually, non-determinism in a grammar is local---the parser is ``in |
4911 |
|
doubt'' only for a few tokens at a time. Therefore, the current data |
4912 |
|
structure should generally be adequate. On LALR(1) portions of a |
4913 |
|
grammar, in particular, it is only slightly slower than with the default |
4914 |
|
Bison parser. |
4915 |
|
|
4916 |
@node Stack Overflow |
@node Stack Overflow |
4917 |
@section Stack Overflow, and How to Avoid It |
@section Stack Overflow, and How to Avoid It |
4918 |
@cindex stack overflow |
@cindex stack overflow |
6201 |
Bison declaration to create a header file meant for the scanner. |
Bison declaration to create a header file meant for the scanner. |
6202 |
@xref{Decl Summary}. |
@xref{Decl Summary}. |
6203 |
|
|
6204 |
|
@item %dprec |
6205 |
|
Bison declaration to assign a precedence to a rule that is used at parse |
6206 |
|
time to resolve reduce/reduce conflicts. @xref{GLR Parsers}. |
6207 |
|
|
6208 |
@item %file-prefix="@var{prefix}" |
@item %file-prefix="@var{prefix}" |
6209 |
Bison declaration to set tge prefix of the output files. @xref{Decl |
Bison declaration to set the prefix of the output files. @xref{Decl |
6210 |
Summary}. |
Summary}. |
6211 |
|
|
6212 |
|
@item %glr-parser |
6213 |
|
Bison declaration to produce a GLR parser. @xref{GLR Parsers}. |
6214 |
|
|
6215 |
@c @item %source-extension |
@c @item %source-extension |
6216 |
@c Bison declaration to specify the generated parser output file extension. |
@c Bison declaration to specify the generated parser output file extension. |
6217 |
@c @xref{Decl Summary}. |
@c @xref{Decl Summary}. |
6224 |
Bison declaration to assign left associativity to token(s). |
Bison declaration to assign left associativity to token(s). |
6225 |
@xref{Precedence Decl, ,Operator Precedence}. |
@xref{Precedence Decl, ,Operator Precedence}. |
6226 |
|
|
6227 |
|
@item %merge |
6228 |
|
Bison declaration to assign a merging function to a rule. If there is a |
6229 |
|
reduce/reduce conflict with a rule having the same merging function, the |
6230 |
|
function is applied to the two semantic values to get a single result. |
6231 |
|
@xref{GLR Parsers}. |
6232 |
|
|
6233 |
@item %name-prefix="@var{prefix}" |
@item %name-prefix="@var{prefix}" |
6234 |
Bison declaration to rename the external symbols. @xref{Decl Summary}. |
Bison declaration to rename the external symbols. @xref{Decl Summary}. |
6235 |
|
|
6342 |
parsed, and the states correspond to various stages in the grammar |
parsed, and the states correspond to various stages in the grammar |
6343 |
rules. @xref{Algorithm, ,The Bison Parser Algorithm }. |
rules. @xref{Algorithm, ,The Bison Parser Algorithm }. |
6344 |
|
|
6345 |
|
@item Generalized LR (GLR) |
6346 |
|
A parsing algorithm that can handle all context-free grammars, including those |
6347 |
|
that are not LALR(1). It resolves situations that Bison's usual LALR(1) |
6348 |
|
algorithm cannot by effectively splitting off multiple parsers, trying all |
6349 |
|
possible parsers, and discarding those that fail in the light of additional |
6350 |
|
right context. @xref{Generalized LR Parsing, ,Generalized LR Parsing}. |
6351 |
|
|
6352 |
@item Grouping |
@item Grouping |
6353 |
A language construct that is (in general) grammatically divisible; |
A language construct that is (in general) grammatically divisible; |
6354 |
for example, `expression' or `declaration' in C. |
for example, `expression' or `declaration' in C. |