/[bison]/bison/doc/bison.texinfo
ViewVC logotype

Diff of /bison/doc/bison.texinfo

Parent Directory Parent Directory | Revision Log Revision Log | View Patch Patch

revision 1.63 by akim, Thu Jun 27 12:08:20 2002 UTC revision 1.64 by hilfinger, Fri Jun 28 02:26:43 2002 UTC
# Line 282  The Bison Parser Algorithm Line 282  The Bison Parser Algorithm
282  * Parser States::     The parser is a finite-state-machine with stack.  * Parser States::     The parser is a finite-state-machine with stack.
283  * Reduce/Reduce::     When two rules are applicable in the same situation.  * Reduce/Reduce::     When two rules are applicable in the same situation.
284  * Mystery Conflicts::  Reduce/reduce conflicts that look unjustified.  * Mystery Conflicts::  Reduce/reduce conflicts that look unjustified.
285    * Generalized LR Parsing::  Parsing arbitrary context-free grammars.
286  * Stack Overflow::    What happens when stack gets full.  How to avoid it.  * Stack Overflow::    What happens when stack gets full.  How to avoid it.
287    
288  Operator Precedence  Operator Precedence
# Line 388  use Bison or Yacc, we suggest you start Line 389  use Bison or Yacc, we suggest you start
389                          a semantic value (the value of an integer,                          a semantic value (the value of an integer,
390                          the name of an identifier, etc.).                          the name of an identifier, etc.).
391  * Semantic Actions::  Each rule can have an action containing C code.  * Semantic Actions::  Each rule can have an action containing C code.
392    * GLR Parsers::       Writing parsers for general context-free languages
393  * Locations Overview::    Tracking Locations.  * Locations Overview::    Tracking Locations.
394  * Bison Parser::      What are Bison's input and output,  * Bison Parser::      What are Bison's input and output,
395                          how is the output used?                          how is the output used?
# Line 418  specify the language Algol 60.  Any gram Line 420  specify the language Algol 60.  Any gram
420  context-free grammar.  The input to Bison is essentially machine-readable  context-free grammar.  The input to Bison is essentially machine-readable
421  BNF.  BNF.
422    
423  Not all context-free languages can be handled by Bison, only those  @cindex LALR(1) grammars
424  that are LALR(1).  In brief, this means that it must be possible to  @cindex LR(1) grammars
425    There are various important subclasses of context-free grammar.  Although it
426    can handle almost all context-free grammars, Bison is optimized for what
427    are called LALR(1) grammars.
428    In brief, in these grammars, it must be possible to
429  tell how to parse any portion of an input string with just a single  tell how to parse any portion of an input string with just a single
430  token of look-ahead.  Strictly speaking, that is a description of an  token of look-ahead.  Strictly speaking, that is a description of an
431  LR(1) grammar, and LALR(1) involves additional restrictions that are  LR(1) grammar, and LALR(1) involves additional restrictions that are
# Line 427  hard to explain simply; but it is rare i Line 433  hard to explain simply; but it is rare i
433  LR(1) grammar that fails to be LALR(1).  @xref{Mystery Conflicts, ,  LR(1) grammar that fails to be LALR(1).  @xref{Mystery Conflicts, ,
434  Mysterious Reduce/Reduce Conflicts}, for more information on this.  Mysterious Reduce/Reduce Conflicts}, for more information on this.
435    
436    @cindex GLR parsing
437    @cindex generalized LR (GLR) parsing
438    @cindex ambiguous grammars
439    @cindex non-deterministic parsing
440    Parsers for LALR(1) grammars are @dfn{deterministic}, meaning roughly that
441    the next grammar rule to apply at any point in the input is uniquely
442    determined by the preceding input and a fixed, finite portion (called
443    a @dfn{look-ahead}) of the remaining input.
444    A context-free grammar can be @dfn{ambiguous}, meaning that
445    there are multiple ways to apply the grammar rules to get the some inputs.
446    Even unambiguous grammars can be @dfn{non-deterministic}, meaning that no
447    fixed look-ahead always suffices to determine the next grammar rule to apply.
448    With the proper declarations, Bison is also able to parse these more general
449    context-free grammars, using a technique known as GLR parsing (for
450    Generalized LR).  Bison's GLR parsers are able to handle any context-free
451    grammar for which the number of possible parses of any given string
452    is finite.  
453    
454  @cindex symbols (abstract)  @cindex symbols (abstract)
455  @cindex token  @cindex token
456  @cindex syntactic grouping  @cindex syntactic grouping
# Line 632  expr: expr '+' expr   @{ $$ = $1 + $3; @ Line 656  expr: expr '+' expr   @{ $$ = $1 + $3; @
656  The action says how to produce the semantic value of the sum expression  The action says how to produce the semantic value of the sum expression
657  from the values of the two subexpressions.  from the values of the two subexpressions.
658    
659    @node GLR Parsers
660    @section Writing GLR Parsers
661    @cindex GLR parsing
662    @cindex generalized LR (GLR) parsing
663    @findex %glr-parser
664    @cindex conflicts
665    @cindex shift/reduce conflicts
666    
667    In some grammars, there will be cases where Bison's standard LALR(1)
668    parsing algorithm cannot decide whether to apply a certain grammar rule
669    at a given point.  That is, it may not be able to decide (on the basis
670    of the input read so far) which of two possible reductions (applications
671    of a grammar rule) applies, or whether to apply a reduction or read more
672    of the input and apply a reduction later in the input.  These are known
673    respectively as @dfn{reduce/reduce} conflicts (@pxref{Reduce/Reduce}),
674    and @dfn{shift/reduce} conflicts (@pxref{Shift/Reduce}).
675    
676    To use a grammar that is not easily modified to be LALR(1), a more
677    general parsing algorithm is sometimes necessary.  If you include
678    @code{%glr-parser} among the Bison declarations in your file
679    (@pxref{Grammar Outline}), the result will be a Generalized LR (GLR)
680    parser.  These parsers handle Bison grammars that contain no unresolved
681    conflicts (i.e., after applying precedence declarations) identically to
682    LALR(1) parsers.  However, when faced with unresolved shift/reduce and
683    reduce/reduce conflicts, GLR parsers use the simple expedient of doing
684    both, effectively cloning the parser to follow both possibilities.  Each
685    of the resulting parsers can again split, so that at any given time,
686    there can be any number of possible parses being explored.  The parsers
687    proceed in lockstep; that is, all of them consume (shift) a given input
688    symbol before any of them proceed to the next.  Each of the cloned
689    parsers eventually meets one of two possible fates: either it runs into
690    a parsing error, in which case it simply vanishes, or it merges with
691    another parser, because the two of them have reduced the input to an
692    identical set of symbols.
693    
694    During the time that there are multiple parsers, semantic actions are
695    recorded, but not performed.  When a parser disappears, its recorded
696    semantic actions disappear as well, and are never performed.  When a
697    reduction makes two parsers identical, causing them to merge, Bison
698    records both sets of semantic actions.  Whenever the last two parsers
699    merge, reverting to the single-parser case, Bison resolves all the
700    outstanding actions either by precedences given to the grammar rules
701    involved, or by performing both actions, and then calling a designated
702    user-defined function on the resulting values to produce an arbitrary
703    merged result.
704    
705    Let's consider an example, vastly simplified from C++.  
706    
707    @example
708    %@{
709      #define YYSTYPE const char*
710    %@}
711    
712    %token TYPENAME ID
713    
714    %right '='
715    %left '+'
716    
717    %glr-parser
718    
719    %%
720    
721    prog :
722         | prog stmt   @{ printf ("\n"); @}
723         ;
724    
725    stmt : expr ';'  %dprec 1
726         | decl      %dprec 2
727         ;
728    
729    expr : ID               @{ printf ("%s ", $$); @}
730         | TYPENAME '(' expr ')'  
731                            @{ printf ("%s <cast> ", $1); @}
732         | expr '+' expr    @{ printf ("+ "); @}
733         | expr '=' expr    @{ printf ("= "); @}
734         ;
735    
736    decl : TYPENAME declarator ';'
737                            @{ printf ("%s <declare> ", $1); @}
738         | TYPENAME declarator '=' expr ';'
739                            @{ printf ("%s <init-declare> ", $1); @}
740         ;
741    
742    declarator : ID         @{ printf ("\"%s\" ", $1); @}
743         | '(' declarator ')'
744         ;
745    @end example
746    
747    @noindent
748    This models a problematic part of the C++ grammar---the ambiguity between
749    certain declarations and statements.  For example,
750    
751    @example
752    T (x) = y+z;
753    @end example
754    
755    @noindent
756    parses as either an @code{expr} or a @code{stmt}
757    (assuming that @samp{T} is recognized as a TYPENAME and @samp{x} as an ID).
758    Bison detects this as a reduce/reduce conflict between the rules
759    @code{expr : ID} and @code{declarator : ID}, which it cannot resolve at the
760    time it encounters @code{x} in the example above.  The two @code{%dprec}
761    declarations, however, give precedence to interpreting the example as a
762    @code{decl}, which implies that @code{x} is a declarator.
763    The parser therefore prints
764    
765    @example
766    "x" y z + T <init-declare>
767    @end example
768    
769    Consider a different input string for this parser:
770    
771    @example
772    T (x) + y;
773    @end example
774    
775    @noindent
776    Here, there is no ambiguity (this cannot be parsed as a declaration).
777    However, at the time the Bison parser encounters @code{x}, it does not
778    have enough information to resolve the reduce/reduce conflict (again,
779    between @code{x} as an @code{expr} or a @code{declarator}).  In this
780    case, no precedence declaration is used.  Instead, the parser splits
781    into two, one assuming that @code{x} is an @code{expr}, and the other
782    assuming @code{x} is a @code{declarator}.  The second of these parsers
783    then vanishes when it sees @code{+}, and the parser prints
784    
785    @example
786    x T <cast> y +
787    @end example
788    
789    Suppose that instead of resolving the ambiguity, you wanted to see all
790    the possibilities.  For this purpose, we must @dfn{merge} the semantic
791    actions of the two possible parsers, rather than choosing one over the
792    other.  To do so, you could change the declaration of @code{stmt} as
793    follows:
794    
795    @example
796    stmt : expr ';'  %merge <stmtMerge>
797         | decl      %merge <stmtMerge>
798         ;
799    @end example
800    
801    @noindent
802    
803    and define the @code{stmtMerge} function as:
804    
805    @example
806    static YYSTYPE stmtMerge (YYSTYPE x0, YYSTYPE x1)
807    @{
808      printf ("<OR> ");
809      return "";
810    @}
811    @end example
812    
813    @noindent
814    with an accompanying forward declaration
815    in the C declarations at the beginning of the file:
816    
817    @example
818    %@{
819      #define YYSTYPE const char*
820      static YYSTYPE stmtMerge (YYSTYPE x0, YYSTYPE x1);
821    %@}
822    @end example
823    
824    @noindent
825    With these declarations, the resulting parser will parse the first example
826    as both an @code{expr} and a @code{decl}, and print
827    
828    @example
829    "x" y z + T <init-declare> x T <cast> y z + = <OR>
830    @end example
831    
832    
833  @node Locations Overview  @node Locations Overview
834  @section Locations  @section Locations
835  @cindex location  @cindex location
# Line 2913  the location of the grouping (the result Line 3111  the location of the grouping (the result
3111  is an array holding locations of all right hand side elements of the rule  is an array holding locations of all right hand side elements of the rule
3112  being matched. The last one is the size of the right hand side rule.  being matched. The last one is the size of the right hand side rule.
3113    
3114  By default, it is defined this way:  By default, it is defined this way for simple LALR(1) parsers:
3115    
3116  @example  @example
3117  @group  @group
# Line 2925  By default, it is defined this way: Line 3123  By default, it is defined this way:
3123  @end group  @end group
3124  @end example  @end example
3125    
3126    @noindent
3127    and like this for GLR parsers:
3128    
3129    @example
3130    @group
3131    #define YYLLOC_DEFAULT(Current, Rhs, N)          \
3132      Current.first_line   = YYRHSLOC(Rhs,1).first_line;      \
3133      Current.first_column = YYRHSLOC(Rhs,1).first_column;    \
3134      Current.last_line    = YYRHSLOC(Rhs,N).last_line;       \
3135      Current.last_column  = YYRHSLOC(Rhs,N).last_column;
3136    @end group
3137    @end example
3138    
3139  When defining @code{YYLLOC_DEFAULT}, you should consider that:  When defining @code{YYLLOC_DEFAULT}, you should consider that:
3140    
3141  @itemize @bullet  @itemize @bullet
# Line 3890  Return immediately from @code{yyparse}, Line 4101  Return immediately from @code{yyparse},
4101  @findex YYBACKUP  @findex YYBACKUP
4102  Unshift a token.  This macro is allowed only for rules that reduce  Unshift a token.  This macro is allowed only for rules that reduce
4103  a single value, and only when there is no look-ahead token.  a single value, and only when there is no look-ahead token.
4104    It is also disallowed in GLR parsers.
4105  It installs a look-ahead token with token type @var{token} and  It installs a look-ahead token with token type @var{token} and
4106  semantic value @var{value}; then it discards the value that was  semantic value @var{value}; then it discards the value that was
4107  going to be reduced by this rule.  going to be reduced by this rule.
# Line 4030  This kind of parser is known in the lite Line 4242  This kind of parser is known in the lite
4242  * Parser States::     The parser is a finite-state-machine with stack.  * Parser States::     The parser is a finite-state-machine with stack.
4243  * Reduce/Reduce::     When two rules are applicable in the same situation.  * Reduce/Reduce::     When two rules are applicable in the same situation.
4244  * Mystery Conflicts::  Reduce/reduce conflicts that look unjustified.  * Mystery Conflicts::  Reduce/reduce conflicts that look unjustified.
4245    * Generalized LR Parsing::  Parsing arbitrary context-free grammars.
4246  * Stack Overflow::    What happens when stack gets full.  How to avoid it.  * Stack Overflow::    What happens when stack gets full.  How to avoid it.
4247  @end menu  @end menu
4248    
# Line 4624  return_spec: Line 4837  return_spec:
4837          ;          ;
4838  @end example  @end example
4839    
4840    @node Generalized LR Parsing
4841    @section Generalized LR (GLR) Parsing
4842    @cindex GLR parsing
4843    @cindex generalized LR (GLR) parsing
4844    @cindex ambiguous grammars
4845    @cindex non-deterministic parsing
4846    
4847    Bison produces @emph{deterministic} parsers that choose uniquely
4848    when to reduce and which reduction to apply
4849    based on a summary of the preceding input and on one extra token of lookahead.
4850    As a result, normal Bison handles a proper subset of the family of
4851    context-free languages.
4852    Ambiguous grammars, since they have strings with more than one possible
4853    sequence of reductions cannot have deterministic parsers in this sense.
4854    The same is true of languages that require more than one symbol of
4855    lookahead, since the parser lacks the information necessary to make a
4856    decision at the point it must be made in a shift-reduce parser.
4857    Finally, as previously mentioned (@pxref{Mystery Conflicts}),
4858    there are languages where Bison's particular choice of how to
4859    summarize the input seen so far loses necessary information.
4860    
4861    When you use the @samp{%glr-parser} declaration in your grammar file,
4862    Bison generates a parser that uses a different algorithm, called
4863    Generalized LR (or GLR).  A Bison GLR parser uses the same basic
4864    algorithm for parsing as an ordinary Bison parser, but behaves
4865    differently in cases where there is a shift-reduce conflict that has not
4866    been resolved by precedence rules (@pxref{Precedence}) or a
4867    reduce-reduce conflict.  When a GLR parser encounters such a situation, it
4868    effectively @emph{splits} into a several parsers, one for each possible
4869    shift or reduction.  These parsers then proceed as usual, consuming
4870    tokens in lock-step.  Some of the stacks may encounter other conflicts
4871    and split further, with the result that instead of a sequence of states,
4872    a Bison GLR parsing stack is what is in effect a tree of states.  
4873    
4874    In effect, each stack represents a guess as to what the proper parse
4875    is.  Additional input may indicate that a guess was wrong, in which case
4876    the appropriate stack silently disappears.  Otherwise, the semantics
4877    actions generated in each stack are saved, rather than being executed
4878    immediately.  When a stack disappears, its saved semantic actions never
4879    get executed.  When a reduction causes two stacks to become equivalent,
4880    their sets of semantic actions are both saved with the state that
4881    results from the reduction.  We say that two stacks are equivalent
4882    when they both represent the same sequence of states,
4883    and each pair of corresponding states represents a
4884    grammar symbol that produces the same segment of the input token
4885    stream.
4886    
4887    Whenever the parser makes a transition from having multiple
4888    states to having one, it reverts to the normal LALR(1) parsing
4889    algorithm, after resolving and executing the saved-up actions.
4890    At this transition, some of the states on the stack will have semantic
4891    values that are sets (actually multisets) of possible actions.  The
4892    parser tries to pick one of the actions by first finding one whose rule
4893    has the highest dynamic precedence, as set by the @samp{%dprec}
4894    declaration.  Otherwise, if the alternative actions are not ordered by
4895    precedence, but there the same merging function is declared for both
4896    rules by the @samp{%merge} declaration,
4897    Bison resolves and evaluates both and then calls the merge function on
4898    the result.  Otherwise, it reports an ambiguity.
4899    
4900    It is possible to use a data structure for the GLR parsing tree that
4901    permits the processing of any LALR(1) grammar in linear time (in the
4902    size of the input), any unambiguous (not necessarily LALR(1)) grammar in
4903    quadratic worst-case time, and any general (possibly ambiguous)
4904    context-free grammar in cubic worst-case time.  However, Bison currently
4905    uses a simpler data structure that requires time proportional to the
4906    length of the input times the maximum number of stacks required for any
4907    prefix of the input.  Thus, really ambiguous or non-deterministic
4908    grammars can require exponential time and space to process.  Such badly
4909    behaving examples, however, are not generally of practical interest.
4910    Usually, non-determinism in a grammar is local---the parser is ``in
4911    doubt'' only for a few tokens at a time.  Therefore, the current data
4912    structure should generally be adequate.  On LALR(1) portions of a
4913    grammar, in particular, it is only slightly slower than with the default
4914    Bison parser.
4915    
4916  @node Stack Overflow  @node Stack Overflow
4917  @section Stack Overflow, and How to Avoid It  @section Stack Overflow, and How to Avoid It
4918  @cindex stack overflow  @cindex stack overflow
# Line 5912  Equip the parser for debugging.  @xref{D Line 6201  Equip the parser for debugging.  @xref{D
6201  Bison declaration to create a header file meant for the scanner.  Bison declaration to create a header file meant for the scanner.
6202  @xref{Decl Summary}.  @xref{Decl Summary}.
6203    
6204    @item %dprec
6205    Bison declaration to assign a precedence to a rule that is used at parse
6206    time to resolve reduce/reduce conflicts.  @xref{GLR Parsers}.
6207    
6208  @item %file-prefix="@var{prefix}"  @item %file-prefix="@var{prefix}"
6209  Bison declaration to set tge prefix of the output files. @xref{Decl  Bison declaration to set the prefix of the output files. @xref{Decl
6210  Summary}.  Summary}.
6211    
6212    @item %glr-parser
6213    Bison declaration to produce a GLR parser.  @xref{GLR Parsers}.
6214    
6215  @c @item %source-extension  @c @item %source-extension
6216  @c Bison declaration to specify the generated parser output file extension.  @c Bison declaration to specify the generated parser output file extension.
6217  @c @xref{Decl Summary}.  @c @xref{Decl Summary}.
# Line 5928  Summary}. Line 6224  Summary}.
6224  Bison declaration to assign left associativity to token(s).  Bison declaration to assign left associativity to token(s).
6225  @xref{Precedence Decl, ,Operator Precedence}.  @xref{Precedence Decl, ,Operator Precedence}.
6226    
6227    @item %merge
6228    Bison declaration to assign a merging function to a rule.  If there is a
6229    reduce/reduce conflict with a rule having the same merging function, the
6230    function is applied to the two semantic values to get a single result.
6231    @xref{GLR Parsers}.
6232    
6233  @item %name-prefix="@var{prefix}"  @item %name-prefix="@var{prefix}"
6234  Bison declaration to rename the external symbols. @xref{Decl Summary}.  Bison declaration to rename the external symbols. @xref{Decl Summary}.
6235    
# Line 6040  machine.  In the case of the parser, the Line 6342  machine.  In the case of the parser, the
6342  parsed, and the states correspond to various stages in the grammar  parsed, and the states correspond to various stages in the grammar
6343  rules.  @xref{Algorithm, ,The Bison Parser Algorithm }.  rules.  @xref{Algorithm, ,The Bison Parser Algorithm }.
6344    
6345    @item Generalized LR (GLR)
6346    A parsing algorithm that can handle all context-free grammars, including those
6347    that are not LALR(1).  It resolves situations that Bison's usual LALR(1)
6348    algorithm cannot by effectively splitting off multiple parsers, trying all
6349    possible parsers, and discarding those that fail in the light of additional
6350    right context.  @xref{Generalized LR Parsing, ,Generalized LR Parsing}.
6351    
6352  @item Grouping  @item Grouping
6353  A language construct that is (in general) grammatically divisible;  A language construct that is (in general) grammatically divisible;
6354  for example, `expression' or `declaration' in C.  for example, `expression' or `declaration' in C.

Legend:
Removed from v.1.63  
changed lines
  Added in v.1.64

savannah-hackers-public@gnu.org
ViewVC Help
Powered by ViewVC 1.1.26