maingperf - Support: sr #110791, Patch from Sei Lisa

 
 

sr #110791: Patch from Sei Lisa

Submitter:  Frediano Ziglio <freddyz77>
Submitted:  Tue 29 Nov 2022 04:01:49 PM UTC
   
 
Category:  None Priority:  5 - Normal
Severity:  3 - Normal Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open Operating System:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 04 Dec 2022 07:19:20 PM UTC, comment #6: 


>
> Indeed, it uses C++ for class inheritance only. It uses the C library and nothing from the C++ library. So, better use 'char *' instead of std::string, and arrays instead of std::vector. I don't think you will need hash tables?
>


At the moment not feel the need to hash tables. It was more lambda functions and std::function.
Could I add xasprintf from GNU libtool or similar string functions ?

> The source structure is a bit unusual, due to the .icc files that contain inline functions. That's needed because (last time I looked) gdb was still causing problems with inline functions (step into, step out, breakpoints).


Honestly I have issues even with plain C nowadays. Did you try "-O0 -g" ? Usually disabling optimization is the solution to debugging issues and breakpoint.

Some starting work at https://github.com/freddy77/gperf/compare/master...golang. Beside some minor patches the main one, splitting C code generation,
is at https://github.com/freddy77/gperf/commit/12d219c097fbcdfc61151020713bfcb53a9a4053. Can you have a look and tell me if it makes sense?

Frediano Ziglio <freddyz77>
Thu 01 Dec 2022 11:25:10 AM UTC, comment #5: 


> I managed to split C generation and generic one having a base class (which mainly has the output method) and some partial Golang generation.


Good, nice! When you have some more, please show it to me. I would like to make the refactoring, and let you concentrate on the Go generation part.

> Indentation is GNU


Yes. While GNU indentation and C++ is not a best fit in some places, let's try to adhere to it, for consistency with other GNU source code.

> it looks the code is using some kind of pre C++-98 which I'm not familiar with. I cannot find any usage of std::string or any other base C++ containers. Can I use them?


Indeed, it uses C++ for class inheritance only. It uses the C library and nothing from the C++ library. So, better use 'char *' instead of std::string, and arrays instead of std::vector. I don't think you will need hash tables?

The source structure is a bit unusual, due to the .icc files that contain inline functions. That's needed because (last time I looked) gdb was still causing problems with inline functions (step into, step out, breakpoints).

Bruno Haible <haible>
Group administrator
Thu 01 Dec 2022 11:00:44 AM UTC, comment #4: 


>
> > Do you think "-L HFN" makes sense or should I try adding Golang support to gperf?
>
> Due to the existence of language-specific command-line options, I don't think the "-L HFN" approach makes much sense. It's better if you try adding Go support directly. Doing so in C++ can not be much harder than doing it in Go. So, the source code structure that I would favour is to have 3 source files
>   output.cc
>   output-c.cc
>   output-go.cc
>
> As for the way to get there, it will take several rounds, because I'd need to split output.cc into output.cc and output-c.cc, and I have not looked in detail yet how that would work out.


At the moment (do not expect a full solution very quickly) I managed to split C generation and generic one having a base class (which mainly has the output method) and some partial Golang generation.

One issue is the syntax and level of compatibility. Indentation is GNU and although not much used to it's not a big concern, but it looks the code is using some kind of pre C++-98 which I'm not familiar with. I cannot find any usage of std::string or any other base C++ containers. Can I use them? I tried to look in the project for some files pointing to the coding style but I didn't find any.

Frediano Ziglio <freddyz77>
Wed 30 Nov 2022 02:47:52 PM UTC, comment #3: 


> Another issue about adding a new language generation to gperf is that possibly different options will have to be added for different languages making the options a bit complicated.


Yes, that is to be expected. Some of the existing options (--readonly-tables, --enum, --includes, --global-table, --pic, --null-strings) are possibly no-ops for other languages.

> For instance I would add a --utf8.


Yes, such an option that declares the encoding of the input source file would be useful, also for the existing C/C++ case. But certainly there can also be command-line options that are Go specific.

> Do you think "-L HFN" makes sense or should I try adding Golang support to gperf?


Due to the existence of language-specific command-line options, I don't think the "-L HFN" approach makes much sense. It's better if you try adding Go support directly. Doing so in C++ can not be much harder than doing it in Go. So, the source code structure that I would favour is to have 3 source files
  output.cc
  output-c.cc
  output-go.cc

As for the way to get there, it will take several rounds, because I'd need to split output.cc into output.cc and output-c.cc, and I have not looked in detail yet how that would work out.

Bruno Haible <haible>
Group administrator
Tue 29 Nov 2022 09:12:57 PM UTC, comment #2: 

comment #1:

> > Why not integrating other languages instead?
>
> Changing the code to support other output languages than C/C++ would be possible. It would require some serious refactoring of output.cc, though.
>
> In https://lists.gnu.org/archive/html/bug-gperf/2016-11/msg00003.html I objected against a proposal that interprets the resulting hash table parameters. I don't think it makes sense to have a super-optimizing gperf as a first pass, followed by an inherently slow interpreter as a second pass. If someone doesn't want the hash table lookup to be really fast, they can just as well use some general-purpose hash functions (defined in some library) and insert all the elements during the initialization routine of the program.
>


Oh... now I understand why the objection. No, my approach is different. Yes, I interpret the output of "-L HFN" but then I use that output to generate Golang source code (like output.cc does for C and C++).

Yes, I tried changing output.cc and it was not that easy, so I tried the "HFN" approach. Why using Golang to generate Golang code? In the end using Golang was something easier, for instance the strings follows the same behaviour as Golang. I had some doubts about different encoding... but I suppose UTF-8 should work the same for both C++ and Golang (Golang is somehow cabled into UTF-8 but with length encoded strings instead of 0-terminated strings).

The generated Golang code uses a long string with all keywords, an array of offsets and "wordlist" just contain the index of the keywords, given an input like

  one
  two

something like the following is generated

  const keywords = "onetwo"
  var indexes = [...]uint16{0, 3, 6}
  var wordlist = [...]int16{-1, 0, -1, -1, 1}

  func computedHash(keyword string) uint32 {
     ...
  }

so I can do something like

  found := false
  hash := computeHash(keyword)
  if hash < uint32(len(wordlist)) {
    idx := wordlist[hash]
    if idx >= 0 {
      k := keywords[indexes[idx]:indexes[idx+1]]
      if k == keyword {
        found = true
      }
    }
  }

Not sure how easy is to generate the long string and offsets in a portable way from C++ unless you do some assumptions. In another project I used gperf to generate C code with the --pic options and a large structure using lot of sizeofs is generated but to generate Golang code I possibly have to assume that bytes in C++ are the same they would end up in Golang.

Another issue about adding a new language generation to gperf is that possibly different options will have to be added for different languages making the options a bit complicated. For instance I would add a --utf8.

What do you think? Do you think "-L HFN" makes sense or should I try adding Golang support to gperf?

> > I think that program could be also split into lookup generation and code generation.
>
> Semantically, gperf operates like this already, yes.
>
> > This way you can generate code again from the same set of strings without having to go through the slow lookup generation.
>
> The use-case would be to run gperf again, with the same input, with some of the "Options for fine tuning Details in the Output Code" changed, so that the table structure from the previous run could be reused. This would be possible. But the added complexity (when is that temporary file created, when is it reused, when is it deleted?) and its associated hassles for the user is not worth the few saved seconds of computation time.
>


I suppose that, yes, by default you can avoid -m option and make first phase relatively quick and then use a more "strong" -m option for final version.

Frediano Ziglio <freddyz77>
Tue 29 Nov 2022 06:14:18 PM UTC, comment #1: 


> Why not integrating other languages instead?


Changing the code to support other output languages than C/C++ would be possible. It would require some serious refactoring of output.cc, though.

In https://lists.gnu.org/archive/html/bug-gperf/2016-11/msg00003.html I objected against a proposal that interprets the resulting hash table parameters. I don't think it makes sense to have a super-optimizing gperf as a first pass, followed by an inherently slow interpreter as a second pass. If someone doesn't want the hash table lookup to be really fast, they can just as well use some general-purpose hash functions (defined in some library) and insert all the elements during the initialization routine of the program.

> I think that program could be also split into lookup generation and code generation.


Semantically, gperf operates like this already, yes.

> This way you can generate code again from the same set of strings without having to go through the slow lookup generation.


The use-case would be to run gperf again, with the same input, with some of the "Options for fine tuning Details in the Output Code" changed, so that the table structure from the previous run could be reused. This would be possible. But the added complexity (when is that temporary file created, when is it reused, when is it deleted?) and its associated hassles for the user is not worth the few saved seconds of computation time.

Bruno Haible <haible>
Group administrator
Tue 29 Nov 2022 04:01:49 PM UTC, original submission:  

In some internal project I'm trying to use GNU gperf to generate code for Golang. After a quick attempt to change code generation code in output.cc I found a fork from Sei Lisa at https://github.com/Sei-Lisa/gperf-hfn, specifically the commit at https://github.com/Sei-Lisa/gperf-hfn/commit/5069a9ece3662e1dd652b9845c7fffc207dd0b5e.

I saw the change was discussed at https://lists.gnu.org/archive/html/bug-gperf/2016-11/msg00000.html.

Personally I think that the change is pretty useful and should be integrated. That allows to use GNU gperf for languages not supported. Why not integrating other languages instead? One of the reason is that it could clutter GNU gperf sources with languages used rarely. Another is that there could be generators written in other languages which are easier to use.

Personally I think that program could be also split into lookup generation and code generation. This way you can generate code again from the same set of strings without having to go through the slow lookup generation.

Frediano Ziglio <freddyz77>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by haible (Posted a comment)
  • -email is unavailable- added by freddyz77 (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code