bugGNU roff - Bugs: bug #65098, merge branch deri-gropdf-ng to...

 
 

bug #65098: merge branch deri-gropdf-ng to master

Submitter:  Deri James <deri>
Submitted:  Sat 30 Dec 2023 11:01:14 PM UTC
   
 
Category:  General Severity:  1 - Wish
Item Group:  Feature change Status:  In Progress
Privacy:  Public Assigned to:  gbranden
Open/Closed:  Open Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 08 Jan 2024 02:30:49 AM UTC, comment #6: 

Deri posted a helpful dependency graph of the new gropdf features.

One update is required: the changes to "pdf.tmac" that expose the new "pdfpagenumbering" convenience macro are merged, but other changes to that file are not (yet).

G. Branden Robinson <gbranden>
Group administrator
Wed 03 Jan 2024 12:11:03 PM UTC, comment #5: 

Made some headway here.


8d320bd7b gropdf(1): Recast `pagenumbering` description.
8dc1df406 [gropdf]: Add `pdfpagenumbering` macro.
7256dfb9f gropdf(1): Update.
fbde1f13b [build]: Increment Perl dependency to 5.8.
77fb2e809 [gropdf]: Add font subsetting and Type 1 parser.


G. Branden Robinson <gbranden>
Group administrator
Wed 03 Jan 2024 12:30:33 AM UTC, comment #4: 

Red Tree
========
I assume you have built my branch at some point, any Titian hues?

Special Pleading (MR)
=====================
You were correct I was bending a general case to try to fit a special case.

For Iterator
============
I can see many uses for this mechanism. One of my design goals for the new gropdf was to allow non-latin languages to use groff to use all the pdf features available to latin languages, particular using the currently available fonts which cover a wide range of languages. The size of these fonts are huge so the current practice of embedding the entire font is probably a non-starter, hence the need for subsetting and being able to use more than 256 glyphs from the font.

Part of this requires unicode to be passed to gropdf, to cater for code such as:-

.pdfbookmark 1 "Chinese text"

Preconv will have converted the chinese text to a group of \[uXXXX] which in turn is converted by troff to nodes, so the for loop will set N for each chinese character and I end up with an empty string. The same problem occurs if chinese is used to set a named destination within the pdf, but it is worse because the destination name is also used in the pdf:look array and if each call results in the empty string, it will not work.

So, I have a few doubts it will completely remove the need for stringhex, but you may find a way.

Cheers

Deri

Deri James <deri>
Group Member
Tue 02 Jan 2024 10:39:19 PM UTC, comment #3: 

comment #2:

> Dealing with gropdf.pl separately, I have cobbled together a document
> which may help you in the task. It is a list of all the diffs with a
> following comment which describes what the changes accomplish. One way
> of handling this is to introduce each file as a separate commit and
> include a description of what the changes in each file achieve (from
> the embedded comments). Note that I have removed from the diff changes
> to master since deri-gropdf-ng was branched.


Yes!  That sounds much more tractable.  Thank you--I'll have a look.

> Gropdf.pl is a slightly different kettle of fish, it has grown by over
> 25% (over 1000 lines), and the number of new lines of code is more
> than that because there has been significant code deletion as well.


Yes.

> You may remember there was a code freeze for over 6 months before
> 1.23.0 and after 3 months I got an "itch" and wrote the new version,
> you then suggested I start a new branch, so I slammed the new version
> into that. I have already written about the two main drivers for the
> change: making gropdf subset fonts in the pdf, and switching from a
> word based input model to an input stream of glyph names (you said
> that my description  was helpful).


Indeed.

> So just as adding a new program to a project does not lead to a raft
> of commits detailing every new function in the code, but rather the
> commit would describe what new features the new program adds to the
> project, it would probably be best to treat the new gropdf in a
> similar way - describing what has been added to groff's capabilities.


True, but gropdf _is_ an existing program, so for maintainability I
think that ideally, we'd handle it the same way as the changes described
in the first paragraph above.  This would also be consistent with other
major changes to groff features we see in the commit and changelog
histories, particularly in the Werner Lemberg era of maintenance.

That said, I don't want to gate the merge of your changes on such a
documentary record.  Perfect being the enemy of the good and all that.

> I'm afraid I did not get your red tree reference, my only experience
> with red/black trees is in "binary chop" and there is a self balancing
> algorithm for that.


Right, sorry about that.  I refer to a convention in continuous
integration (and some other development) systems such that builds are
automated after every commit, and if a build fails on any of the N
supported configurations, the "tree (of source code) goes red", that is,
becomes unacceptable for deployment to production.  Normally, the tree
is "green", and ready to ship any time a manager/release engineer/"Agile
scrum master" elects to do so.

> Maintainability is more concerned with understanding the code as it
> is, the git log may help you understand how it got there but
> understanding what the code actually achieves can only be understood
> by reading the code. For example if I want to know what
> ps_output::put_string does then git log tells me that isascii got
> changed to is_ascii in 1994, but it tells me nothing about what the
> routine does. Maintainability of code is far more about the quality of
> code and the knowledge and experience of the person looking at the
> code.


I think I got lost here with respect to which of these distinct aspects
is more important to maintainability, but I agree that they're both
virtues.

> I've had a rethink about an.tmac and stripped out all the stuff added
> to MR, since it was only there to support internal links to other
> pages within a man page book (such as groff_man_pages.pdf), and I have
> found an alternative way of achieving the same thing without touching
> an.tmac. All that is left is stuff to make it work with the new
> gropdf, don't use \E* do use \\*, and add support for UR/UE and MT/ME.


Oh, that's interesting.  The "special pleading" for `MR` was one of the
things that felt not-quite-baked to me.

> I know you are working on/planning your for iterator, but please don't
> hold up merging my branch to wait for it, since in its current state
> it all works without it.


Agreed.  It is hard for me to predict when any particular thunderbolt
will appear in my bucket ready for hurling.  I thought resolution of bug
#62471 was months in the future for a long time (a year or more?), and
then all of a sudden the fog lifted and, prompted by
a discussion with
Colin Watson and наб in a man-db merge request, I was able to land a fix
with little effort by attacking the problem from a different angle.

> When your code is complete we can revisit which parts are helpful for
> gropdf.


Agreed, and this can likely serve as a form of regression test to boot.

> Just out of interest: what does ch contain if N is set (i.e. it is a
> node).


Stage 1: I was thinking it would return nothing, an empty string.  You'd
just pump the iterator again to get the next thing.

But that's pretty weak sauce because we need a way to signal that
iteration has stopped.  Python raises a StopIteration exception, for
example.  But the *roff language has neither exceptions or even return
values, strictly considered.

So, initially, your iterator "character" would contain nothing if you
had hit a node or the end of the iterand.

That will not actually be unworkable because:

1.  If you use the `for` or `rfor` requests, they will only exit
    (barring a `break`) once iteration has stopped.

2.  I don't at this point imagine a more primitive means of driving an
    iterator, but if that should happen, one can always use the
    `length` request beforehand to measure the iterand, and make sure
    not to pump the iterator more than $length times.

But that seems to me only a little less fiddly than the existing things
we have to do when iterating strings.  So...

Stage 2: Add a new conditional operator, "N".


.ie N ch .tm it's a node
.el      .tm it's not a node


That seems like it should be easy/simple to do.  So if `ch` interpolates
nothing and is not a node, iteration has stopped.

Stage 3: Add a new `pnode` request to dump information about a node to
the standard error output.  Not useful for *roff programming (that is,
it's not really "in the language" per se), but potentially very valuable
for *roff development (including applications).

My instincts tell me I'd want all 3 of these together, so these stages
would not necessarily be distantly separated.  If they prove to be easy,
they might be a sequence of adjacent commits in a single push.

Feedback very much desired.

Regards,
Branden

G. Branden Robinson <gbranden>
Group administrator
Tue 02 Jan 2024 09:06:45 PM UTC, comment #2: 

Dealing with gropdf.pl separately, I have cobbled together a document which may help you in the task. It is a list of all the diffs with a following comment which describes what the changes accomplish. One way of handling this is to introduce each file as a separate commit and include a description of what the changes in each file achieve (from the embedded comments). Note that I have removed from the diff changes to master since deri-gropdf-ng was branched.

Gropdf.pl is a slightly different kettle of fish, it has grown by over 25% (over 1000 lines), and the number of new lines of code is more than that because there has been significant code deletion as well. You may remember there was a code freeze for over 6 months before 1.23.0 and after 3 months I got an "itch" and wrote the new version, you then suggested I start a new branch, so I slammed the new version into that. I have already written about the two main drivers for the change: making gropdf subset fonts in the pdf, and switching from a word based input model to an input stream of glyph names (you said that my description  was helpful). So just as adding a new program to a project does not lead to a raft of commits detailing every new function in the code, but rather the commit would describe what new features the new program adds to the project, it would probably be best to treat the new gropdf in a similar way - describing what has been added to groff's capabilities.

I'm afraid I did not get your red tree reference, my only experience with red/black trees is in "binary chop" and there is a self balancing algorithm for that.

Maintainability is more concerned with understanding the code as it is, the git log may help you understand how it got there but understanding what the code actually achieves can only be understood by reading the code. For example if I want to know what ps_output::put_string does then git log tells me that isascii got changed to is_ascii in 1994, but it tells me nothing about what the routine does. Maintainability of code is far more about the quality of code and the knowledge and experience of the person looking at the code.

I've had a rethink about an.tmac and stripped out all the stuff added to MR, since it was only there to support internal links to other pages within a man page book (such as groff_man_pages.pdf), and I have found an alternative way of achieving the same thing without touching an.tmac. All that is left is stuff to make it work with the new gropdf, don't use \E* do use \\*, and add support for UR/UE and MT/ME.

I know you are working on/planning your for iterator, but please don't hold up merging my branch to wait for it, since in its current state it all works without it. When your code is complete we can revisit which parts are helpful for gropdf. Just out of interest: what does ch contain if N is set (i.e. it is a node).



(file #55511)

Deri James <deri>
Group Member
Sat 30 Dec 2023 11:24:59 PM UTC, comment #1: 

original submission:

> I thought it might be helpful to have a place where we can see
> progress on this.


I agree.  This is difficult because the main blocker I'm experiencing is
finding enough spoons to properly changelog the giant change to
gropdf.pl.  This could also involve dividing it into discrete change
sets (commits), but that seems even more daunting (under the "don't make
the tree go red" principle) and this merge has been held up for a while.

> I have noticed some cherry picking from the branch happening so I'm
> hopeful there will be movement on the rest.


Yes.  I was very happy to see that these bug fixes were independent of
the "next generation" code changes.  :)

> If you are too snowed under at the moment just assign it back to me
> and I can do the merge of the rest.


It's not the merging per se that is the hard part; it is bundling the
merge with enough developer-facing information that we don't degrade the
maintainability of the system.

I guess I could mention that some of the changes on the branch that
aren't to gropdf.pl, like an.tmac-related stuff, is not in the "final
form" I envision it taking.  I'd like to clean up the design (and
implement missing formatter or macro package features where
appropriate).  But as a proof of concept it's fine and I'm glad it's
getting good exercise in the Linux man-pages, with Alejandro's
enthusiastic adoption.  That gives me a good baseline from which I can
measure (and hopefully avoid) regressions.

Now that we have a dependency graph, I think I'd be a good idea if I
created a "1.24 release goals" ticket, and made it depend on this one.

Thoughts?

Regards,
Branden

G. Branden Robinson <gbranden>
Group administrator
Sat 30 Dec 2023 11:01:14 PM UTC, original submission:  

I thought it might be helpful to have a place where we can see progress on this.

I have noticed some cherry picking from the branch happening so I'm hopeful there will be movement on the rest. If you are too snowed under at the moment just assign it back to me and I can do the merge of the rest.

Deri James <deri>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #55511:  diff.html added by deri (60KiB - text/html)

 

Depends on the following items: None found

Items that depend on this one

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by gbranden (Posted a comment)
  • -email is unavailable- added by deri (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-01-03 gbranden StatusNone In Progress
    2024-01-02 deri Attached File- Added diff.html, #55511
    2024-01-01 gbranden CategoryCore General
        Item GroupBuild/Installation Feature change
    2023-12-31 gbranden Dependencies- bugs #65099 is dependent
    2023-12-31 gbranden SummaryMerging branch deri-gropdf-ng with master merge branch deri-gropdf-ng to master

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code