original submission:

> I learned that this project is important for automating entry
> additions to the FSD, and that it needs some help for synchroization
> (https://directory.fsf.org/wiki/Free_Software_Directory:Participate/Project_Team#Free_Software_Directory_.28Savannah.29).
>
> So I took a look at the code and understood what it does:
>
> 1. Download package, source and translation_en (descriptions) information from debian / amo
> 2. Process (load) them and save as HDF .h5 files
> 3. For each package, download its copyright and changelogs
> 4. Process (load) the cr and cl
> 5. Parse and export all data obtained from steps 2 and 4 to FSD mediawiki format.
>
> I did a incomplete run of the script (didn't wait for it to list all
> copyright / changelog files as it took too long.
>
> As of [2020-12-24] there were 85453 pacakges, 28489 sources, and 57417
> descriptions in the files downloaded from debian.
>
> To proceed and understand how I can help, I have some questions:
> - How many of these packages are already in FSD?

It looks like around 8,000 entries submitted by wikisysop with 'Debian import' comment field, which is about half of the entries in the directory. But there could be things that were submitted separately that are also in Debian, but there's no clear way to know what those are.

> - Has this script already been used for adding entries to FSD?

Yes, it looks like around 8,000 entries submitted by wikisysop with 'Debian import' comment field.

> - Do we need more sources?

I'm not sure what this means? Like do we want to import from other repos? Possibly in the future, but for now the goal is just the stuff from debian.

> - What does the synchronisation mean? Scheduled execution of the
>   script to achieve synchronisation with sources?

Unfortunately I don't know.

>
> On a lower level, the project would benefit from some clean up,
> including adding an end-to-end test so that one does not have to
> download 100k files to see that it works.

I believe that the reason we haven't done it all is because they were doing it in batches. Like there was one batch in 2013, then another in 2015. I think that likely makes sense going forward as well, to just do a smaller sample of the packages at a time, make sure it goes well, then do another batch.