bugFree Software Directory - Bugs: bug #59761, Some questions about this project

 
 

bug #59761: Some questions about this project

Submitter:  None
Submitted:  Thu 24 Dec 2020 12:02:57 PM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 08 Jan 2021 06:34:27 PM UTC, comment #1: 

original submission:

> I learned that this project is important for automating entry
> additions to the FSD, and that it needs some help for synchroization
> (https://directory.fsf.org/wiki/Free_Software_Directory:Participate/Project_Team#Free_Software_Directory_.28Savannah.29).
>
> So I took a look at the code and understood what it does:
>
> 1. Download package, source and translation_en (descriptions) information from debian / amo
> 2. Process (load) them and save as HDF .h5 files
> 3. For each package, download its copyright and changelogs
> 4. Process (load) the cr and cl
> 5. Parse and export all data obtained from steps 2 and 4 to FSD mediawiki format.
>
> I did a incomplete run of the script (didn't wait for it to list all
> copyright / changelog files as it took too long.
>
> As of [2020-12-24] there were 85453 pacakges, 28489 sources, and 57417
> descriptions in the files downloaded from debian.
>
> To proceed and understand how I can help, I have some questions:
> - How many of these packages are already in FSD?


It looks like around 8,000 entries submitted by wikisysop with 'Debian import' comment field, which is about half of the entries in the directory. But there could be things that were submitted separately that are also in Debian, but there's no clear way to know what those are.

> - Has this script already been used for adding entries to FSD?


Yes, it looks like around 8,000 entries submitted by wikisysop with 'Debian import' comment field.

> - Do we need more sources?


I'm not sure what this means? Like do we want to import from other repos? Possibly in the future, but for now the goal is just the stuff from debian.

> - What does the synchronisation mean? Scheduled execution of the
>   script to achieve synchronisation with sources?


Unfortunately I don't know.

>
> On a lower level, the project would benefit from some clean up,
> including adding an end-to-end test so that one does not have to
> download 100k files to see that it works.


I believe that the reason we haven't done it all is because they were doing it in batches. Like there was one batch in 2013, then another in 2015. I think that likely makes sense going forward as well, to just do a smaller sample of the packages at a time, make sure it goes well, then do another batch.

Donald R Robertson III <donald>
Group administrator
Thu 24 Dec 2020 12:02:57 PM UTC, original submission:  

I learned that this project is important for automating entry
additions to the FSD, and that it needs some help for synchroization
(https://directory.fsf.org/wiki/Free_Software_Directory:Participate/Project_Team#Free_Software_Directory_.28Savannah.29).

So I took a look at the code and understood what it does:

1. Download package, source and translation_en (descriptions) information from debian / amo
2. Process (load) them and save as HDF .h5 files
3. For each package, download its copyright and changelogs
4. Process (load) the cr and cl
5. Parse and export all data obtained from steps 2 and 4 to FSD mediawiki format.

I did a incomplete run of the script (didn't wait for it to list all
copyright / changelog files as it took too long.

As of [2020-12-24] there were 85453 pacakges, 28489 sources, and 57417
descriptions in the files downloaded from debian.

To proceed and understand how I can help, I have some questions:
- How many of these packages are already in FSD?
- Has this script already been used for adding entries to FSD?
- Do we need more sources?
- What does the synchronisation mean? Scheduled execution of the
  script to achieve synchronisation with sources?

On a lower level, the project would benefit from some clean up,
including adding an end-to-end test so that one does not have to
download 100k files to see that it works.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by donald (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code