patchidutils - Patches: patch #5358, scanners for Python and R


You are not allowed to post comments on this tracker with your current authentication level.

patch #5358: scanners for Python and R

Submitted by:  None
Submitted on:  Fri Sep 1 08:39:18 2006  
Category: NonePriority: 5 - Normal
Status: In ProgressPrivacy: Public
Assigned to: Claudio Fontana <sick_soul>Originator Email: -unavailable-
Open/Closed: Open

(Jump to the original submission Jump to the original submission)

Fri Jan 12 18:08:34 2007, comment #14:

Hi Claudio,

> Sorry for the long wait. However, the patch will remain pending
> for the new maintainer to continue working on this.


> primarily analyzing exactly why the provided tests fail.
> Guessing is not enough.

I am not sure what you mean by "guessing": it is not a guess that the tests fail because "consistency" pipes the output of xtokid to sort, and that messes the output when there are multiline tokens.



Ramon Diaz-Uriarte <rdiaz02>
Thu Jan 11 23:23:21 2007, comment #13:

I am stepping down as maintainer for this package.
Sorry for the long wait. However, the patch will remain pending
for the new maintainer to continue working on this.

There are probably still things to work on,
primarily analyzing exactly why the provided tests fail.
Guessing is not enough.

Secondarily, other problems should be addressed (possibly after
integration?), like those shared with the perl scanner already
discussed, and some remaining commented code.


Claudio Fontana <sick_soul>In charge of this item.
Fri Dec 1 22:26:38 2006, comment #12:

Oops, sorry. I am attaching the version with commented out code removed as needed.


(file #11428)

Ramon Diaz-Uriarte <rdiaz02>
Tue Nov 14 23:40:43 2006, comment #11:

Hello again,
could you remove commented out code from the patch,
and eventually uncomment where necessary?



Claudio Fontana <sick_soul>In charge of this item.
Sat Nov 4 10:41:33 2006, comment #10:

Hi Claudio,

Thanks. No problem (and I am not exactly the speediest myself :-).

Ramon Diaz-Uriarte <rdiaz02>
Sat Nov 4 03:48:44 2006, comment #9:

Hello Ramon,
sorry for being unresponsive, work takes its toll sometimes.
I will look into the issue asap (probably next weekend).

Adding Python support is of particular interest to me,
so fear not, we will sort things out and integrate it.

I do not think we should take any shortcut there;
we'll look at the remaining glitches in detail.

time_t *next;

Claudio Fontana <sick_soul>In charge of this item.
Sat Nov 4 02:49:53 2006, comment #8:

Claudio, I do not see a simple and reasonable solution, except for modifying the multiline comments in the test file, and addind a starting letter to them, to make sure "sort" will not mess out the output. (I think the consistency check assumes, I think quite reasonably most of the time, that there are no multiline tokens.).

Would this be reasonable? If so, I can do it in a few minutes and send you a new patch.


Ramon Diaz-Uriarte <rdiaz02>
Sun Oct 29 21:58:41 2006, comment #7:

I think I know what is happeing. In both cases, the problem seems to be the ordering of the output. It is not the individual tokens that differ, but where they appear in test-file.{R,py}.bad-{xtokid,fid}.

In the file "consistency" (within the /testsuite dir of idutils) I see that the xtokid output is piped through sort (whereas the fid output seems to be sorted "internally"). I think this might be the cause of the apparent problem: the sort is breaking mutliline tokens (and sorting within a token), whereas fid is doing it "right".

I think this is happening in{xtokid,fid}. The original multiline comment

"""A file that does an atomic-rename to move into place.
This also causes hardlinks to break when it's written out.
Open this as for a regular file, then use commit() to move into
place or abort() to cancel.
An encoding can be specified; otherwise the default is ascii.

is preserved in fid's output. But the sorted xtokid has all these lines reordered.

So, is it possible that the consitency check, because of the sort of xtokid's output, is leading us to believe there is a problem (when there is none)?


Ramon Diaz-Uriarte <rdiaz02>
Sun Oct 29 15:54:43 2006, comment #6:

The patch looks good,
and I would not worry too much about language keywords in ID.
This can be improved in the future (same for the Perl patch I think).

What troubles me is that after applying the patch,
the python and R scanner seems to fail its own tests:

Oops! xtokid and fid disagree for test-file.R--look in test-file.R.bad-{xtokid,fid}
Oops! xtokid and fid disagree for in{xtokid,fid}

I attach those files.


Claudio Fontana <sick_soul>In charge of this item.
Sun Oct 8 15:38:49 2006, comment #5:

waiting for assignment registration on gnu servers.

Claudio Fontana <sick_soul>In charge of this item.
Wed Sep 20 21:05:11 2006, comment #4:

Sure, of course. I am willing to assign the copyright to the Free Software Foundation. (And, in case it could matter, this is all coding I have done on my free time, at home, so my employer has nothing to do with it).

Ramon Diaz-Uriarte <rdiaz02>
Wed Sep 20 19:59:34 2006, comment #3:

there is a substantial amount of code in your patch,
and since the software is (c) FSF, I must ask you:
would you be willing to assign the copyright to the Free Software Foundation?


Claudio Fontana <sick_soul>In charge of this item.
Sat Sep 2 18:34:52 2006, comment #2:

Fixed bugs with escaped quotes (Python, R) and triple quotes (Python)

Ramon Diaz-Uriarte <rdiaz02>
Fri Sep 1 09:35:03 2006, comment #1:

Submitter is Ramon Diaz-Uriarte <>

Fri Sep 1 08:39:18 2006, original submission:

Adds scanners for Python and R (the "GNU S" statistical computing language).

Both scanners derive from the Perl scanner. The only options allowed are whether or not to tokenize within quotes.

The patch includes some testing scripts in /testsuite.

The patch is a patch against the current cvs version (2006-08-22).

The patch modifies:
* libidu/scanners.c
* libidu/

The patch adds:
* testsuite/Python.VERIFIED
* testsuite/R.VERIFIED
* testsuite/README.test.Python.R
* testsuite/test-file.R
* testsuite/
* testsuite/Python.wo.quotes.VERIFIED
* testsuite/R.wo.quotes.VERIFIED
* testsuite/test-file.Python
* testsuite/
* testsuite/



Attached Files
file #11428:  idutils-R-Python-scanner.patch.bz2 added by rdiaz02 (10kB - application/octet-stream)
file #11101:  testsuite-test-failed.tgz added by sick_soul (49kB - application/octet-stream - test failures)
file #10673:  idutils-R-Python-scanner.patch.bz2 added by rdiaz02 (10kB - application/octet-stream - bzipped patch against current cvs)
file #10660:  idutils-R-Python-scanner.patch.bz2 added by None (10kB - application/octet-stream - bzipped patched against current cvs)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -unavailable- added by sick_soul (Posted a comment)
  • -unavailable- added by rdiaz02 (Updated the item)

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.


    Please enter the title of George Orwell's famous dystopian book (it's a date):



    Follow 7 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Fri Dec 1 22:26:38 2006rdiaz02Attached File-=>Added idutils-R-Python-scanner.patch.bz2, #11428
    Sun Oct 29 15:54:43 2006sick_soulAttached File-=>Added testsuite-test-failed.tgz, #11101
    Wed Sep 20 19:59:34 2006sick_soulStatusNone=>In Progress
    Sun Sep 10 20:12:27 2006sick_soulAssigned toNone=>sick_soul
    Sat Sep 2 18:34:52 2006rdiaz02Attached File-=>Added idutils-R-Python-scanner.patch.bz2, #10673
      Carbon-Copy-=>Added rdiaz02
    Fri Sep 1 08:39:18 2006NoneAttached File-=>Added idutils-R-Python-scanner.patch.bz2, #10660

    Back to the top

    Powered by Savane 3.1-cleanup1