patchidutils - Patches: patch #5358, scanners for Python and R

 
 

You are not allowed to post comments on this tracker with your current authentification level.

patch #5358: scanners for Python and R

Submitted by:  None
Submitted on:  Fri 01 Sep 2006 08:39:18 AM UTC  
 
Category: NonePriority: 5 - Normal
Status: In ProgressPrivacy: Public
Assigned to: Claudio Fontana <sick_soul>Originator Email: -unavailable-
Open/Closed: Open

(Jump to the original submission Jump to the original submission)

Fri 12 Jan 2007 06:08:34 PM UTC, comment #14:

Hi Claudio,

> Sorry for the long wait. However, the patch will remain pending
> for the new maintainer to continue working on this.
>


OK.

> primarily analyzing exactly why the provided tests fail.
> Guessing is not enough.
>


I am not sure what you mean by "guessing": it is not a guess that the tests fail because "consistency" pipes the output of xtokid to sort, and that messes the output when there are multiline tokens.

Best,

R.

Ramon Diaz-Uriarte <rdiaz02>
Thu 11 Jan 2007 11:23:21 PM UTC, comment #13:

I am stepping down as maintainer for this package.
Sorry for the long wait. However, the patch will remain pending
for the new maintainer to continue working on this.

There are probably still things to work on,
primarily analyzing exactly why the provided tests fail.
Guessing is not enough.

Secondarily, other problems should be addressed (possibly after
integration?), like those shared with the perl scanner already
discussed, and some remaining commented code.

Claudio

Claudio Fontana <sick_soul>In charge of this item.
Fri 01 Dec 2006 10:26:38 PM UTC, comment #12:

Oops, sorry. I am attaching the version with commented out code removed as needed.

R.

(file #11428)

Ramon Diaz-Uriarte <rdiaz02>
Tue 14 Nov 2006 11:40:43 PM UTC, comment #11:

Hello again,
could you remove commented out code from the patch,
and eventually uncomment where necessary?

Thanks,

Claudio

Claudio Fontana <sick_soul>In charge of this item.
Sat 04 Nov 2006 10:41:33 AM UTC, comment #10:

Hi Claudio,

Thanks. No problem (and I am not exactly the speediest myself :-).

Ramon Diaz-Uriarte <rdiaz02>
Sat 04 Nov 2006 03:48:44 AM UTC, comment #9:

Hello Ramon,
sorry for being unresponsive, work takes its toll sometimes.
I will look into the issue asap (probably next weekend).

Adding Python support is of particular interest to me,
so fear not, we will sort things out and integrate it.

I do not think we should take any shortcut there;
we'll look at the remaining glitches in detail.

time_t *next;

Claudio Fontana <sick_soul>In charge of this item.
Sat 04 Nov 2006 02:49:53 AM UTC, comment #8:

Claudio, I do not see a simple and reasonable solution, except for modifying the multiline comments in the test file, and addind a starting letter to them, to make sure "sort" will not mess out the output. (I think the consistency check assumes, I think quite reasonably most of the time, that there are no multiline tokens.).

Would this be reasonable? If so, I can do it in a few minutes and send you a new patch.

R.

Ramon Diaz-Uriarte <rdiaz02>
Sun 29 Oct 2006 09:58:41 PM UTC, comment #7:

I think I know what is happeing. In both cases, the problem seems to be the ordering of the output. It is not the individual tokens that differ, but where they appear in test-file.{R,py}.bad-{xtokid,fid}.

In the file "consistency" (within the /testsuite dir of idutils) I see that the xtokid output is piped through sort (whereas the fid output seems to be sorted "internally"). I think this might be the cause of the apparent problem: the sort is breaking mutliline tokens (and sorting within a token), whereas fid is doing it "right".

I think this is happening in test-file.py.bad-{xtokid,fid}. The original multiline comment

"""A file that does an atomic-rename to move into place.
This also causes hardlinks to break when it's written out.
Open this as for a regular file, then use commit() to move into
place or abort() to cancel.
An encoding can be specified; otherwise the default is ascii.
"""

is preserved in fid's output. But the sorted xtokid has all these lines reordered.

So, is it possible that the consitency check, because of the sort of xtokid's output, is leading us to believe there is a problem (when there is none)?

R.

Ramon Diaz-Uriarte <rdiaz02>
Sun 29 Oct 2006 03:54:43 PM UTC, comment #6:

The patch looks good,
and I would not worry too much about language keywords in ID.
This can be improved in the future (same for the Perl patch I think).

What troubles me is that after applying the patch,
the python and R scanner seems to fail its own tests:

Oops! xtokid and fid disagree for test-file.R--look in test-file.R.bad-{xtokid,fid}
Oops! xtokid and fid disagree for test-file.py--look in test-file.py.bad-{xtokid,fid}

I attach those files.

Claudio

Claudio Fontana <sick_soul>In charge of this item.
Sun 08 Oct 2006 03:38:49 PM UTC, comment #5:

waiting for assignment registration on gnu servers.

Claudio Fontana <sick_soul>In charge of this item.
Wed 20 Sep 2006 09:05:11 PM UTC, comment #4:

Sure, of course. I am willing to assign the copyright to the Free Software Foundation. (And, in case it could matter, this is all coding I have done on my free time, at home, so my employer has nothing to do with it).

Ramon Diaz-Uriarte <rdiaz02>
Wed 20 Sep 2006 07:59:34 PM UTC, comment #3:

Hello,
there is a substantial amount of code in your patch,
and since the software is (c) FSF, I must ask you:
would you be willing to assign the copyright to the Free Software Foundation?

Claudio

Claudio Fontana <sick_soul>In charge of this item.
Sat 02 Sep 2006 06:34:52 PM UTC, comment #2:

Fixed bugs with escaped quotes (Python, R) and triple quotes (Python)

Ramon Diaz-Uriarte <rdiaz02>
Fri 01 Sep 2006 09:35:03 AM UTC, comment #1:

Submitter is Ramon Diaz-Uriarte <rdiaz02@gmail.com>

Anonymous
Fri 01 Sep 2006 08:39:18 AM UTC, original submission:

Adds scanners for Python and R (the "GNU S" statistical computing language).

Both scanners derive from the Perl scanner. The only options allowed are whether or not to tokenize within quotes.

The patch includes some testing scripts in /testsuite.

The patch is a patch against the current cvs version (2006-08-22).

The patch modifies:
* libidu/scanners.c
* libidu/id-lang.map

The patch adds:
* testsuite/Python.VERIFIED
* testsuite/R.VERIFIED
* testsuite/README.test.Python.R
* testsuite/test-file.R
* testsuite/test.Python.sh
* testsuite/Python.wo.quotes.VERIFIED
* testsuite/R.wo.quotes.VERIFIED
* testsuite/test-file.Python
* testsuite/test-file.py
* testsuite/test.R.sh

Anonymous

 

Attached Files
file #11428:  idutils-R-Python-scanner.patch.bz2 added by rdiaz02 (10kB - application/octet-stream)
file #11101:  testsuite-test-failed.tgz added by sick_soul (49kB - application/octet-stream - test failures)
file #10673:  idutils-R-Python-scanner.patch.bz2 added by rdiaz02 (10kB - application/octet-stream - bzipped patch against current cvs)
file #10660:  idutils-R-Python-scanner.patch.bz2 added by None (10kB - application/octet-stream - bzipped patched against current cvs)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by sick_soul (Posted a comment)
  • -unavailable- added by rdiaz02 (Updated the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 7 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Fri 01 Dec 2006 10:26:38 PM UTCrdiaz02Attached File-=>Added idutils-R-Python-scanner.patch.bz2, #11428
    Sun 29 Oct 2006 03:54:43 PM UTCsick_soulAttached File-=>Added testsuite-test-failed.tgz, #11101
    Wed 20 Sep 2006 07:59:34 PM UTCsick_soulStatusNone=>In Progress
    Sun 10 Sep 2006 08:12:27 PM UTCsick_soulAssigned toNone=>sick_soul
    Sat 02 Sep 2006 06:34:52 PM UTCrdiaz02Attached File-=>Added idutils-R-Python-scanner.patch.bz2, #10673
      Carbon-Copy-=>Added rdiaz02
    Fri 01 Sep 2006 08:39:18 AM UTCNoneAttached File-=>Added idutils-R-Python-scanner.patch.bz2, #10660

    Back to the top


    Powered by Savane 3.1-cleanup