patchidutils - Patches: patch #5358, scanners for Python and R

 
 

You are not allowed to post comments on this tracker with your current authentication level.

patch #5358: scanners for Python and R

Submitter:  None
Submitted:  Fri 01 Sep 2006 08:39:18 AM UTC
   
 
Category:  None Priority:  5 - Normal
Status:  In Progress Privacy:  Public
Assigned to:  sick_soul Originator Email:  -email is unavailable-
Open/Closed:  Open

Jump to the original submission

Fri 12 Jan 2007 06:08:34 PM UTC, comment #14: 

Hi Claudio,

> Sorry for the long wait. However, the patch will remain pending
> for the new maintainer to continue working on this.
>


OK.

> primarily analyzing exactly why the provided tests fail.
> Guessing is not enough.
>


I am not sure what you mean by "guessing": it is not a guess that the tests fail because "consistency" pipes the output of xtokid to sort, and that messes the output when there are multiline tokens.


Best,

R.

Ramon Diaz-Uriarte <rdiaz02>
Thu 11 Jan 2007 11:23:21 PM UTC, comment #13: 

I am stepping down as maintainer for this package.
Sorry for the long wait. However, the patch will remain pending
for the new maintainer to continue working on this.

There are probably still things to work on,
primarily analyzing exactly why the provided tests fail.
Guessing is not enough.

Secondarily, other problems should be addressed (possibly after
integration?), like those shared with the perl scanner already
discussed, and some remaining commented code.

Claudio

Claudio Fontana <sick_soul>
Fri 01 Dec 2006 10:26:38 PM UTC, comment #12: 

Oops, sorry. I am attaching the version with commented out code removed as needed.

R.

(file #11428)

 Ramon Diaz-Uriarte <rdiaz02>
Tue 14 Nov 2006 11:40:43 PM UTC, comment #11: 

Hello again,
could you remove commented out code from the patch,
and eventually uncomment where necessary?

Thanks,

Claudio


Claudio Fontana <sick_soul>
Sat 04 Nov 2006 10:41:33 AM UTC, comment #10: 

Hi Claudio,

Thanks. No problem (and I am not exactly the speediest myself :-).

 Ramon Diaz-Uriarte <rdiaz02>
Sat 04 Nov 2006 03:48:44 AM UTC, comment #9: 

Hello Ramon,
sorry for being unresponsive, work takes its toll sometimes.
I will look into the issue asap (probably next weekend).

Adding Python support is of particular interest to me,
so fear not, we will sort things out and integrate it.

I do not think we should take any shortcut there;
we'll look at the remaining glitches in detail.

time_t *next;

Claudio Fontana <sick_soul>
Sat 04 Nov 2006 02:49:53 AM UTC, comment #8: 

Claudio, I do not see a simple and reasonable solution, except for modifying the multiline comments in the test file, and addind a starting letter to them, to make sure "sort" will not mess out the output. (I think the consistency check assumes, I think quite reasonably most of the time, that there are no multiline tokens.).

Would this be reasonable? If so, I can do it in a few minutes and send you a new patch.

R.

 Ramon Diaz-Uriarte <rdiaz02>
Sun 29 Oct 2006 09:58:41 PM UTC, comment #7: 

I think I know what is happeing. In both cases, the problem seems to be the ordering of the output. It is not the individual tokens that differ, but where they appear in test-file.{R,py}.bad-{xtokid,fid}.

In the file "consistency" (within the /testsuite dir of idutils) I see that the xtokid output is piped through sort (whereas the fid output seems to be sorted "internally"). I think this might be the cause of the apparent problem: the sort is breaking mutliline tokens (and sorting within a token), whereas fid is doing it "right".

I think this is happening in test-file.py.bad-{xtokid,fid}. The original multiline comment

"""A file that does an atomic-rename to move into place.
    This also causes hardlinks to break when it's written out.
    Open this as for a regular file, then use commit() to move into
    place or abort() to cancel.
    An encoding can be specified; otherwise the default is ascii.
    """

is preserved in fid's output. But the sorted xtokid has all these lines reordered.

So, is it possible that the consitency check, because of the sort of xtokid's output, is leading us to believe there is a problem (when there is none)?

R.

 Ramon Diaz-Uriarte <rdiaz02>
Sun 29 Oct 2006 03:54:43 PM UTC, comment #6: 

The patch looks good,
and I would not worry too much about language keywords in ID.
This can be improved in the future (same for the Perl patch I think).

What troubles me is that after applying the patch,
the python and R scanner seems to fail its own tests:

Oops!  xtokid and fid disagree for test-file.R--look in test-file.R.bad-{xtokid,fid}
Oops!  xtokid and fid disagree for test-file.py--look in test-file.py.bad-{xtokid,fid}

I attach those files.

Claudio

Claudio Fontana <sick_soul>
Sun 08 Oct 2006 03:38:49 PM UTC, comment #5: 

waiting for assignment registration on gnu servers.


Claudio Fontana <sick_soul>
Wed 20 Sep 2006 09:05:11 PM UTC, comment #4: 

Sure, of course. I am willing to assign the copyright to the Free Software Foundation. (And, in case it could matter, this is all coding I have done on my free time, at home, so my employer has nothing to do with it).

 Ramon Diaz-Uriarte <rdiaz02>
Wed 20 Sep 2006 07:59:34 PM UTC, comment #3: 


Hello,
there is a substantial amount of code in your patch,
and since the software is (c) FSF, I must ask you:
would you be willing to assign the copyright to the Free Software Foundation?

Claudio

Claudio Fontana <sick_soul>
Sat 02 Sep 2006 06:34:52 PM UTC, comment #2: 

Fixed bugs with escaped quotes (Python, R) and triple quotes (Python)

 Ramon Diaz-Uriarte <rdiaz02>
Fri 01 Sep 2006 09:35:03 AM UTC, comment #1: 

Submitter is Ramon Diaz-Uriarte <rdiaz02@gmail.com>

 Anonymous
Fri 01 Sep 2006 08:39:18 AM UTC, original submission:  

Adds scanners for Python and R (the "GNU S" statistical computing language).

Both scanners derive from the Perl scanner. The only options allowed are whether or not to tokenize within quotes.

The patch includes some testing scripts in /testsuite.

The patch is a patch against the current cvs version (2006-08-22).

The patch modifies:
    * libidu/scanners.c
    * libidu/id-lang.map

The patch adds:
    * testsuite/Python.VERIFIED
    * testsuite/R.VERIFIED
    * testsuite/README.test.Python.R 
    * testsuite/test-file.R
    * testsuite/test.Python.sh
    * testsuite/Python.wo.quotes.VERIFIED 
    * testsuite/R.wo.quotes.VERIFIED 
    * testsuite/test-file.Python
    * testsuite/test-file.py 
    * testsuite/test.R.sh

 Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #11428:  idutils-R-Python-scanner.patch.bz2 added by rdiaz02 (10KiB - application/octet-stream)
file #11101:  testsuite-test-failed.tgz added by sick_soul (49KiB - application/octet-stream - test failures)
file #10673:  idutils-R-Python-scanner.patch.bz2 added by rdiaz02 (10KiB - application/octet-stream - bzipped patch against current cvs)
file #10660:  idutils-R-Python-scanner.patch.bz2 added by None (10KiB - application/octet-stream - bzipped patched against current cvs)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by sick_soul (Posted a comment)
  • -email is unavailable- added by rdiaz02 (Updated the item)

    •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2006-12-01 rdiaz02 Attached File- Added idutils-R-Python-scanner.patch.bz2, #11428
    2006-10-29 sick_soul Attached File- Added testsuite-test-failed.tgz, #11101
    2006-09-20 sick_soul StatusNone In Progress
    2006-09-10 sick_soul Assigned toNone sick_soul
    2006-09-02 rdiaz02 Attached File- Added idutils-R-Python-scanner.patch.bz2, #10673
        Carbon-Copy- Added rdiaz02
    2006-09-01 None Attached File- Added idutils-R-Python-scanner.patch.bz2, #10660

    Back to the top

    Copyright © 2025 Free Software Foundation, Inc.
    Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.
    The Levitating, Meditating, Flute-playing Gnu logo is a GNU GPL'ed image provided by the Nevrax Design Team.
    Page source code

    Powered by Savane 3.14-04e1.
    Corresponding source code