bugPSPP - Bugs: bug #40864, Implement machine-parseable data...

 
 

bug #40864: Implement machine-parseable data definition format

Submitter:  None
Submitted:  Tue 10 Dec 2013 04:07:50 PM UTC
   
 
Category:  None Severity:  5 - Average
Status:  Invalid Assigned to:  None
Open/Closed:  Closed Release:  None
Effort:  0.00
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 06 May 2019 01:23:49 PM UTC, comment #15: 

Reading the history of this item it seems to me that there is nothing to be done.  Accordingly, I'm closing it.

If that is not the case please re-open it.

John Darrington <jmd>
Group administrator
Tue 18 Feb 2014 04:40:13 PM UTC, comment #14: 

Ok, thanks for the info.

Andre Müller <andrem>
Tue 18 Feb 2014 04:31:17 PM UTC, comment #13: 


> It seems like the "Label:" marker gets injected before the

quoting is done.

Yes.  That's not a bug.  The table cell contains both the key and the value, and entire table cells are either quoted or not quoted.

Ben Pfaff <blp>
Group administrator
Tue 18 Feb 2014 11:01:29 AM UTC, comment #12: 

I found a nit with the "Label:" marker:

When a label gets quoted, the "Label:" marker is included.
It should not.

This is what I get:
"Label: something ""quoted"" in here."

It should be:
Label: "something ""quoted"" in here."

It seems like the "Label:" marker gets injected before the
quoting is done.


Andre Müller <andrem>
Thu 23 Jan 2014 06:12:50 PM UTC, comment #11: 

Andre posted an example of the DDI-2.5 format here:

http://lists.gnu.org/archive/html/bug-gnu-pspp/2014-01/msg00022.html

Ben Pfaff <blp>
Group administrator
Thu 23 Jan 2014 05:44:13 AM UTC, comment #10: 

Updating title to reflect remaining content in this bug.

Ben Pfaff <blp>
Group administrator
Sun 22 Dec 2013 07:11:38 PM UTC, comment #9: 


> @comment5, adding "Label:" to the varlabel:
>
> This sounds like a good idea. Easy and unambiguous can only help.


I made this change.

Ben Pfaff <blp>
Group administrator
Wed 11 Dec 2013 01:32:28 PM UTC, comment #8: 

@comment5, adding "Label:" to the varlabel:

This sounds like a good idea. Easy and unambiguous can only help.
OTOH, this
sed 's/^\([a-zA-Z0-9]\+\t\)\(Format: [AF][0-9.]\+\)\(\t[0-9]\+\)/\1\3\n\t\2 /'
is all it takes to "fix" the empty varlabel for me, converting
<Var>   "Format: foo"  <Position>
to
<Var>           <Position>
        "Format: foo"

Ambiguity has a hard time with this (unless fabricated).


@DDI-XML:

Yes, we will use the DDI-XML as our archive format, and intend to distribute it with the .csv as well.
That said, there are several DDI dialects, as in DDI-2(.x), DDI2-Workbook (which we use for inhouse-transfers), DDI-3 and DDI-3 Lifecycle.

While I dare rule out DDI Lifecycle, as it is overly complex for this task, I intend to develop and propose DDI2- and DDI-3 compliant formats for this.
This will not come to pass this year, but I will happily provide you with the definition we get to, once I get green lights by either a member of the technical committee or the committee itself, let's see where this goes.


Digression:

But XML being XML, I developed a format for humans to read really easily and do ad-hoc parsing/filtering on.

Some properties:
- Variable name shall be prominently visible
- never use more than one whitespace char bewteen things
- No quoting, ever. Some studies use quotes, brackets and whatnot even in value labels.
- All content is located after the first tab
- A space is only used for offsetting variable attributes (and in content, obviously)
- One attribute per line, so filtering can always operate on a line basis (think grep)
- Attributes are prefixed with "V_", again to allow easy filters/parsers
- A blank line as record separator, again, both for readability and allowing more methods to separate records.

In effect, it introduces redundant unambiguity all around and looks like this:

Var:    caseno
 V_Label:       original respondent number
 V_Index:       6
 V_Format:      F4.0
 V_Measure:     Scale

Var:    v1
 V_Label:       how important in your life: work (Q1A)
 V_Index:       7
 V_Format:      F1.0
 V_Measure:     Scale
 V_Missing:     -5 THRU -1
        -5      other missing
        -4      question not asked
        -3      not applicable
        -2      no answer
        -1      don't know
        1       very important
        2       quite important
        3       not important
        4       not at all important

Yes, so what, another newly invented format. But I consulted a lot of different outputs, and none of them was that friendly.
To my eyes :-)
End of commercial.

/Digression.

@comment6, recutils:

Thanks for pointing out recutils, this looks rather sexy, and has escaped me so far. Will go play :-)


Anonymous
Tue 10 Dec 2013 09:55:17 PM UTC, comment #7: 


>One format which is very simple and arguably convenient is GNU recutils format.


I was thinking of formats that actually define semantics for metadata, rather than just carriers like CSV or recutils or HTML or what-have-you.  DDI-XML as Andre mentioned is an example; Andre, is that your preferred format?

Ben Pfaff <blp>
Group administrator
Tue 10 Dec 2013 09:36:26 PM UTC, comment #6: 

One format which is very simple and arguably convenient is GNU recutils format.

Also, I note that the current format is very nearly compliant to recutils format.  It could probably be made so with little effort.

John Darrington <jmd>
Group administrator
Tue 10 Dec 2013 08:42:52 PM UTC, comment #5: 

I see.

I recently added a tool to the PSPP tree named "pspp-convert", for converting data files among formats.  Currently conversions to text formats drop metadata, but in the long term I'd like to be able to output metadata to some convenient format.  What format would you find most useful?  (I will probably not implement this right away.)

One action I could take immediately, now that I understand the problem, is to prefix the variable label with "Label: " when it is present, to make the format unambiguous.  Would that help somewhat?

Ben Pfaff <blp>
Group administrator
Tue 10 Dec 2013 06:03:20 PM UTC, comment #4: 

My higher level goal... actually, there's several. Let me digress :-)

I'm working at a social science data archive, and we've got a couple thousand .sav files in our archive which until now are considered the master files.

Now we intend to switch master to plain text files for all data and metadata.

I'm exporting the data and metadata using two fully distinct tools and map the exports on each other to make sure the export is really, positively, absolutely right and reproduces the dataset with all info and errors therein.

I first tried pspp-dump-sav, which looks nice, but it's output is unconditionally latin1-encoded, while I require UTF-8. (There are datasets with 20+ original languages, including non-latin alphabets as greek and russian). So I fell back to DISPLAY DICTIONARY in csv, which looks sane enough. At least, I still feel sane. That said, I'm a complete newbie to PSPP, so it's probable that there's an output format that's better suited.                                             
 
Further goals are then:
- have a metadata format that's easy both on the eyes and to parse.
- to fix SPSS screwups that assembled by carrying files through the versions for 30+ years and create new ones. (I find them by the dozen by mapping the independent exports.)
- have scripts that convert the metadata to DDI-XML, to SPSS syntax and whatnot (as in STATA, R, prolly SAS, etc).

Thanks,
Andre

Anonymous
Tue 10 Dec 2013 04:58:19 PM UTC, comment #3: 


> Oh, I thought of it as a non-obviuous pitfall for a parser
> and increasing the parser's complexity.


Does this mean that you're writing a parser for the CSV output of DISPLAY DICTIONARY?  What's the higher-level goal?

Ben Pfaff <blp>
Group administrator
Tue 10 Dec 2013 04:57:13 PM UTC, comment #2: 

Oh, I thought of it as a non-obviuous pitfall for a parser
and increasing the parser's complexity.

But I can see your point. Sorry for not seeing the rationale.

Thanks,
Andre

Anonymous
Tue 10 Dec 2013 04:11:54 PM UTC, comment #1: 

When there's no label, PSPP omits the label.  It would seem to be a waste of space to insert a blank line.

Ben Pfaff <blp>
Group administrator
Tue 10 Dec 2013 04:07:50 PM UTC, original submission:  

When a variable does not have a label, the output of "DISPLAY DICTIONARY" places the "Format:" line in the label's place.

The call is (separator is a tab, will be mangled):
# /usr/bin/pspp -b -O format=csv -O separator="   " pspp-meta.syntax > demo-no_varlabel.txt

I tried the standard output format as well, it provides the same result.

with pspp-meta.syntax:

GET FILE="demo-no_varlabel.sav"
DISPLAY DICTIONARY

The output is:

Variable        Description             Position
Var0001 Format: F1.0            1
        Measure: Scale
        Display Alignment: Right
        Display Width: 8
Var0002 Has Label               2
        Format: F1.0
        Measure: Scale
        Display Alignment: Right
        Display Width: 8

Var0002 "Has Label" gets written properly,
Var0001 without a label should be:

Var0001             1
        Format: F1.0
        Measure: Scale
        Display Alignment: Right
        Display Width: 8

This happens with version 0.8.1.
I have looked the bug list's archives since August, this at least has not been reported recently.

The "demo-no_varlabel.sav" file is attached.

I will happily provide more info and do testing on request.


Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #29854:  demo-no_varlabel.sav added by None (525B - application/spss)
file #29855:  pspp-meta.syntax added by None (51B - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by andrem (Posted a comment)
  • -email is unavailable- added by jmd (Posted a comment)
  • -email is unavailable- added by blp (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-05-06 jmd StatusNone Invalid
        Open/ClosedOpen Closed
    2014-01-23 blp SummaryWhen a variable has no label, DISPLAY DICTIONARY puts the &quot;Format:&quot; line in it\'s place Implement machine-parseable data definition format
    2013-12-10 None Attached File- Added demo-no_varlabel.sav, #29854
        Attached File- Added pspp-meta.syntax, #29855

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code