bugPSPP - Bugs: bug #40864, Implement machine-parseable data...


bug #40864: Implement machine-parseable data definition format

Submitted by:  None
Submitted on:  Tue Dec 10 16:07:50 2013  
Category: NoneSeverity: 5 - Average
Status: NoneAssigned to: None
Open/Closed: OpenRelease: None
Effort: 0.00

Add a New Comment (Rich MarkupRich Markup):

You are not logged in

Please log in, so followups can be emailed to you.


(Jump to the original submission Jump to the original submission)

Tue Feb 18 16:40:13 2014, comment #14:

Ok, thanks for the info.

Andre Müller <andrem>
Tue Feb 18 16:31:17 2014, comment #13:

> It seems like the "Label:" marker gets injected before the

quoting is done.

Yes. That's not a bug. The table cell contains both the key and the value, and entire table cells are either quoted or not quoted.

Ben Pfaff <blp>
Project Administrator
Tue Feb 18 11:01:29 2014, comment #12:

I found a nit with the "Label:" marker:

When a label gets quoted, the "Label:" marker is included.
It should not.

This is what I get:
"Label: something ""quoted"" in here."

It should be:
Label: "something ""quoted"" in here."

It seems like the "Label:" marker gets injected before the
quoting is done.

Andre Müller <andrem>
Thu Jan 23 18:12:50 2014, comment #11:

Andre posted an example of the DDI-2.5 format here:


Ben Pfaff <blp>
Project Administrator
Thu Jan 23 05:44:13 2014, comment #10:

Updating title to reflect remaining content in this bug.

Ben Pfaff <blp>
Project Administrator
Sun Dec 22 19:11:38 2013, comment #9:

> @comment5, adding "Label:" to the varlabel:
> This sounds like a good idea. Easy and unambiguous can only help.

I made this change.

Ben Pfaff <blp>
Project Administrator
Wed Dec 11 13:32:28 2013, comment #8:

@comment5, adding "Label:" to the varlabel:

This sounds like a good idea. Easy and unambiguous can only help.
OTOH, this
sed 's/^\([a-zA-Z0-9]\+\t\)\(Format: [AF][0-9.]\+\)\(\t[0-9]\+\)/\1\3\n\t\2 /'
is all it takes to "fix" the empty varlabel for me, converting
<Var> "Format: foo" <Position>
<Var> <Position>
"Format: foo"

Ambiguity has a hard time with this (unless fabricated).


Yes, we will use the DDI-XML as our archive format, and intend to distribute it with the .csv as well.
That said, there are several DDI dialects, as in DDI-2(.x), DDI2-Workbook (which we use for inhouse-transfers), DDI-3 and DDI-3 Lifecycle.

While I dare rule out DDI Lifecycle, as it is overly complex for this task, I intend to develop and propose DDI2- and DDI-3 compliant formats for this.
This will not come to pass this year, but I will happily provide you with the definition we get to, once I get green lights by either a member of the technical committee or the committee itself, let's see where this goes.


But XML being XML, I developed a format for humans to read really easily and do ad-hoc parsing/filtering on.

Some properties:
- Variable name shall be prominently visible
- never use more than one whitespace char bewteen things
- No quoting, ever. Some studies use quotes, brackets and whatnot even in value labels.
- All content is located after the first tab
- A space is only used for offsetting variable attributes (and in content, obviously)
- One attribute per line, so filtering can always operate on a line basis (think grep)
- Attributes are prefixed with "V_", again to allow easy filters/parsers
- A blank line as record separator, again, both for readability and allowing more methods to separate records.

In effect, it introduces redundant unambiguity all around and looks like this:

Var: caseno
V_Label: original respondent number
V_Index: 6
V_Format: F4.0
V_Measure: Scale

Var: v1
V_Label: how important in your life: work (Q1A)
V_Index: 7
V_Format: F1.0
V_Measure: Scale
V_Missing: -5 THRU -1
-5 other missing
-4 question not asked
-3 not applicable
-2 no answer
-1 don't know
1 very important
2 quite important
3 not important
4 not at all important

Yes, so what, another newly invented format. But I consulted a lot of different outputs, and none of them was that friendly.
To my eyes :-)
End of commercial.


@comment6, recutils:

Thanks for pointing out recutils, this looks rather sexy, and has escaped me so far. Will go play :-)

Tue Dec 10 21:55:17 2013, comment #7:

>One format which is very simple and arguably convenient is GNU recutils format.

I was thinking of formats that actually define semantics for metadata, rather than just carriers like CSV or recutils or HTML or what-have-you. DDI-XML as Andre mentioned is an example; Andre, is that your preferred format?

Ben Pfaff <blp>
Project Administrator
Tue Dec 10 21:36:26 2013, comment #6:

One format which is very simple and arguably convenient is GNU recutils format.

Also, I note that the current format is very nearly compliant to recutils format. It could probably be made so with little effort.

John Darrington <jmd>
Project Administrator
Tue Dec 10 20:42:52 2013, comment #5:

I see.

I recently added a tool to the PSPP tree named "pspp-convert", for converting data files among formats. Currently conversions to text formats drop metadata, but in the long term I'd like to be able to output metadata to some convenient format. What format would you find most useful? (I will probably not implement this right away.)

One action I could take immediately, now that I understand the problem, is to prefix the variable label with "Label: " when it is present, to make the format unambiguous. Would that help somewhat?

Ben Pfaff <blp>
Project Administrator
Tue Dec 10 18:03:20 2013, comment #4:

My higher level goal... actually, there's several. Let me digress :-)

I'm working at a social science data archive, and we've got a couple thousand .sav files in our archive which until now are considered the master files.

Now we intend to switch master to plain text files for all data and metadata.

I'm exporting the data and metadata using two fully distinct tools and map the exports on each other to make sure the export is really, positively, absolutely right and reproduces the dataset with all info and errors therein.

I first tried pspp-dump-sav, which looks nice, but it's output is unconditionally latin1-encoded, while I require UTF-8. (There are datasets with 20+ original languages, including non-latin alphabets as greek and russian). So I fell back to DISPLAY DICTIONARY in csv, which looks sane enough. At least, I still feel sane. That said, I'm a complete newbie to PSPP, so it's probable that there's an output format that's better suited.

Further goals are then:
- have a metadata format that's easy both on the eyes and to parse.
- to fix SPSS screwups that assembled by carrying files through the versions for 30+ years and create new ones. (I find them by the dozen by mapping the independent exports.)
- have scripts that convert the metadata to DDI-XML, to SPSS syntax and whatnot (as in STATA, R, prolly SAS, etc).


Tue Dec 10 16:58:19 2013, comment #3:

> Oh, I thought of it as a non-obviuous pitfall for a parser
> and increasing the parser's complexity.

Does this mean that you're writing a parser for the CSV output of DISPLAY DICTIONARY? What's the higher-level goal?

Ben Pfaff <blp>
Project Administrator
Tue Dec 10 16:57:13 2013, comment #2:

Oh, I thought of it as a non-obviuous pitfall for a parser
and increasing the parser's complexity.

But I can see your point. Sorry for not seeing the rationale.


Tue Dec 10 16:11:54 2013, comment #1:

When there's no label, PSPP omits the label. It would seem to be a waste of space to insert a blank line.

Ben Pfaff <blp>
Project Administrator
Tue Dec 10 16:07:50 2013, original submission:

When a variable does not have a label, the output of "DISPLAY DICTIONARY" places the "Format:" line in the label's place.

The call is (separator is a tab, will be mangled):
# /usr/bin/pspp -b -O format=csv -O separator=" " pspp-meta.syntax > demo-no_varlabel.txt

I tried the standard output format as well, it provides the same result.

with pspp-meta.syntax:

GET FILE="demo-no_varlabel.sav"

The output is:

Variable Description Position
Var0001 Format: F1.0 1
Measure: Scale
Display Alignment: Right
Display Width: 8
Var0002 Has Label 2
Format: F1.0
Measure: Scale
Display Alignment: Right
Display Width: 8

Var0002 "Has Label" gets written properly,
Var0001 without a label should be:

Var0001 1
Format: F1.0
Measure: Scale
Display Alignment: Right
Display Width: 8

This happens with version 0.8.1.
I have looked the bug list's archives since August, this at least has not been reported recently.

The "demo-no_varlabel.sav" file is attached.

I will happily provide more info and do testing on request.



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):

Attached Files
file #29854:  demo-no_varlabel.sav added by None (525B - application/spss)
file #29855:  pspp-meta.syntax added by None (51B - application/octet-stream)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -unavailable- added by andrem (Posted a comment)
  • -unavailable- added by jmd (Posted a comment)
  • -unavailable- added by blp (Posted a comment)

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.


    Please enter the title of George Orwell's famous dystopian book (it's a date):



    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu Jan 23 05:44:13 2014blpSummaryWhen a variable has no label, DISPLAY DICTIONARY puts the "Format:" line in it\'s place=>Implement machine-parseable data definition format
    Tue Dec 10 16:07:50 2013NoneAttached File-=>Added demo-no_varlabel.sav, #29854
      Attached File-=>Added pspp-meta.syntax, #29855

    Back to the top

    Powered by Savane 3.1-cleanup1