bugPSPP - Bugs: bug #61809, Import dialog: Preview shows...

 
 

bug #61809: Import dialog: Preview shows correct data but import results in empty data view (Text Delimeter)

Submitted by:  Friedrich Beckmann <beckmanf>
Submitted on:  Thu 13 Jan 2022 10:41:13 AM UTC
 
Category:  Graphical User Interface Severity:  1 - Wishlist
Status:  Confirmed Assigned to:  None
Open/Closed:  Open Release:  None
Effort:  0.00
* Mandatory Fields

Add a New Comment (Rich Markup)
   

Comment Type & Canned Response:
       

 

Mon 11 Apr 2022 02:29:38 AM UTC, comment #3: 

The assistant also guesses the wrong delimiters for this file.  The heuristic that the code uses to guess delimiters is to count the number of times each potential delimiter occurs on each line, then select the delimiter whose occurrences are most consistent.  The idea is that, if some delimiter always occurs the same number of times, then it's probably the delimiter, even if it didn't occur the greatest number of times overall.

For this file, by applying the following patch:

diff --git a/src/ui/gui/psppire-import-textfile.c b/src/ui/gui/psppire-import-textfile.c
index f73280b66b..a58cddb235 100644
--- a/src/ui/gui/psppire-import-textfile.c
+++ b/src/ui/gui/psppire-import-textfile.c
@@ -145,6 +145,8 @@ choose_likely_separators (PsppireImportAssistant *ia)
           struct separator_count_node *next;
           HMAP_FOR_EACH_SAFE (cn, next, struct separator_count_node, node, &count_map[j])
             {
+              printf ("%d occurrences of %s happened %d times\n",
+                      cn->occurance, separators[j].name, cn->quantity);
               if (largest < cn->quantity)
                 {
                   largest = cn->quantity;

we see that colon always occurs the same number of times (6 times on every line) and thus it gets chosen. The file has a lot more semicolons but the number varies slightly between 87 and 88:

14 occurrences of space happened 2 times
19 occurrences of space happened 2 times
15 occurrences of space happened 1 times
6 occurrences of colon happened 5 times
12 occurrences of comma happened 1 times
9 occurrences of comma happened 2 times
7 occurrences of comma happened 2 times
12 occurrences of hyphen happened 2 times
11 occurrences of hyphen happened 3 times
88 occurrences of semicolon happened 2 times
87 occurrences of semicolon happened 3 times
4 occurrences of slash happened 3 times
5 occurrences of slash happened 2 times

Maybe we can find a better heuristic.

Ben Pfaff <blp>
Project Administrator
Wed 06 Apr 2022 01:12:23 PM UTC, comment #2: 

It might be related to localization. I can reproduce the problem with german localization. I select ";" as field delimiter manually during the import process. Then most variables are identified as "numeric" and everything seems o.k. in the preview. But then during import I see these warnings:

/Users/fritz/Downloads/data_bug.csv:2.1-2.4: Warnung: Die Daten für die Variable lfdn sind im Format F ungültig: Feldinhalte sind nicht numerisch.
/Users/fritz/Downloads/data_bug.csv:2.6-2.8: Warnung: Die Daten für die Variable external_lfdn sind im Format F ungültig: Feldinhalte sind nicht numerisch.
/Users/fritz/Downloads/data_bug.csv:2.10-2.12: Warnung: Die Daten für die Variable tester sind im Format F ungültig: Feldinhalte sind nicht numerisch.
...

The log window shows the following import command:

GET DATA
  /TYPE=TXT
  /FILE="/Users/fritz/Downloads/data_bug.csv"
  /ARRANGEMENT=DELIMITED
  /DELCASE=LINE
  /FIRSTCASE=2
  /DELIMITERS=";"
  /VARIABLES=
    lfdn F2.0
    external_lfdn F1.0
    tester F1.0
    dispcode F2.0
    lastpage F7.0
    quality F3.0
    duration F5.0
    v_1 F2.0
    v_2 F1.0
    v_3 F1.0
    v_4 F1.0
    v_10 F1.0
    v_11 F1.0
    v_12 F1.0
    v_15 F1.0
    v_67 F1.0
    v_16 F1.0
    v_68 F1.0
    v_17 F1.0
    v_69 F1.0
    v_25 F1.0
    v_26 F1.0
    v_27 F1.0
    v_28 F1.0
    v_29 F1.0
    v_30 F1.0
    v_31 F1.0
    v_32 F1.0
    v_33 F1.0
    v_34 F1.0
    v_35 F1.0
    v_36 F1.0
    v_37 F1.0
    v_38 F1.0
    v_39 F1.0
    v_40 F1.0
    v_41 F1.0
    v_42 F1.0
    v_43 F1.0
    v_44 F1.0
    v_45 F1.0
    v_46 F1.0
    v_47 F1.0
    v_48 F1.0
    v_49 F1.0
    v_50 F1.0
    v_51 F1.0
    v_52 F1.0
    v_53 F1.0
    v_54 F1.0
    v_55 F1.0
    v_56 F1.0
    v_57 F1.0
    v_58 F1.0
    v_59 F1.0
    v_60 F1.0
    v_61 F1.0
    v_62 F1.0
    v_63 F1.0
    v_64 F1.0
    v_65 F1.0
    v_66 F1.0
    browser A135
    referer F3.0
    device_type F1.0
    quota F1.0
    quota_assignment F3.0
    quota_rejected_id F1.0
    page_history DOT40.0
    hflip F1.0
    vflip F1.0
    output_mode F1.0
    javascript F3.0
    flash F3.0
    session_id A32
    language F1.0
    cleaned F1.0
    ats F10.0
    datetime YMDHMS19.0
    date_of_last_access YMDHMS19.0
    date_of_first_mail DATETIME20.0
    rts5542751 F5.0
    rts5542786 F5.0
    rts5542805 F5.0
    rts5542806 F5.0
    rts5542847 F5.0
    rts5591843 F5.0.

When I check the the ";" as delimiter and I check " as quoting character, then the import works. The working import command is then

GET DATA
  /TYPE=TXT
  /FILE="/Users/fritz/Downloads/data_bug.csv"
  /ARRANGEMENT=DELIMITED
  /DELCASE=LINE
  /FIRSTCASE=2
  /DELIMITERS=";"
  /QUALIFIER='"'
  /VARIABLES=
    lfdn F2.0
    external_lfdn F1.0
    tester F1.0
    dispcode F2.0
    lastpage F7.0
    quality F3.0
    duration F5.0
    v_1 F2.0
    v_2 F1.0
    v_3 F1.0
    v_4 F1.0
    v_10 F1.0
    v_11 F1.0
    v_12 F1.0
    v_15 F1.0
    v_67 F1.0
    v_16 F1.0
    v_68 F1.0
    v_17 F1.0
    v_69 F1.0
    v_25 F1.0
    v_26 F1.0
    v_27 F1.0
    v_28 F1.0
    v_29 F1.0
    v_30 F1.0
    v_31 F1.0
    v_32 F1.0
    v_33 F1.0
    v_34 F1.0
    v_35 F1.0
    v_36 F1.0
    v_37 F1.0
    v_38 F1.0
    v_39 F1.0
    v_40 F1.0
    v_41 F1.0
    v_42 F1.0
    v_43 F1.0
    v_44 F1.0
    v_45 F1.0
    v_46 F1.0
    v_47 F1.0
    v_48 F1.0
    v_49 F1.0
    v_50 F1.0
    v_51 F1.0
    v_52 F1.0
    v_53 F1.0
    v_54 F1.0
    v_55 F1.0
    v_56 F1.0
    v_57 F1.0
    v_58 F1.0
    v_59 F1.0
    v_60 F1.0
    v_61 F1.0
    v_62 F1.0
    v_63 F1.0
    v_64 F1.0
    v_65 F1.0
    v_66 F1.0
    browser A135
    referer F3.0
    device_type F1.0
    quota F1.0
    quota_assignment F3.0
    quota_rejected_id F1.0
    page_history DOT40.0
    hflip F1.0
    vflip F1.0
    output_mode F1.0
    javascript F3.0
    flash F3.0
    session_id A32
    language F1.0
    cleaned F1.0
    ats F10.0
    datetime YMDHMS19.0
    date_of_last_access YMDHMS19.0
    date_of_first_mail DATETIME20.0
    rts5542751 F5.0
    rts5542786 F5.0
    rts5542805 F5.0
    rts5542806 F5.0
    rts5542847 F5.0
    rts5591843 F5.0.

The difference is the "QUALIFIER" setting. But the "QUALIFIER" is not used during preview, i.e. preview looks o.k. but the import fails.

Friedrich Beckmann <beckmanf>
Project Member
Tue 05 Apr 2022 03:00:54 PM UTC, comment #1: 

I just tried to reproduce this problem.  I do see some problems with the import process: it picks the wrong delimiter (colon instead of semicolon) and it fails to pick " as a quoting character.  But I don't see the particular problem described. That is, the final output looks the same to me as the preview.  I also don't a checkbox labeled "Text Delimiter".  So, I need some more guidance in looking at this bug.

Ben Pfaff <blp>
Project Administrator
Thu 13 Jan 2022 10:41:13 AM UTC, original submission:  

Niklas had some problems when he does a data import of the attached data file. When you do not check the "Text Delimiter" checkbox in the import dialog, then the data is shown correctly in the preview window, but the following data import fails because the data is not recognized. The numerical data is encapsulated with ". The import fails because the data is not recognized as numerical data.

The expected behaviour would be that either the import dialog data preview should not be correct or the import works.

When the "Text Delimeter" checkbox is checked, then the import works.

Friedrich Beckmann <beckmanf>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #52658:  data_bug.csv added by beckmanf (4KiB - text/csv)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by blp (Posted a comment)
  • -email is unavailable- added by beckmanf (Submitted the item)
  • -email is unavailable- added by beckmanf (Niklas reported this bug)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

     

     

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-04-06 beckmanf StatusNone => Confirmed
    2022-01-13 beckmanf Attached File- => Added data_bug.csv, #52658
        Carbon-Copy- => Added -email is unavailable-

    Back to the top


    Powered by Savane 3.9