bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #49843, --csv as output for GNU sql

 
 

bug #49843: --csv as output for GNU sql

Submitted by:  Ole Tange <tange>
Submitted on:  Wed 14 Dec 2016 05:59:52 PM UTC  
 
Category: NoneSeverity: 1 - Wish
Item Group: NoneStatus: Confirmed
Privacy: PublicAssigned to: None
Open/Closed: Open

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Fri 16 Dec 2016 02:08:06 AM UTC, comment #1:

A better approach is to use an SQLite-base and then convert that to CSV:

DBURL=sqlite3:///%2Ftmp%2Fmy.sqlite
DBTABLEURL=$DBURL/mytable
parallel --sqlandworker $DBTABLEURL seq ::: {1..10}
sql $DBURL '.headers on' '.mode csv' 'select * from mytable;'

This takes around a second per job.

If you have access to a real database system, such as PostgreSQL, it is even
faster:

DBURL=pg://user:pass@host/mydb
DBTABLEURL=$DBURL/mytable
parallel --sqlandworker $DBTABLEURL seq ::: {1..10}
sql $DBURL "COPY (SELECT * FROM mytable) TO stdout DELIMITER ',' CSV HEADER;"

Or MySQL:

DBURL=mysql://user:pass@host/mydb
DBTABLEURL=$DBURL/mytable
parallel --sqlandworker $DBTABLEURL seq ::: {1..10}
sql -p -B $DBURL "SELECT * FROM mytable;" > mytable.tsv
perl -pe 's/"/""/g;s/\t/","/g;s/^/"/;s/$/"/;s/\\\\/\\/g;s/\\t/\t/g;s/\\n/\n/g;' mytable.tsv

Ole Tange <tange>
Project Administrator
Wed 14 Dec 2016 05:59:52 PM UTC, original submission:

sql --csv $DBURL/table

Ole Tange <tange>
Project Administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tange (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    No Changes Have Been Made to This Item

    Back to the top


    Powered by Savane 3.1-cleanup1