Northern Mike
Posts: 43
Joined: Mon Oct 31, 2016 11:41 am

Large CSV import into MYSQL

Wed Nov 16, 2016 2:43 am

This is partially a PI related issue, and am looking for some suggestions on how to get around this.
Looking to import a large CSV file into MYSQL on my pi3 to be used as part of a project I'm working on.
The issue I'm running into is that the MYSQLIMPORT appears to be hitting the 32bit limits MS Excel use to have, and as a result, I'm only getting 375771 records instead of the 1452138 records contained in the csv.
The file is ~108mb in size. Not huge, no odd characters
My memory may be failing me, but I could have sworn I had exceeded this limit back when I was using Slackware 7.

the command I'm using

Code: Select all

mysqlimport --ignore-lines=1 --fields-terminated-by=',' --local -p celltower /home/pi/extract.csv

User avatar
Paul Webster
Posts: 822
Joined: Sat Jul 30, 2011 4:49 am
Location: London, UK
Contact: Twitter

Re: Large CSV import into MYSQL

Wed Nov 16, 2016 7:15 am

I have not investigated the underlying problem but a workaround would be to split the original file into smaller files and import each individually.
E.g. using "split --lines=30000"

Return to “Advanced users”