I have csv file which minimum size 15 GB. I have to upload the data from file to mysql tables. I created the tables and I used the Load data infile command. It worked well on small size file of 50MB but when i tried the original file it took long time (more then 12 hours) so I pressed ctrl+c. now no row is updated.
I would suggest looking at this : http://stackoverflow.com/questions/6258717/myslqimport-use-threads
Plus: splitting your input fie in manageable chunks (e.g. 100 MB each) and running multiple imports in parallel onto different tables. You can even configure the storage space on different disks to prevent disk IO bottlenecks.
I would suggest looking at this : http://stackoverflow.com/questions/6258717/myslqimport-use-threads
Plus: splitting your input fie in manageable chunks (e.g. 100 MB each) and running multiple imports in parallel onto different tables. You can even configure the storage space on different disks to prevent disk IO bottlenecks.
I think that would be good way but I dont know how to break the files and manage it.
i have attached the screen shot
You can use ETL based tools for bulk loading. In open source can go for Kettle or Talend.
1. KETTLE - http://community.pentaho.com/projects/data-integration/
2. Talend - https://www.talend.com/
Thank you Dipankar sengupta but Keetle and Talend are commercial tools. I cannot afford to buy commercial softwares. please tell me will it help me to upload to MySQL?
You don't need to purchase it. I have been using KETTLE since last 4 years for processing large files to MySQL db. You can also process and clean the data if required. Download it from the link provided (you don't need to create account in Sourceforge for it).
Thank you very much Prof. Dipankar Sengupta. I am new to big data. I read weka is good for data mining and other things. Could you please direct me where to look for materials to read for data mining and visualize bid data. using MySQL,Python, R and other software.(I know little bit of MySQL, Python, R). My project related to Algorithm trading. I have to work on data of minimum of 300 GB.
To split a file in pieces is natively supported on linux and macOS
http://linux.die.net/man/1/split
For windows you may want to download the open source split: http://sourceforge.net/projects/fsplit/
Weka is not ideal for learning big data as the processing is all done in memory.
You should look at Hadoop / map reduce type of setups using distributed processing.
http://www.cs.waikato.ac.nz/ml/weka/bigdata.html
Another option is to obtain a licence for Pentaho
http://www.pentaho.com/product/big-data-analytics
my best, Peter
Thank you Dr.Peter Goedtkindt . I tried split it works but My professor gave me more than 500 files like this. what should i do? should i have to always break every file and then upload for processing. then processing multiple table. I think that will take lot of time and I have only one week to finish this.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Programming Languages
- Python