Splitting your database seems like a very messy approach. Many database technologies support data access out of memory, and can also be very quick if your data is structured properly. I would suggest looking at HDF5 or something similar. Python has some good tools for this work with PyTables and Pandas. Good luck. http://stackoverflow.com/questions/14262433/large-data-work-flows-using-pandas 

If you are on *nix platform (Max OSX, Linux), you can use split command line utility. Write a simple Bash/Python script around this utility to automate the task of splitting.

$ split --help

Alternatively, you can use command line interfaces of 7z, or other freely available tools.

I subscribe Vivek answer together with some other suggestions. The problem you're facing isn't new. If you look at it in google (like "fastest way to split files") you'll see there are several software to do the job (free example: http://www.jaist.ac.jp/~hoangle/filesj/)

8 to 10 hours seems like an awful lot of time to process 5 GB. I'm guessing you're doing it in Python or R. Is it ASCII or binary? Does it have a split criteria (like data blocks) or can you just split in every "n" bytes?

Is the data on any known file format? If so look for API specifically for that format. It may already have what you are looking for.

http://www.jaist.ac.jp/~hoangle/filesj/

Interesting question, Mayur... but could you tell us:

i) which format are your large files (8-bit ASCII textfile? binary file?);

ii) how was this large file created? (it may be possible to reduce its size, for instance); 

iii) is data structure regular (table-like) in the large file e.g. 8 values per line?;

iv) what scripting environment do you use to import the files (Python? MATLAB? R? A compiled executable? A commercial solution?)?;

v) do you have access to supercomputers or something equivalent?

If you can, answer all of these questions and people reading the thread will surely be able to give you "ready-to-serve" solutions.

use code below in a GNU/Linux distro's(like Debian or Ubuntu) terminal

split -b 53750k

this would cut your data into 50meg chunks(original file is not going to be over-written), then each chunk could be fed to separate processing units, if your work is parallel in nature this is the best idea i could think of;

btw, i need more detail about your job but i suggest Hadoop, it would help you process your files in parallel;

@All, Marc Laurencelle:

Adding more information for question as: I am working on windows(right now bit difficult to Linux). The files are database files, I can produce both Mysql DB files or tables in the form of .txt file. I want to split based on the selection and filter criteria on this files(Its not after every n bytes). I am using R for this problem. The server working on is 8 core, 32 GB RAM server.

Thank you for the details! I took a couple of minutes to write this example script that may do the work. Can you try it (after changing file paths) and tell me if that's fast enough? The thing is, working with text files is quite slow, because monitoring line changes (#13#10 characters) requires a byte-per-byte analysis of the input file as it is being read... So, is my script efficient enough with your 5GB file? To test it, you may want to add some criteria to stop after the creation of n output files... or simply force R to stop the execution of the script.      Looking forward to hearing from you!

Have you considered a KNIME worklfow?

Your problem, Joseph, is not that similar. You simply look for some software or efficient tool within the Linux OS to copy a large (huge, indeed) file from a disk to another. I would suggest you simply search for a light & efficient tool e.g. from SourceForge. For instance, I found this, after googling "copy huge files over network" (see the link); you'll surely find something equivalent for Linux systems.

http://copyfile.sourceforge.net/

@Marc Laurencelle : Thank you for your comments and help in this question, I tried running your script, It takes lot of time for a single file probably more than 18 hrs for file size approx 5GB. Actually the solution which i am using does this in 4 to 5 hrs. I still wanted to make it parallel or any mechanism which reduces time. I am working with data.table package to get done it faster. also the problem of R session stopped working comes when i keep my script running on this problem. hence i wanted it to be done in more faster and efficient way.

Oh. So my simple script is definitely too slow for your needs. Still, I have other ideas for R scripting... but it might be simpler to find a solution available through some R package, as you already began to do. However, there might be other, more efficient, solutions than with the data.table package: after all, there are so many packages for R.

I invite you to explore software from Revolution Analytics. As I am in a university, I could install and use their Revolution R Enterprise version, and I was quite impressed of what this enhanced R environment has to offer. Moreover, it is made to tackle the challenges of Big Data problems. If you can get a license for the full featured version (Enterprise), great; otherwise, you should at least try their Open version, which is a boosted R distribution (it replaces the common R distribution hassle-free).

If you're concerned only about the splitting of the large files, you may get even better performances if you find an executable file (command-line style) that does precisely what you want. The best, if you're familiar with programming, would be to create it yourself, or else to share the details of your needs to some programmer willing to take on the challenge.

But if you'd prefer to optimize the analyses as well, exploring advanced solutions for Big Data such as abovementioned Revolution R is probably the most promising option... at least if you wish to develop an R solution to solve your problem.

http://www.revolutionanalytics.com/

Very helpful. Thank you Marc Laurencelle. This is surely going to useful for me.

Im just curious as to why you would need to split one file into thousands. A little more detail would be helpful. Could you use a look up table to point you to useful bits of your database for instance to avoid this? 

There are various reason i had chosen to split operation on my data as it it available in very large amount. lets say i have 20 files of 5 GB each. and i want to fetch only 500 MB data executable from each file to be getting loaded into RAM would need 10GB memory data to be executed. if we does not perform split data we have to load 20 * 5 GB of data into RAM for execution. Basically we need to drill down our data for execution. Hence once if we have split data we can execute that many time in less time rather than executing large data many times

Splitting your database seems like a very messy approach. Many database technologies support data access out of memory, and can also be very quick if your data is structured properly. I would suggest looking at HDF5 or something similar. Python has some good tools for this work with PyTables and Pandas. Good luck. http://stackoverflow.com/questions/14262433/large-data-work-flows-using-pandas