How to write fast into a file in python?

Question

I need to write numbers into a file upto 50mb and it should be fast can any one help me how to do that? I had written the following code..

def create_file_numbers_old(filename, size):
start = time.clock()

value = 0
with open(filename, "w") as f:
while f.tell()< size:
f.write("{0}n".format(value))
value += 1

end = time.clock()

print "time taken to write a file of size", size, " is ", (end -start), "seconds n"

it takes about 20sec i need 5 to 10 times less than that.

Answer 1

20 seconds to write how many numbers? If you are doing

create_file_numbers_old(filename, 5)

then 20 seconds is really slow. But if you are doing:

create_file_numbers_old(filename, 50000000000000)

then 20 seconds is amazingly fast.

Try this instead, it may be a little faster:

def create_file_numbers_old(filename, size):
 count = value = 0
 with open(filename, 'w') as f:
 while count < size:
 s = '%dn' % value
 f.write(s)
 count += len(s)
 value += 1

If this is still too slow, you can try three other tactics:

1) pre-calculate the largest integer that will fit in size bytes, then use a for-loop instead of a while loop:

 maxn = calculation(...)
 with open(filename, 'w') as f:
 for i in xrange(maxn):
 f.write('%dn' % i)

2) Write an extension module in C that writes to the file.
3) Get a faster hard drive, and avoid writing over a network.

Answer 2

Most of the time is spent figuring out whether the file has reached its limit size. If you want Python to go fast, just specify the data. On my Linux system, it takes 11 seconds to write the first 6338888 values,
which is just under 50mb. If I write the obvious loop, writing that many values takes .25 seconds.