top button
Flag Notify
    Connect to us
      Site Registration

Site Registration

Fastest way to write data into a file using Python?

0 votes
2,848 views

I need to write into a file for a project which will be evaluated on the basis of time. What is the fastest way to write 200 Mb of data, accumulated as a list into a file.

Presently I am using this:

with open('index.txt','w') as f:
 f.write("".join(data))
 f.close()

where data is a list of strings which I want to dump into the index.txt file

posted Jan 14, 2014 by Luv Kumar

Share this question
Facebook Share Button Twitter Share Button LinkedIn Share Button

1 Answer

+1 vote 
with open('index.txt','w') as f:
 for hunk in data:
 f.write(hunk)

You don't need to f.close() - that's what the 'with' block guarantees. Iterating over data and writing each block separately means you don't have to first build up a 200MB string. After that, your performance is going to be mainly tied to the speed of your disk, not anything that Python can affect.

answer Jan 14, 2014 by Mandeep Sehgal
Thanks for the tip on the closing of the file. I did not know that with ensures closing of the file after iteration is over.

Which is more fast?
Creating a 200 Mb string and then dumping into a file or dividing the 200 Mb string into chunks and then writing those chunks. Won't writing the chunks call more i/o operation?
commented Jan 14, 2014 by anonymous
Similar Questions
+3 votes
Change a file type in Python?

When opening a file, you'd say whether you want to read or write to a file. This is fine, but say for example later on in the program I change my mind and I want to write to a file instead of reading it. Yes, I could just say 'r+w' when opening the file, but what if I don't know if I'm going to do both? What if I have the user decide, and then later on I let the user change this.

Is it possible to do so without opening the file again and using the same file object?

+1 vote
How to write fast into a file in python?

I need to write numbers into a file upto 50mb and it should be fast can any one help me how to do that? I had written the following code..

def create_file_numbers_old(filename, size):
start = time.clock()

value = 0
with open(filename, "w") as f:
while f.tell()< size:
f.write("{0}n".format(value))
value += 1

end = time.clock()

print "time taken to write a file of size", size, " is ", (end -start), "seconds n"

it takes about 20sec i need 5 to 10 times less than that.

0 votes
Using sudo to write to a file as root from a python script

I'm trying to write a python script that writes some content to a file root through sudo, but it's not working at all. I am using:

 channel = 'stable'
 config_file = '/opt/ldg/etc/channel.conf'
 command = ['echo', '-n', channel, '|', 'sudo', 'tee', config_file]
 p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
 out, _ = p.communicate()

But it seems as if this isn't doing anything.

I just want to write the contents of the variable channel to the file /opt/ldg/etc/channel.conf. But whatever I try just doesn't work. Can anyone offer any pointers?

...