top button
Flag Notify
    Connect to us
      Site Registration

Site Registration

How pickle helps in reading huge csv files?

+1 vote
728 views

I've a huge csv file and I want to read stuff from it again and again. Is it useful to pickle it and keep and then unpickle it whenever I need to use that data? Is it faster that accessing that file simply by opening it again and again? Please explain?

posted Oct 16, 2013 by Sanketi Garg

Share this question
Facebook Share Button Twitter Share Button LinkedIn Share Button
Keep it in memory
commented Oct 16, 2013 by anonymous

1 Answer

+1 vote

What's your definition of huge? Maybe it would be effective to pickle and unpickle but until you try it, perhaps with a relatively small data sample, how can you know? Why can't you leave the file open and keep iterating over the contents?

answer Oct 16, 2013 by Satish Mishra
Similar Questions
+2 votes
How to read CSV file in python?

Is there a better way to do that:

def Read_CSV_File(filename):
 file = open(filename, "r")
 reader = csv.DictReader(file)
 line = 1
 for row in reader:
 if line < 6:
 reader.next()
 line++
# process the CSV
0 votes
How to export certain fields from PHP to CSV in Wordpress?

I am trying to export certain data from my PHP form to CSV. I can echo out to screen during testing and I can also export to CSV the static test data (stored in the $contents array) you see below. But I am stuck trying to export the certain fields that I only need to export.
This is my code

// How do I get this info into the CSV?
/*foreach ( $entries as $entry ) :  
    echo $entry['2'];
    echo $entry['3'];
    echo $entry['6'];
endforeach;*/

$csv_headers = [
    'Organisation Name',
    'Registered Charity Number',
    'Address',
    'Phone',
];

$contents = [
  [2014, 6, '1st half', 'roland@fsjinvestor.com', 0, 0],
  [2014, 6, '1st half', 'steve@neocodesoftware.com', 0, 0],
  [2014, 6, '1st half', 'susanne@casamanager.com', 0, 0],
  [2014, 6, '1st half', 'tim', 0, 0]
];

fputcsv($output_handle, $csv_headers);

foreach ( $contents as $content) :
    fputcsv($output_handle, $content);
endforeach;
...