python - pickling pandas dataframe does multiply by 5 the file size -


i reading 800 mo csv file panda.csv_reader, , use original pythoin pickle.dump(datfarame) save it. result 4 gb pkl file, csv size multipled 5.

i expected pickle compress data rather extend it. because can gzip on csv file compress 200 mo, dividing 4.

i willing accelerate loading time of program, , thought pickling help, considering disk access main bottleneck understanding rather have compress files , use compression option pandas.csv_read speed loading time.

is correct ?

is normal pickling pandas dataframe extend data size ?

how speed loading time ?

thks.

edit

considering answers here below question emerge : data-size limit load pandas ?

it in best interest stash csv file in database of sort , perform operations on rather loading csv file ram, kathirmani suggested. see speedup in loading time expect due fact not filling 800 mb worth of ram every time load script.

file compression , loading time 2 conflicting elements of seem trying accomplish. compressing csv file , loading take more time; you've added step of having decompress file, doesn't solve problem.

consider precursory step ship data sqlite3 database, described here: importing csv file sqlite3 database table using python.

you have pleasure of being able query subset of data , load pandas.dataframe further use, follows:

from pandas.io import sql import sqlite3  conn = sqlite3.connect('your/database/path') query = "select * foo bar = 'foobar';"  results_df = sql.read_frame(query, con=conn) ...

conversely, can use pandas.dataframe.to_sql() save these later use.


Comments

Post a Comment

Popular posts from this blog

c++ - Difference between pre and post decrement in recursive function argument -

i have following sample code used pre-decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(--c); cout << c << endl; return; } } this function give output input 4 : dp3 dp2 dp1 dp0 0 1 2 3 but when use post decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(c--); cout << c << endl; return; } } output goes in infinite loop dp4 can explain me in detail why happens ? this happens because function(c--) called same value of c , when finishes c decremented. recursively called, hence called same value no chance of return , hit stack overflow error. assume this, int x = 1, y = 0; y = x++ result y == 1 , x == 2. if y = ++x , bot x , y 2.
Read more

php - Nothing but 'run(); ' when browsing to my local project, how do I fix this? -

i added new project. made mistake in projectname.conf file in sites-available apache2, led me reinstalling apache2. after reinstalling , re-enabling projects on every page of every project .. <body> run(); </body> .. i've run composer update, restarted service, restarted pc, checked config files, tried disabling new project. there nothing in error log, , access.log: 127.0.0.1 - - [15/may/2015:01:40:47 +0200] "get / http/1.1" 304 181 "-" "mozilla/5.0 (x11; ubuntu; linux x86_64; rv:38.0) gecko/20100101 firefox/38.0" maybe i'm forgetting apache2 reinstall? don't think did different last time installed it. looks haven't setup php correctly. take @ this great guide , maybe you're forgetting link php config apache2.
Read more

php - How can I echo out this array? -

i don't know how go grabbing data in variable , put foreach loop. class tree{ private $info = array( array('name'=>'jim','state'=>'nu','company'=>'nu','phone'=>array('cell'=>'5615111111','office'=>'5611111111'),'email'=>array('primary'=>'exs@example.com','ex@there.com')), array('name'=>'joe smith','city'=>'phoenix','phone'=>'4805551111','email'=>'jsmith@some_email.com'), array('name'=>'john doe','city'=>'chandler','company'=>'doe co.','email'=>array('jdoe@gmail.com','personal'=>'email@email.com'),'phone'=>'6025550002') ); } if you're inside of class, can access private variable using $this->variablename . example, i...
Read more