python - Pandas read csv out of memory -


i try manipulate large csv file using pandas, when wrote this

df = pd.read_csv(strfilename,sep='\t',delimiter='\t')

it raises "pandas.parser.cparsererror: error tokenizing data. c error: out of memory" wc -l indicate there 13822117 lines, need aggregate on csv file data frame, there way handle other split csv several files , write codes merge results? suggestions on how that?

the input this:

columns=[ka,kb_1,kb_2,timeofevent,timeinterval] 0:'3m' '2345' '2345' '2014-10-5',3000 1:'3m' '2958' '2152' '2015-3-22',5000 2:'ge' '2183' '2183' '2012-12-31',515 3:'3m' '2958' '2958' '2015-3-10',395 4:'ge' '2183' '2285' '2015-4-19',1925 5:'ge' '2598' '2598' '2015-3-17',1915

and desired output this:

columns=[ka,kb,errornum,errorrate,totalnum of records] '3m','2345',0,0%,1 '3m','2958',1,50%,2 'ge','2183',1,50%,2 'ge','2598',0,0%,1

if data set small, below code used provided another

df2 = df.groupby(['ka','kb_1'])['iserror'].agg({ 'errornum':  'sum',                                              'recordnum': 'count' })  df2['errorrate'] = df2['errornum'] / df2['recordnum']  ka kb_1  recordnum  errornum  errorrate  3m 2345          1         0        0.0    2958          2         1        0.5 ge 2183          2         1        0.5    2598          1         0        0.0

(definition of error record: when kb_1!=kb_2,the corresponding record treated abnormal record)

based on snippet in out of memory error when reading csv file in chunk, when reading line-by-line.

i assume kb_2 error indicator,

groups={} open("data/petajoined.csv", "r") large_file:     line in large_file:         arr=line.split('\t')         #assuming structure: ka,kb_1,kb_2,timeofevent,timeinterval         k=arr[0]+','+arr[1]         if not (k in groups.keys())             groups[k]={'record_count':0, 'error_sum': 0}         groups[k]['record_count']=groups[k]['record_count']+1         groups[k]['error_sum']=groups[k]['error_sum']+float(arr[2]) k,v in groups.items:     print ('{group}: {error_rate}'.format(group=k,error_rate=v['error_sum']/v['record_count']))

this code snippet stores groups in dictionary, , calculates error rate after reading entire file.

it encounter out-of-memory exception, if there many combinations of groups.


Comments

Post a Comment

Popular posts from this blog

c++ - Difference between pre and post decrement in recursive function argument -

i have following sample code used pre-decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(--c); cout << c << endl; return; } } this function give output input 4 : dp3 dp2 dp1 dp0 0 1 2 3 but when use post decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(c--); cout << c << endl; return; } } output goes in infinite loop dp4 can explain me in detail why happens ? this happens because function(c--) called same value of c , when finishes c decremented. recursively called, hence called same value no chance of return , hit stack overflow error. assume this, int x = 1, y = 0; y = x++ result y == 1 , x == 2. if y = ++x , bot x , y 2.
Read more

php - Nothing but 'run(); ' when browsing to my local project, how do I fix this? -

i added new project. made mistake in projectname.conf file in sites-available apache2, led me reinstalling apache2. after reinstalling , re-enabling projects on every page of every project .. <body> run(); </body> .. i've run composer update, restarted service, restarted pc, checked config files, tried disabling new project. there nothing in error log, , access.log: 127.0.0.1 - - [15/may/2015:01:40:47 +0200] "get / http/1.1" 304 181 "-" "mozilla/5.0 (x11; ubuntu; linux x86_64; rv:38.0) gecko/20100101 firefox/38.0" maybe i'm forgetting apache2 reinstall? don't think did different last time installed it. looks haven't setup php correctly. take @ this great guide , maybe you're forgetting link php config apache2.
Read more

php - How can I echo out this array? -

i don't know how go grabbing data in variable , put foreach loop. class tree{ private $info = array( array('name'=>'jim','state'=>'nu','company'=>'nu','phone'=>array('cell'=>'5615111111','office'=>'5611111111'),'email'=>array('primary'=>'exs@example.com','ex@there.com')), array('name'=>'joe smith','city'=>'phoenix','phone'=>'4805551111','email'=>'jsmith@some_email.com'), array('name'=>'john doe','city'=>'chandler','company'=>'doe co.','email'=>array('jdoe@gmail.com','personal'=>'email@email.com'),'phone'=>'6025550002') ); } if you're inside of class, can access private variable using $this->variablename . example, i...
Read more