python - Break pandas DataFrame column into multiple pieces and combine with other DataFrame -


i have table of phrases , have table of individual words make these phrases. want break phrases individual words, gather , reduce information these individual words , add new column in phrase data. there smart way using pandas dataframes?

    df_multigram = pd.dataframe([         ["happy birthday", 23],         ["used below", 10],         ["frame for", 2]     ], columns=["multigram", "frequency"])     df_onegram = pd.dataframe([         ["happy", 35],         ["birthday", 25],         ["used", 14],         ["below", 11],         ["frame", 2],         ["for", 13]     ], columns=["onegram", "frequency"])      ###### do here????? #######      sum_freq_onegrams = list(df_multigram["sum_freq_onegrams"])     self.assertequal(sum_freq_onegrams, [60, 25, 15])

just clarify, desire sum_freq_onegrams equal [60, 25, 15], 60 frequency of "happy" plus frequency of "birthday".

you use

freq = df_onegram.set_index(['onegram'])['frequency'] sum_freq_onegrams = df_multigram['multigram'].str.split().apply(     lambda x: pd.series(x).map(freq).sum())

which yields

in [43]: sum_freq_onegrams out[45]:  0    60 1    25 2    15 name: multigram, dtype: int64

but note calling (lambda) function once every row , building new (tiny) series each time may rather slow. using different data structure -- plain python lists , dicts -- may faster. example, if defined list phrases , dict freq_dict,

phrases = df_multigram['multigram'].tolist() freq_dict = freq.to_dict()

then list comprehension (below) 280x faster pandas-based method:

in [65]: [sum(freq_dict.get(item, 0) item in phrase.split()) phrase in phrases] out[65]: [60, 25, 15]  in [38]: %timeit [sum(freq_dict.get(item, 0)for item in phrase.split()) phrase in phrases] 100000 loops, best of 3: 3.6 µs per loop  in [41]: %timeit df_multigram['multigram'].str.split().apply(lambda x: pd.series(x).map(freq).sum()) 1000 loops, best of 3: 1.01 ms per loop

thus, using pandas dataframe here hold phrases might not right data structure problem.


Comments

Post a Comment

Popular posts from this blog

c++ - Difference between pre and post decrement in recursive function argument -

i have following sample code used pre-decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(--c); cout << c << endl; return; } } this function give output input 4 : dp3 dp2 dp1 dp0 0 1 2 3 but when use post decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(c--); cout << c << endl; return; } } output goes in infinite loop dp4 can explain me in detail why happens ? this happens because function(c--) called same value of c , when finishes c decremented. recursively called, hence called same value no chance of return , hit stack overflow error. assume this, int x = 1, y = 0; y = x++ result y == 1 , x == 2. if y = ++x , bot x , y 2.
Read more

php - Nothing but 'run(); ' when browsing to my local project, how do I fix this? -

i added new project. made mistake in projectname.conf file in sites-available apache2, led me reinstalling apache2. after reinstalling , re-enabling projects on every page of every project .. <body> run(); </body> .. i've run composer update, restarted service, restarted pc, checked config files, tried disabling new project. there nothing in error log, , access.log: 127.0.0.1 - - [15/may/2015:01:40:47 +0200] "get / http/1.1" 304 181 "-" "mozilla/5.0 (x11; ubuntu; linux x86_64; rv:38.0) gecko/20100101 firefox/38.0" maybe i'm forgetting apache2 reinstall? don't think did different last time installed it. looks haven't setup php correctly. take @ this great guide , maybe you're forgetting link php config apache2.
Read more

php - How can I echo out this array? -

i don't know how go grabbing data in variable , put foreach loop. class tree{ private $info = array( array('name'=>'jim','state'=>'nu','company'=>'nu','phone'=>array('cell'=>'5615111111','office'=>'5611111111'),'email'=>array('primary'=>'exs@example.com','ex@there.com')), array('name'=>'joe smith','city'=>'phoenix','phone'=>'4805551111','email'=>'jsmith@some_email.com'), array('name'=>'john doe','city'=>'chandler','company'=>'doe co.','email'=>array('jdoe@gmail.com','personal'=>'email@email.com'),'phone'=>'6025550002') ); } if you're inside of class, can access private variable using $this->variablename . example, i...
Read more