elasticsearch - Updating field of million of document from a worker -


currently have update field in on 1 million documents indexed in elasticsearch. complex task due field contains metadata generated xml files, evaluating xpath expressions. have loop on documents in index , update field. so, in order avoid overkill system, decide use ironworker platform.
have read several post how update millions of docs in elasticsearch, this one, given gonna use ironworkers there restrictions, task can run 60 minutes.

question: how loop on documents , update fields, considering restriction of 60 min.
thought opening , scroll , pass scroll_id next worker, don't have idea of how long take execute next task, scroll expire , have start over.

it sounds description chain ironworker tasks, easy. if have idea of how long takes through updating single item, extrapolate how long need. let's took 100ms update item, 10 per second, or 600 per minute maybe 6000 (which should take 10 minutes), queue next 1 code. queuing next task easy queuing first task: http://dev.iron.io/worker/reference/api/#queue_a_task (can use client library language too).

or stop after x minutes , queue next worker.

or if want make things faster, how queue 26 @ same time, 1 each letter of alphabet? each 1 can query items starting letter it's assigned (prefix query ) .

there's many ways slice problem.


Comments

Post a Comment

Popular posts from this blog

c++ - Difference between pre and post decrement in recursive function argument -

i have following sample code used pre-decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(--c); cout << c << endl; return; } } this function give output input 4 : dp3 dp2 dp1 dp0 0 1 2 3 but when use post decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(c--); cout << c << endl; return; } } output goes in infinite loop dp4 can explain me in detail why happens ? this happens because function(c--) called same value of c , when finishes c decremented. recursively called, hence called same value no chance of return , hit stack overflow error. assume this, int x = 1, y = 0; y = x++ result y == 1 , x == 2. if y = ++x , bot x , y 2.
Read more

php - Nothing but 'run(); ' when browsing to my local project, how do I fix this? -

i added new project. made mistake in projectname.conf file in sites-available apache2, led me reinstalling apache2. after reinstalling , re-enabling projects on every page of every project .. <body> run(); </body> .. i've run composer update, restarted service, restarted pc, checked config files, tried disabling new project. there nothing in error log, , access.log: 127.0.0.1 - - [15/may/2015:01:40:47 +0200] "get / http/1.1" 304 181 "-" "mozilla/5.0 (x11; ubuntu; linux x86_64; rv:38.0) gecko/20100101 firefox/38.0" maybe i'm forgetting apache2 reinstall? don't think did different last time installed it. looks haven't setup php correctly. take @ this great guide , maybe you're forgetting link php config apache2.
Read more

php - How can I echo out this array? -

i don't know how go grabbing data in variable , put foreach loop. class tree{ private $info = array( array('name'=>'jim','state'=>'nu','company'=>'nu','phone'=>array('cell'=>'5615111111','office'=>'5611111111'),'email'=>array('primary'=>'exs@example.com','ex@there.com')), array('name'=>'joe smith','city'=>'phoenix','phone'=>'4805551111','email'=>'jsmith@some_email.com'), array('name'=>'john doe','city'=>'chandler','company'=>'doe co.','email'=>array('jdoe@gmail.com','personal'=>'email@email.com'),'phone'=>'6025550002') ); } if you're inside of class, can access private variable using $this->variablename . example, i...
Read more