mongodb - Endless recovering state of secondary -


i build replication set 1 primary, 1 secondary , 1 arbiter on mongodb 3.0.2. primary , arbiter on same host , secondary on host.

with growing of write overload, secondary can't follow primary , step state of recovering. primary can connect secondary can log secondary server mongo shell on host of primary.

i stop operations , watch secondary's state command rs.status() , type command rs.syncfrom("primary's ip:port") on secondary.

then result of rs.status() command shows optimedate of secondary far behind of primary , 1 message appears intermittently below:

"set" : "shard01", "date" : isodate("2015-05-15t02:10:55.382z"), "mystate" : 3, "members" : [ { "_id" : 0, "name" : "xxx.xxx.xxx.xxx:xxx", "health" : 1, "state" : 1, "statestr" : "primary", "uptime" : 135364, "optime" : timestamp(1431655856, 6), "optimedate" : isodate("2015-05-15t02:10:56z"), "lastheartbeat" : isodate("2015-05-15t02:10:54.306z"), "lastheartbeatrecv" : isodate("2015-05-15t02:10:53.634z"), "pingms" : 0, "electiontime" : timestamp(1431520398, 2), "electiondate" : isodate("2015-05-13t12:33:18z"), "configversion" : 3 }, { "_id" : 1, "name" : "xxx.xxx.xxx.xxx:xxx", "health" : 1, "state" : 7, "statestr" : "arbiter", "uptime" : 135364, "lastheartbeat" : isodate("2015-05-15t02:10:53.919z"), "lastheartbeatrecv" : isodate("2015-05-15t02:10:54.076z"), "pingms" : 0, "configversion" : 3 }, { "_id" : 2, "name" : "xxx.xxx.xxx.xxx:xxx", "health" : 1, "state" : 3, "statestr" : "recovering", "uptime" : 135510, "optime" : timestamp(1431602631, 134), "optimedate" : isodate("2015-05-14t11:23:51z"), "infomessage" : "could not find member sync from", "configversion" : 3, "self" : true } ], "ok" : 1

"infomessage" : "could not find member sync from"

the primary , arbiter both ok. want know reason of message , how change secondary's state "recovering" "secondary".

the problem (most likely)

the last operation on primary "2015-05-15t02:10:56z", whereas last operation of going secondary "2015-05-14t11:23:51z", difference of 15 hours. window may exceed replication oplog window (the difference between time of first , last operation entry in oplog). put simply, there many operations on primary secondary catch up.

a bit more elaborated (though simplified): during initial sync, data secondary syncs data of given point in time. when data of point in time synced over, secondary connects oplog , applies changes made between said point in time , according oplog entries. works long oplog holds operations between mentioned point in time. oplog has limited size (it called capped collection). if there more operations happening on primary oplog can hold during initial sync, oldest operations "fade out". secondary recognises not operations available necessary "construct" same data primary , refuses complete sync, staying in recovery mode.

the solution(s)

the problem known 1 , not bug, result of inner workings of mongodb , several fail-safe assumptions made development team. hence, there several ways deal situation. sadly, since have 2 data bearing nodes, involve downtime.

option 1: increase oplog size

this preferred method, since deals problem once , (kind of) all. it's bit more complicated other solutions, though. high level perspective, these steps take.

  1. shut down primary
  2. create backup of oplog using direct access data files
  3. restart mongod in standalone mode
  4. copy current oplog temporary collection
  5. delete current oplog
  6. recreate oplog desired size
  7. copy oplog entries temporary collection shiny new oplog
  8. restart mongod part of replica set

do not forget increase oplog of secondary before doing initial sync, since may become primary @ time in future!

for details, please read "change size of oplog" in tutorials regarding replica set maintenance.

option 2: shut down app during sync

if option 1 not viable, real other solution shut down application causing load on replica set, restart sync , wait complete. depending on amount of data transferred, calculate several hours.

a personal note

the oplog window problem known one. while replica sets , sharded clusters easy set mongodb, quite knowledge , bit of experience needed maintain them properly. not run important database complex setup without knowing basics - in case bad (tm) happens, might lead situation fubar.


Comments

Post a Comment

Popular posts from this blog

c++ - Difference between pre and post decrement in recursive function argument -

i have following sample code used pre-decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(--c); cout << c << endl; return; } } this function give output input 4 : dp3 dp2 dp1 dp0 0 1 2 3 but when use post decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(c--); cout << c << endl; return; } } output goes in infinite loop dp4 can explain me in detail why happens ? this happens because function(c--) called same value of c , when finishes c decremented. recursively called, hence called same value no chance of return , hit stack overflow error. assume this, int x = 1, y = 0; y = x++ result y == 1 , x == 2. if y = ++x , bot x , y 2.
Read more

php - Nothing but 'run(); ' when browsing to my local project, how do I fix this? -

i added new project. made mistake in projectname.conf file in sites-available apache2, led me reinstalling apache2. after reinstalling , re-enabling projects on every page of every project .. <body> run(); </body> .. i've run composer update, restarted service, restarted pc, checked config files, tried disabling new project. there nothing in error log, , access.log: 127.0.0.1 - - [15/may/2015:01:40:47 +0200] "get / http/1.1" 304 181 "-" "mozilla/5.0 (x11; ubuntu; linux x86_64; rv:38.0) gecko/20100101 firefox/38.0" maybe i'm forgetting apache2 reinstall? don't think did different last time installed it. looks haven't setup php correctly. take @ this great guide , maybe you're forgetting link php config apache2.
Read more

php - How can I echo out this array? -

i don't know how go grabbing data in variable , put foreach loop. class tree{ private $info = array( array('name'=>'jim','state'=>'nu','company'=>'nu','phone'=>array('cell'=>'5615111111','office'=>'5611111111'),'email'=>array('primary'=>'exs@example.com','ex@there.com')), array('name'=>'joe smith','city'=>'phoenix','phone'=>'4805551111','email'=>'jsmith@some_email.com'), array('name'=>'john doe','city'=>'chandler','company'=>'doe co.','email'=>array('jdoe@gmail.com','personal'=>'email@email.com'),'phone'=>'6025550002') ); } if you're inside of class, can access private variable using $this->variablename . example, i...
Read more