Spark Streaming And Flume Integration -


i trying integrate flume spark streaming.

i have following workflow defined , seeing issue spark not able commit transaction flume channel when data volumes higher.

i new both flume , spark streaming if there's scalability option have not tried please let me know.

my workflow

spool dir -> file-channel-1 -> avro sink -> avro source -> file-channel-2

file-channel-2 consumed 2 sinks

  1. hdfs sink
  2. sparksink

so theres 2 agents,

agent 1 delivers avro sink running @ port 4545 spool directory.

agent 2 reads avro source running @ port 4545 , delivers hdfs , spark sink,

i can see events getting delivered hdfs. no problems there.

the spark sink on other hand throws following errors,

org.apache.spark.streaming.flume.sink.logging$class.loginfo(logging.scala:47)] spark failed commit transaction. reattempt events. org.apache.spark.streaming.flume.sink.logging$class.logwarning(logging.scala:59)] spark not commit transaction, nack received. rolling transaction.

these errors not occur right away. see lot of logs ones below , after time has passed starts throwing errors.

(new i/o worker #18) [info - org.apache.avro.ipc.nettyserver$nettyserveravrohandler.handleupstream(nettyserver.java:171)] [id: 0xab559b05, /127.0.0.1:51736 => /127.0.0.1:11111] interest_changed (new i/o worker #18) [info - org.apache.avro.ipc.nettyserver$nettyserveravrohandler.handleupstream(nettyserver.java:171)] [id: 0xab559b05, /127.0.0.1:51736 => /127.0.0.1:11111] interest_changed (new i/o worker #18) [info - org.apache.avro.ipc.nettyserver$nettyserveravrohandler.handleupstream(nettyserver.java:171)] [id: 0xab559b05, /127.0.0.1:51736 => /127.0.0.1:11111] interest_changed

i tried following options,

  1. i looked sparksink api. assumed if time out. increased timeout 300 seconds. didnt help.
  2. in flume.createpollingstream, specified batch size of 10000 parallellism 1. didn't either.

i know sure scalability issue because if spool small file 50 events spool directory, can see small dataset reaching both hdfs , sparksink.

my file channel properties basic , mentioned below.

agent_1.channels.file-channel-2.type = file agent_1.channels.file-channel-2.checkpointdir = <<some dir 1>> agent_1.channels.file-channel-2.datadirs = << dir 2 >>

they same file channel 1 too. pointing different directories of course. moreover, there's no isses agent1.

it's agent 2 delivers hdfs , spark problematic one.

please find sparksink config below.

agent_2.sinks.sparkpullsink.type = org.apache.spark.streaming.flume.sink.sparksink agent_2.sinks.sparkpullsink.hostname = localhost agent_2.sinks.sparkpullsink.port = 11111

i have spark streaming job runs @ port 11111, polling sink , processing events received. job doesn't receive events process when try large dataset.

thanks in advance inputs on this.


Comments

Post a Comment

Popular posts from this blog

c++ - Difference between pre and post decrement in recursive function argument -

i have following sample code used pre-decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(--c); cout << c << endl; return; } } this function give output input 4 : dp3 dp2 dp1 dp0 0 1 2 3 but when use post decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(c--); cout << c << endl; return; } } output goes in infinite loop dp4 can explain me in detail why happens ? this happens because function(c--) called same value of c , when finishes c decremented. recursively called, hence called same value no chance of return , hit stack overflow error. assume this, int x = 1, y = 0; y = x++ result y == 1 , x == 2. if y = ++x , bot x , y 2.
Read more

php - Nothing but 'run(); ' when browsing to my local project, how do I fix this? -

i added new project. made mistake in projectname.conf file in sites-available apache2, led me reinstalling apache2. after reinstalling , re-enabling projects on every page of every project .. <body> run(); </body> .. i've run composer update, restarted service, restarted pc, checked config files, tried disabling new project. there nothing in error log, , access.log: 127.0.0.1 - - [15/may/2015:01:40:47 +0200] "get / http/1.1" 304 181 "-" "mozilla/5.0 (x11; ubuntu; linux x86_64; rv:38.0) gecko/20100101 firefox/38.0" maybe i'm forgetting apache2 reinstall? don't think did different last time installed it. looks haven't setup php correctly. take @ this great guide , maybe you're forgetting link php config apache2.
Read more

php - How can I echo out this array? -

i don't know how go grabbing data in variable , put foreach loop. class tree{ private $info = array( array('name'=>'jim','state'=>'nu','company'=>'nu','phone'=>array('cell'=>'5615111111','office'=>'5611111111'),'email'=>array('primary'=>'exs@example.com','ex@there.com')), array('name'=>'joe smith','city'=>'phoenix','phone'=>'4805551111','email'=>'jsmith@some_email.com'), array('name'=>'john doe','city'=>'chandler','company'=>'doe co.','email'=>array('jdoe@gmail.com','personal'=>'email@email.com'),'phone'=>'6025550002') ); } if you're inside of class, can access private variable using $this->variablename . example, i...
Read more