hadoop - hive job not making progress due to Number of reduce tasks is set to 0 since there's no reduce operator -



i'm using cloudera distribution , hive's 13th version on cluster.

i came across problem job not making progress after writing log line - "number of reduce tasks set 0 since there's no reduce operator"

below log same, me kind of issue because not code issue if re-run same job completes successfully.

logging initialized using configuration in jar:file:/opt/cloudera/parcels/cdh-5.2.1-1.cdh5.2.1.p0.12/jars/hive-common-0.13.1-cdh5.2.1.jar!/hive-log4j.properties total jobs = 5 launching job 1 out of 5 launching job 2 out of 5 number of reduce tasks not specified. defaulting jobconf value of: 10 in order change average load reducer (in bytes):   set hive.exec.reducers.bytes.per.reducer=<number> in order limit maximum number of reducers:   set hive.exec.reducers.max=<number> in order set constant number of reducers: number of reduce tasks not specified. defaulting jobconf value of: 10   set mapreduce.job.reduces=<number> in order change average load reducer (in bytes):   set hive.exec.reducers.bytes.per.reducer=<number> in order limit maximum number of reducers:   set hive.exec.reducers.max=<number> in order set constant number of reducers:   set mapreduce.job.reduces=<number> starting job = job_1431159077692_1399, tracking url = xyz.com:8088/proxy/application_1431159077692_1399/ starting job = job_1431159077692_1398, tracking url = hxyz.com:8088/proxy/application_1431159077692_1398/ kill command = /opt/cloudera/parcels/cdh-5.2.1-1.cdh5.2.1.p0.12/lib/hadoop/bin/hadoop job  -kill job_1431159077692_1399 kill command = /opt/cloudera/parcels/cdh-5.2.1-1.cdh5.2.1.p0.12/lib/hadoop/bin/hadoop job  -kill job_1431159077692_1398 hadoop job information stage-12: number of mappers: 5; number of reducers: 10 hadoop job information stage-1: number of mappers: 5; number of reducers: 10 2015-05-12 19:59:12,298 stage-1 map = 0%,  reduce = 0% 2015-05-12 19:59:12,298 stage-12 map = 0%,  reduce = 0% 2015-05-12 19:59:20,832 stage-1 map = 20%,  reduce = 0%, cumulative cpu 2.5 sec 2015-05-12 19:59:20,832 stage-12 map = 80%,  reduce = 0%, cumulative cpu 8.63 sec 2015-05-12 19:59:21,905 stage-1 map = 60%,  reduce = 0%, cumulative cpu 7.06 sec 2015-05-12 19:59:22,968 stage-1 map = 80%,  reduce = 0%, cumulative cpu 9.34 sec 2015-05-12 19:59:24,031 stage-1 map = 100%,  reduce = 0%, cumulative cpu 11.46 sec 2015-05-12 19:59:26,265 stage-12 map = 100%,  reduce = 0%, cumulative cpu 10.92 sec 2015-05-12 19:59:32,665 stage-12 map = 100%,  reduce = 30%, cumulative cpu 24.51 sec 2015-05-12 19:59:33,726 stage-12 map = 100%,  reduce = 100%, cumulative cpu 57.61 sec 2015-05-12 19:59:35,021 stage-1 map = 100%,  reduce = 30%, cumulative cpu 20.99 sec mapreduce total cumulative cpu time: 57 seconds 610 msec ended job = job_1431159077692_1399 2015-05-12 19:59:36,084 stage-1 map = 100%,  reduce = 80%, cumulative cpu 39.24 sec 2015-05-12 19:59:37,146 stage-1 map = 100%,  reduce = 90%, cumulative cpu 42.37 sec 2015-05-12 19:59:38,203 stage-1 map = 100%,  reduce = 100%, cumulative cpu 45.97 sec mapreduce total cumulative cpu time: 45 seconds 970 msec ended job = job_1431159077692_1398 2015-05-12 19:59:45,180 warn  [main] conf.configuration (configuration.java:loadproperty(2510)) - file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10014/jobconf.xml:an attempt override final parameter: hadoop.ssl.require.client.cert;  ignoring. 2015-05-12 19:59:45,193 warn  [main] conf.configuration (configuration.java:loadproperty(2510)) - file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10014/jobconf.xml:an attempt override final parameter: mapreduce.job.end-notification.max.retry.interval;  ignoring. 2015-05-12 19:59:45,196 warn  [main] conf.configuration (configuration.java:loadproperty(2510)) - file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10014/jobconf.xml:an attempt override final parameter: hadoop.ssl.client.conf;  ignoring. 2015-05-12 19:59:45,201 warn  [main] conf.configuration (configuration.java:loadproperty(2510)) - file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10014/jobconf.xml:an attempt override final parameter: hadoop.ssl.keystores.factory.class;  ignoring. 2015-05-12 19:59:45,210 warn  [main] conf.configuration (configuration.java:loadproperty(2510)) - file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10014/jobconf.xml:an attempt override final parameter: hadoop.ssl.server.conf;  ignoring. 2015-05-12 19:59:45,258 warn  [main] conf.configuration (configuration.java:loadproperty(2510)) - file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10014/jobconf.xml:an attempt override final parameter: mapreduce.job.end-notification.max.attempts;  ignoring. 2015-05-12 19:59:45,792 warn  [main] conf.hiveconf (hiveconf.java:initialize(1491)) - deprecated: configuration property hive.metastore.local no longer has effect. make sure provide valid value hive.metastore.uris if connecting remote metastore. execution log at: /tmp/srv-hdp-mkt-d/srv-hdp-mkt-d_20150512195858_1b598453-78a8-4867-9402-d972e3c067f2.log 2015-05-12 07:59:46    starting launch local task process map join;    maximum memory = 257949696 2015-05-12 07:59:47    dump side-table file: file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10007/hashtable-stage-4/mapjoin-mapfile10--.hashtable 2015-05-12 07:59:47    uploaded 1 file to: file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10007/hashtable-stage-4/mapjoin-mapfile10--.hashtable (475 bytes) 2015-05-12 07:59:47    dump side-table file: file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10007/hashtable-stage-4/mapjoin-mapfile01--.hashtable 2015-05-12 07:59:47    uploaded 1 file to: file:/tmp/srv-hdp-mkt-d/hive_2015-05-12_19-58-53_081_2145723752519383568-1/-local-10007/hashtable-stage-4/mapjoin-mapfile01--.hashtable (388 bytes) 2015-05-12 07:59:47    end of local task; time taken: 1.209 sec. execution completed mapredlocal task succeeded launching job 3 out of 5 number of reduce tasks set 0 since there's no reduce operator

are setting tasks in scripts, if yes remove , try running again. think no need create multiple tasks if don't have in job.


Comments

Post a Comment

Popular posts from this blog

Email notification in google apps script -

https://support.google.com/docs/answer/91588?hl=en i have sheet the cells filled in data pulled via apps script, website. pull in exact same data, not. on old sheets version use able emails when data "changes" different value. new sheets, email when new data pasted cells, though exact same data......and, can't set @ ranges anymore, entire document. there easy way watch cells when data changes, , email myself? this looks lot you're trying do. for checking whether value has changed after edit, this gives necessary information it. hence, know whether new value same 1 or not. implementing "workaround" talked in second link, able compare , send email myself if value changed. initiate onedit trigger goes off onedit function uses these lines after checks in place have passed. var email = "xyz@whatever.com"; mailapp.sendemail(email, "triggermail", "a change has occurred in sheet"); note: sends "notification...
Read more

c++ - Difference between pre and post decrement in recursive function argument -

i have following sample code used pre-decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(--c); cout << c << endl; return; } } this function give output input 4 : dp3 dp2 dp1 dp0 0 1 2 3 but when use post decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(c--); cout << c << endl; return; } } output goes in infinite loop dp4 can explain me in detail why happens ? this happens because function(c--) called same value of c , when finishes c decremented. recursively called, hence called same value no chance of return , hit stack overflow error. assume this, int x = 1, y = 0; y = x++ result y == 1 , x == 2. if y = ++x , bot x , y 2.
Read more

javascript - IE11 incompatibility with jQuery's 'readonly'? -

the code i'm running has no issues in firefox, or chrome. user typing 1 textbox, tabs next , can continue typing that. when run webpage on ie11, user tabs next textbox , can't type in. textbox has double-clicked before user can insert anything. i've googled , seems ie give problems when using older jquery (i.e. prefers 'prop' 'attr' ). in code below jquery's readonly keyword being used in way ie11 no longer recognise/accept? i assume issue readonly , removeclass('ignore') working expected. //when focus on textbox 1 $("#field1").focus(function() { //remove readonly property textbox 1 $('#field1').prop('readonly', false).removeclass('ignore'); //add readonly property other textboxes $('#field2, #field3, #field4') .prop('readonly', 'readonly').addclass('ignore').val(''); }); i'm using jquery 2.1.3 ...
Read more