Want to Compare values of 2 maps having billions of data in java -


i getting values database storing in maps. want compare values of 2 maps , load database problem is, maps having 100ks of data. right code written in such way comparing every entry eg if both maps having 200k of entries looping 200k * 200k times , taking days & days. need compare string , long values separately.

so there efficient way compare can avoid unnecessary looping?

here code written , old code:

below getallmaps method retrieving data database , storing in maps:

protected hashtable recordslist = new hashtable(); protected hashtable recordslistic = new hashtable();  public void getallmaps() throws exception {      int count = 0;     int = 0;     int j = 0;     int k = 0;     long nicphyport, nicid;     string ipv4val, ipv6val;     try {         trace.debug("ruuning pvc sql map");         nicpvcstmt.setfetchsize(5000);         resultset rs = nicpvcstmt.executequery();         int rcnt = 0;         while (rs.next()) {             rcnt++;             hashtable rownic = new hashtable();             nicid = rs.getlong("id");             pvcid = rs.getlong("icore_pvc_id_value");             nicphyport = rs.getlong("id_phy_port");             nicsiteid = rs.getlong("id_site_inventory_site_rep");             rownic.put("niclogport", nicid);             rownic.put("nicphyport", nicphyport);             rownic.put("nicpvc", pvcid);             rownic.put("nicsite", nicsiteid);             recordslist.put(count, rownic);             count++;         }          if (rs != null) rs.close();         geticorepvcunifiedrec.setfetchsize(5000);         rs = geticorepvcunifiedrec.executequery();         int rcnt1 = 0;         while (rs.next()) {             rcnt1++;             hashtable row = new hashtable();             icoreid = rs.getlong("id");             icorephyid = rs.getlong("id_phy_port");             pvcid = rs.getlong("icore_pvc_id_value");             row.put("icoreid", icoreid);             row.put("icphyport", icorephyid);             row.put("pvcid", pvcid);              recordslistic.put(i, row);             i++;         }          if (rs != null) rs.close();      } catch (exception ex) {         ex.getmessage();         ex.printstacktrace();     } }

below method checkpvcunificationapplies maps getting compared:

both maps recordslist , recordslistic having billions of data , because of 2nd loop taking lots of time

public void checkpvcunificationapplies(connection gcon) throws exception {     long icoreid, icorepvc, phyportid;      int cnt = 0;     try {         trace.debug("checkunificationapplies--");         (int = 0; < recordslist.size(); i++) {              hashtable tbl = (hashtable) recordslist.get(i);             //trace.debug("under check table "+tbl);             long nonicid = (long) tbl.get("niclogport");             long pvcvalue = (long) tbl.get("nicpvc");             long nicphyportid = (long) tbl.get("nicphyport");             long nicsiteid = (long) tbl.get("nicsite");             (int j = 0; j < recordslistic.size(); j++) {                 //trace.debug("list item iteration-->"+j);                 hashtable pvctbl = (hashtable) recordslistic.get(j);                 trace.debug("under pvctbl   values " + pvctbl);                 icoreid = (long) pvctbl.get("icoreid");                 icorepvc = (long) pvctbl.get("pvcid");                 phyportid = (long) pvctbl.get("icphyport");                 //icsite=(long)pvctbl.get("icphyport");                 trace.debug("matching " + pvcvalue + " " + icorepvc);                 if (pvcvalue == icorepvc) {                     icoreid = unify.geocodematchfound(nicsiteid, icoreid, gcon);                     if (icoreid > 0) {                         trace.debug("all values icoreid , nonicore " + icoreid + ":" + nonicid + " " + phyportid + nicphyportid);                         cs.setlong(1, icoreid);                         cs.setlong(2, nonicid);                         cs.setlong(3, phyportid);                         cs.setlong(4, nicphyportid);                         cs.addbatch();                         cnt++;                     }                 }                  if (cnt % 50000 == 0) {                     cs.executebatch();                     cnt = 0;                 }             }         }         if (cnt > 0) {             cs.executebatch();             trace.debug("executed batch >0" + cnt);         }         recordslist.clear();     } catch (exception e) {         e.getmessage();         e.printstacktrace();     } }

i not in java, let me know if there more efficient way compare above maps.

if have option use db procedures pl/sql, on there instead of doing in program. improve performance drastically.

if want extract data , compare in program, trade-off between speed, time , resources. if want fast performance, add more resources, , design effective algorithm fetch data in chunks compare.


Comments

Post a Comment

Popular posts from this blog

c++ - Difference between pre and post decrement in recursive function argument -

i have following sample code used pre-decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(--c); cout << c << endl; return; } } this function give output input 4 : dp3 dp2 dp1 dp0 0 1 2 3 but when use post decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(c--); cout << c << endl; return; } } output goes in infinite loop dp4 can explain me in detail why happens ? this happens because function(c--) called same value of c , when finishes c decremented. recursively called, hence called same value no chance of return , hit stack overflow error. assume this, int x = 1, y = 0; y = x++ result y == 1 , x == 2. if y = ++x , bot x , y 2.
Read more

php - Nothing but 'run(); ' when browsing to my local project, how do I fix this? -

i added new project. made mistake in projectname.conf file in sites-available apache2, led me reinstalling apache2. after reinstalling , re-enabling projects on every page of every project .. <body> run(); </body> .. i've run composer update, restarted service, restarted pc, checked config files, tried disabling new project. there nothing in error log, , access.log: 127.0.0.1 - - [15/may/2015:01:40:47 +0200] "get / http/1.1" 304 181 "-" "mozilla/5.0 (x11; ubuntu; linux x86_64; rv:38.0) gecko/20100101 firefox/38.0" maybe i'm forgetting apache2 reinstall? don't think did different last time installed it. looks haven't setup php correctly. take @ this great guide , maybe you're forgetting link php config apache2.
Read more

php - How can I echo out this array? -

i don't know how go grabbing data in variable , put foreach loop. class tree{ private $info = array( array('name'=>'jim','state'=>'nu','company'=>'nu','phone'=>array('cell'=>'5615111111','office'=>'5611111111'),'email'=>array('primary'=>'exs@example.com','ex@there.com')), array('name'=>'joe smith','city'=>'phoenix','phone'=>'4805551111','email'=>'jsmith@some_email.com'), array('name'=>'john doe','city'=>'chandler','company'=>'doe co.','email'=>array('jdoe@gmail.com','personal'=>'email@email.com'),'phone'=>'6025550002') ); } if you're inside of class, can access private variable using $this->variablename . example, i...
Read more