php - Scraping data using simple html dom and simpleXML -


i'm trying scrape data several links retrieve xml file. keep getting error seem appear on of news. below can see output get

http://www.hltv.org/news/14971-rgn-pro-series-groups-drawnrgn pro series groups drawn  http://www.hltv.org/news/14969-k1ck-reveal-new-teamk1ck reveal new team  http://www.hltv.org/news/14968-world-championships-captains-unveiled fatal error: call member function find() on non-object in  /app/scrape.php on line 266

where line 266

$hltv_full_text = $hltv_deep_link->find("//div[@class='rnewscontent']", 0);

full code

scrape function

function scrape_hltv() {     $hltv = "http://www.hltv.org/news.rss.php";     $sxml = simplexml_load_file($hltv);     global $con;     foreach($sxml->channel->item $item)     {         $hltv_title = (string)$item->title;         $hltv_link = (string)$item->link;         $hltv_date = date('y-m-d h:i:s', strtotime((string)$item->pubdate));         echo $hltv_link;          //if (date('y-m-d', strtotime((string)$item->pubdate)) ==  date('y-m-d')){             if (strpos($hltv_title,'video:') === false) {                 $hltv_deep_link = file_get_html($hltv_link);                 $hltv_full_text = $hltv_deep_link->find("//div[@class='rnewscontent']", 0);                   echo $hltv_title . '<br><br>';              }         //}       }  }  scrape_hltv();

there several occasions when file_get_html() returns false.

see source code here: http://sourceforge.net/p/simplehtmldom/code/head/tree/trunk/simple_html_dom.php#l79

if (empty($contents) || strlen($contents) > max_file_size) {     return false; }

for link

http://www.hltv.org/news/14968-world-championships-captains-unveiled

i think because content of page larger max_file_size(600 000 bytes). page size around 3 mbs.

if want process larger files can try modified version of function:

define('default_target_charset', 'utf-8'); define('default_br_text', "\r\n"); define('default_span_text', " ");  function file_get_html_modified($url, $use_include_path = false, $context=null, $offset = -1, $maxlen=-1, $lowercase = true, $forcetagsclosed=true, $target_charset = default_target_charset, $striprn=true, $defaultbrtext=default_br_text, $defaultspantext=default_span_text) {     $dom = new simple_html_dom(null, $lowercase, $forcetagsclosed, $target_charset, $striprn, $defaultbrtext, $defaultspantext);     $contents = file_get_contents($url, $use_include_path, $context, $offset);     if (empty($contents))     {         return false;     }     $dom->load($contents, $lowercase, $striprn);     return $dom; }

... || strlen($contents) > max_file_size removed.


Comments

Post a Comment

Popular posts from this blog

c++ - Difference between pre and post decrement in recursive function argument -

i have following sample code used pre-decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(--c); cout << c << endl; return; } } this function give output input 4 : dp3 dp2 dp1 dp0 0 1 2 3 but when use post decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(c--); cout << c << endl; return; } } output goes in infinite loop dp4 can explain me in detail why happens ? this happens because function(c--) called same value of c , when finishes c decremented. recursively called, hence called same value no chance of return , hit stack overflow error. assume this, int x = 1, y = 0; y = x++ result y == 1 , x == 2. if y = ++x , bot x , y 2.
Read more

php - Nothing but 'run(); ' when browsing to my local project, how do I fix this? -

i added new project. made mistake in projectname.conf file in sites-available apache2, led me reinstalling apache2. after reinstalling , re-enabling projects on every page of every project .. <body> run(); </body> .. i've run composer update, restarted service, restarted pc, checked config files, tried disabling new project. there nothing in error log, , access.log: 127.0.0.1 - - [15/may/2015:01:40:47 +0200] "get / http/1.1" 304 181 "-" "mozilla/5.0 (x11; ubuntu; linux x86_64; rv:38.0) gecko/20100101 firefox/38.0" maybe i'm forgetting apache2 reinstall? don't think did different last time installed it. looks haven't setup php correctly. take @ this great guide , maybe you're forgetting link php config apache2.
Read more

php - How can I echo out this array? -

i don't know how go grabbing data in variable , put foreach loop. class tree{ private $info = array( array('name'=>'jim','state'=>'nu','company'=>'nu','phone'=>array('cell'=>'5615111111','office'=>'5611111111'),'email'=>array('primary'=>'exs@example.com','ex@there.com')), array('name'=>'joe smith','city'=>'phoenix','phone'=>'4805551111','email'=>'jsmith@some_email.com'), array('name'=>'john doe','city'=>'chandler','company'=>'doe co.','email'=>array('jdoe@gmail.com','personal'=>'email@email.com'),'phone'=>'6025550002') ); } if you're inside of class, can access private variable using $this->variablename . example, i...
Read more