python - Scraping poorly formatted HTML with BeautifulSoup -


update: found this post while throwing spaghetti @ walls , came this, totally works in loop. csv isn't beautiful, can adapted. 

    data = []     table = soup.find('table', border=6)     rows = table.findall('tr')     row in rows:         cols = row.findall('td')         cells = [ele.text.strip() ele in cols]         data = ([ele ele in cells if ele]) # rid of empty values         #print data         record = (data)         writer = csv.writer(open('cpms10.csv', 'ab'))         writer.writerow(record)

i'm trying scrape data series of pages this one beautifulsoup. want data right side of each page in proper order column headings starting year.

i've been using this, doesn't actual year because there's padding in first row , stops after first section; when want four:

    table = soup.find('table', border=6)     data = {}        row in table.findall('tr')[2:]:         cells = row.findall('td')         key = cells[0].text.strip()         value = cells[1].text.strip()         data[key] = value         record = (key, value)         writer = csv.writer(open('cpms.csv', 'ab'))         writer.writerow(record)

i've tried adding , {'height' : '19} , , 'font' after findall('td') narrow down selection doesn't work.

this html first section of table, tho if @ whole page, there's earlier table , td never close out until end of document.

any ideas/assistance greatly, appreciated!

    <table width=845 border=6 cellpadding=0 cellspacing=0 bgcolor=#c0c0c0>     <tr><td height=28 width=19 valign=top bgcolor=#336699>&nbsp;</td>     <td valign=middle colspan=3 bgcolor=#336699><font color=white size=3><b>10-1101 - department of transportation  - dept code:a101101   - class code:01101</b></font></td></tr>     <tr><td colspan=4 height=18 valign=top>&nbsp;</td></tr>     <tr><td valign=top rowspan=54>&nbsp;</td>     <td valign=top height=18 width=255 align=left bgcolor=#ffffb4><font size=2><b>year</b></td>     <td rowspan=54 width=5 valign=top><img src='images/spacer.gif' width=10></td>     <td valign=top align='right' style='{padding-right:150px}' bgcolor=#ffffb4><font size=2>2010</td></tr>     <tr><td height=18 valign=top align=left bgcolor=#ffffb4><font size=2><b>appropriation title</b></td>     <td valign=top bgcolor=#ffffb4><font size=2>dona ana co east mesa area roads & drainage</td></tr>     <tr><td height=18 valign=top align=left bgcolor=#ffffb4><font size=2><b>fund code</b></td>     <td valign=top align='right' style='{padding-right:150px}' bgcolor=#ffffb4><font size=2>severance tax bonds</td></tr>     <tr><td height=18 valign=top align=left bgcolor=#ffffb4><font size=2><b>eo 2013-006 eligibility</b></td>     <td valign=top align='right' style='{padding-right:150px}' bgcolor=#ffffb4><font size=2></td></tr>     <tr><td height=18 valign=top align=left bgcolor=#ffffb4><font size=2><b>bond sale date</b></td>     <td valign=top align='right' style='{padding-right:150px}' bgcolor=#ffffb4><font size=2>***</td></tr>     <tr><td height=18 valign=top align=left bgcolor=#ffffb4><font size=2><b>bond series number</b></td>     <td valign=top align='right' style='{padding-right:150px}' bgcolor=#ffffb4><font size=2></td></tr>     <tr><td height=18 valign=top align=left bgcolor=#ffffb4><font size=2><b>amount of bond sale</b></td>     <td valign=top align='right' style='{padding-right:150px}' bgcolor=#ffffb4><font size=2>$0&nbsp;&nbsp;</td></tr>     <tr><td height=18 valign=top align=left bgcolor=#ffffb4><font size=2><b>category</b></td>     <td valign=top align='right' style='{padding-right:150px}' bgcolor=#ffffb4><font size=2></td></tr>     <tr><td height=18 valign=top align=left bgcolor=#ffffb4><font size=2><b>subcategory</b></td>    <td valign=top align='right' style='{padding-right:150px}' bgcolor=#ffffb4><font size=2></td></tr>    <tr><td height=18 valign=top align=left bgcolor=#ffffb4><font size=2><b>county</b></td>    <td valign=top align='right' style='{padding-right:150px}' bgcolor=#ffffb4><font size=2>dona ana</td></tr>    <tr><td height=18 valign=top align=left bgcolor=#ffffb4><font size=2><b>state amount</b></td>    <td valign=top align='right' style='{padding-right:150px}' bgcolor=#ffffb4><font size=2>$135,000</td></tr>    <tr><td height=18 valign=top align=left bgcolor=#ffffb4><font size=2><b>chapter/section</b></td>    <td valign=top align='right' style='{padding-right:150px}' bgcolor=#ffffb4><font size=2>105 / 18</td></tr>    <tr><td height=18 valign=top align=left bgcolor=#ffffb4><font size=2><b>reversion date</b></td>    <td valign=top align='right' style='{padding-right:150px}' bgcolor=#ffffb4><font size=2>6/30/2014</td></tr>    <tr><th colspan=2>share/bof data</th> <td height=12</td></tr>


Comments

Post a Comment

Popular posts from this blog

c++ - Difference between pre and post decrement in recursive function argument -

i have following sample code used pre-decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(--c); cout << c << endl; return; } } this function give output input 4 : dp3 dp2 dp1 dp0 0 1 2 3 but when use post decrement void function(int c) { if(c == 0) { return; } else { cout << "dp" << c << endl; function(c--); cout << c << endl; return; } } output goes in infinite loop dp4 can explain me in detail why happens ? this happens because function(c--) called same value of c , when finishes c decremented. recursively called, hence called same value no chance of return , hit stack overflow error. assume this, int x = 1, y = 0; y = x++ result y == 1 , x == 2. if y = ++x , bot x , y 2.
Read more

php - Nothing but 'run(); ' when browsing to my local project, how do I fix this? -

i added new project. made mistake in projectname.conf file in sites-available apache2, led me reinstalling apache2. after reinstalling , re-enabling projects on every page of every project .. <body> run(); </body> .. i've run composer update, restarted service, restarted pc, checked config files, tried disabling new project. there nothing in error log, , access.log: 127.0.0.1 - - [15/may/2015:01:40:47 +0200] "get / http/1.1" 304 181 "-" "mozilla/5.0 (x11; ubuntu; linux x86_64; rv:38.0) gecko/20100101 firefox/38.0" maybe i'm forgetting apache2 reinstall? don't think did different last time installed it. looks haven't setup php correctly. take @ this great guide , maybe you're forgetting link php config apache2.
Read more

php - How can I echo out this array? -

i don't know how go grabbing data in variable , put foreach loop. class tree{ private $info = array( array('name'=>'jim','state'=>'nu','company'=>'nu','phone'=>array('cell'=>'5615111111','office'=>'5611111111'),'email'=>array('primary'=>'exs@example.com','ex@there.com')), array('name'=>'joe smith','city'=>'phoenix','phone'=>'4805551111','email'=>'jsmith@some_email.com'), array('name'=>'john doe','city'=>'chandler','company'=>'doe co.','email'=>array('jdoe@gmail.com','personal'=>'email@email.com'),'phone'=>'6025550002') ); } if you're inside of class, can access private variable using $this->variablename . example, i...
Read more