htmllib.py and parsing malformed HTML

Re: htmllib.py and parsing malformed HTML

KC wrote:
[color=blue]
> I have written a parser using htmllib.HTMLPar ser and it functions fine
> unless the HTML is malformed. For example, is some instances, the
> provider of the HTML leaves out the <TR> tags but includes the </TR> tags.
>
> Apparently, htmllib and more likely sgmllib do not parse an end tag if a
> corresponding start tag was not found. Does anyone know a way to "fool"
> the parser into handling the end tag is a start tag was not found?[/color]

Hi,

You could use tidy (http://www.w3.org/People/Raggett/tidy/) before you
parse the html.

thomas

Re: htmllib.py and parsing malformed HTML

Thomas Güttler wrote:[color=blue]
>
> Hi,
>
> You could use tidy (http://www.w3.org/People/Raggett/tidy/) before you
> parse the html.[/color]

I appreciate the suggestion but unfortunately this will not work well
for me as the parser runs as part of a cron job. I wouldn't be able to
review the tidy error log in a timely fashion if there was a problem.

What would be really nice is a way to tell the parser it was "inside" a
<TR> when I encountered a <TD> after a closing </TR>. Browsers still
display the HTML correctly without a starting <TR>, but if the closing
</TR> is omitted everything gets mangled.

Any other suggestions?

Re: htmllib.py and parsing malformed HTML [SOLVED]

KC wrote:[color=blue]
>
> What would be really nice is a way to tell the parser it was "inside" a
> <TR> when I encountered a <TD> after a closing </TR>. Browsers still
> display the HTML correctly without a starting <TR>, but if the closing
> </TR> is omitted everything gets mangled.
>[/color]
I solved this problem, perhaps not the most elegant way, but it is still
solved. Any suggestions on improvements are welcome. I added the
following method to my parser class to make this work:


def parse_endtag(se lf, i) :
rawdata = self.rawdata
tag = rawdata[i+2:i+4].strip().lower( )
if tag == 'tr' :
self.fmtr.write r.send_tag('</TR>')
return htmllib.HTMLPar ser.parse_endta g(self, i)


I should also mention that I added the send_tag method to my writer
implementation which simply writes the given text to the output stream.

Re: htmllib.py and parsing malformed HTML

KC <nskhcarlso@bel lsouth.net> writes:
[color=blue]
> Thomas Güttler wrote:[color=green]
> > Hi,
> > You could use tidy (http://www.w3.org/People/Raggett/tidy/) before
> > you
> > parse the html.[/color]
>
> I appreciate the suggestion but unfortunately this will not work well
> for me as the parser runs as part of a cron job. I wouldn't be able
> to review the tidy error log in a timely fashion if there was a
> problem.[/color]
[...]

So, what about *your* code's error log (or the equivalent --
presumably an unhandled traceback)?? It's not obvious that your
solution (in a later post) will be any more robust than just piping
everything through HTMLTidy. In fact, since you will find a great
variety of nonsense in 'HTML as deployed', it seems likely that
HTMLTidy will do the better job.


John

Re: htmllib.py and parsing malformed HTML

John J. Lee wrote:
[color=blue]
>
> So, what about *your* code's error log (or the equivalent --
> presumably an unhandled traceback)?? It's not obvious that your
> solution (in a later post) will be any more robust than just piping
> everything through HTMLTidy. In fact, since you will find a great
> variety of nonsense in 'HTML as deployed', it seems likely that
> HTMLTidy will do the better job.
>[/color]

If this parser was handling a "great variety of nonsense" I would
wholeheartedly agree with you. However, since this HTML is from a
single vendor and that vendor is a government entity, this solution was
better than integrating a third-party product. As with most
organizations, changing *our* code is much more acceptable to the powers
that be, than bringing in a third-party product that will have to be
evaluated and have countless meetings over its approval. For many of
us, business and policy decisions often forge the direction for
technology usage within our organizations.

Re: htmllib.py and parsing malformed HTML

KC <nskhcarlso@bel lsouth.net> writes:
[color=blue]
> John J. Lee wrote:
>[color=green]
> > So, what about *your* code's error log (or the equivalent --
> > presumably an unhandled traceback)?? It's not obvious that your[/color][/color]
[...][color=blue]
> If this parser was handling a "great variety of nonsense" I would
> wholeheartedly agree with you. However, since this HTML is from a
> single vendor and that vendor is a government entity, this solution[/color]

Oh, got you. Fair enough


[...][color=blue]
> for technology usage within our organizations.[/color]

You can always tell when someone's 'business button' has been pushed
when they use the word 'within' ;-)


John

Re: htmllib.py and parsing malformed HTML

On Thu, 04 Sep 2003 11:50:07 -0400, KC wrote:[color=blue]
> As with most organizations,
> changing *our* code is much more acceptable to the powers that be, than
> bringing in a third-party product that will have to be evaluated and have
> countless meetings over its approval. For many of us, business and policy
> decisions often forge the direction for technology usage within our
> organizations.[/color]

If you are having real problems with poor HTML, HTMLTidy may be worth
going to bat over. If you can find a simple solution that works on the
HTML you are processing, great, go with it, and it's worth researching in
your situation first. But HTML can go bad in more ways then you can
imagine (which is in fact part of the problem); if you are getting HTML
that's bad in a lot of little ways, you'll find the "apply a hack to fix
this file, apply a hack to fix that file" will start stepping on its own
toes.

HTMLTidy represents a ***lot*** of grunt work and a ***lot*** of
functionality that you can *not* replicate in a reasonable amount of time;
it's one of those packages that isn't so much a program that "does
something" as a program that represents many, many man-years of "knowledge
acquired".

I'm not trying to push anything, since I don't know your situation, but
HTMLTidy is one of those rare projects that you really shouldn't allow NMH
to scuttle unless you *really* need to. (Again, I mention if there's some
simple way you can characterize the bad HTML coming out of one single
program, go ahead and try to fix it; maybe you'll get lucky and a regex
will be enough.)

Re: htmllib.py and parsing malformed HTML

Jeremy Bowers wrote:[color=blue]
> On Thu, 04 Sep 2003 11:50:07 -0400, KC wrote:
>[/color]
....
[color=blue]
> that's bad in a lot of little ways, you'll find the "apply a hack to fix
> this file, apply a hack to fix that file" will start stepping on its own
> toes.[/color]
Oh yeah, I couldn't agree more. Any more requests for "hacks" and
HTMLTidy gets brought into the picture.[color=blue]
>
> HTMLTidy represents a ***lot*** of grunt work and a ***lot*** of
> functionality that you can *not* replicate in a reasonable amount of time;
> it's one of those packages that isn't so much a program that "does
> something" as a program that represents many, many man-years of "knowledge
> acquired".
>[/color]
Agreed. I like HTMLTidy very much and it's obvious it could save us
developers a lot of effort.