HTML DOM parser?

Re: HTML DOM parser?

adfgvx wrote:
[color=blue]
> Try tidy. There are two python wrappers : mxtidy and utidy, the latest is[/color]

Where can we get utidy?
[color=blue]
> more recent and use the new tidylib. BUT it will only correct a bad html
> page and transform it to an xml or xhtml output that you load after as a DOM
> with another parser. Personnaly I use pyRXP.[/color]

Bye,
Walter Dörwald

Re: HTML DOM parser?

Paul Rubin <http://[email protected] nvalid> writes:
[color=blue]
> Is there an HTML DOM parser available for Python? Preferably one that
> does a reasonable job with the crappy HTML out there on real web
> pages, that doesn't get upset about unterminated tables and stuff like
> that. Many extra points if it understands Javascript. Application is
> a screen scraping web robot. Thanks.[/color]

glork. I just started working on this myself.

Email me if you'd like the code, such as it is. I've wrapped the
Mozilla JS interpreter but am currently stuck on a segfault, so I
could certainly do with a collaborator.

I'm using utidylib and 4DOM (latter from PyXML).

Mind you, if you actually want to get a job done <wink>, for a
quick-but-bulky (and somewhat closed) solution, try PyKDE (KHTML /
KJS) or IE automation (MSHTML / JScript). Mozilla + XPCOM also, but I
think it requires rebuilding Mozilla to get PyXPCOM support. There's
also httpunit (in Java, useable from Jython).


John

Re: HTML DOM parser?

mailto:gilles@p ilotsystems.net
"Paul Rubin" <http://[email protected] nvalid> a écrit dans le message de news:
[email protected] ha.com...[color=blue]
> Is there an HTML DOM parser available for Python? Preferably one that
> does a reasonable job with the crappy HTML out there on real web
> pages, that doesn't get upset about unterminated tables and stuff like
> that. Many extra points if it understands Javascript. Application is
> a screen scraping web robot. Thanks.[/color]

Windoze IE5(+) + Win32All python package only :

Use IE as COM object, browse the file or URL, then, get it's DOM root.
But any javascript found in that page is executed at page load and may fool
your app.

--Gilles

Re: HTML DOM parser?

[email protected] om writes:[color=blue]
> Here is a quick example of using automation with IE
> # This is a sample of automating IE using Python.[/color]

Thanks, I should have said I'm running under gnu/linux and I was
hoping for a standalone solution (some of the ones suggested sound
worth looking into). Even connecting up Python to Mozilla sounds
awfully heavyweight.

Re: HTML DOM parser?

Paul Rubin <http://[email protected] nvalid> writes:
[color=blue]
> [email protected] om writes:[color=green]
> > Here is a quick example of using automation with IE
> > # This is a sample of automating IE using Python.[/color]
>
> Thanks, I should have said I'm running under gnu/linux and I was
> hoping for a standalone solution (some of the ones suggested sound
> worth looking into). Even connecting up Python to Mozilla sounds
> awfully heavyweight.[/color]

PyKDE is less hassle, I think. It's certainly heavyweight, though.
Probably more lightweight still is HttpUnit on Jython. I haven't used
either, but I have compiled PyKDE recently, and didn't run into
problems (but if you're unlucky, you may have to compile Qt, KDE, sip
and PyQt first!).

I seem to have got a basic JavaScript wrapper working now (I'm using
libjs from Mozilla's standalone spidermonkey distribution), bound 4DOM
to it, and extracted & executed the script from a web page. Quite a
lot more to do, though (browser-like interface of some sort,
javascript: scheme URLs, implement window object, wiring up event
attributes to the JS interpreter, getting the DOM actually working
propertly, understanding what document.write does, trying to connect
the DOM to my Python HTML form and HTTP cookies interfaces...).

Anybody happen to know where JavaScript's document.some_f orm is
documented? Official W3C DOM has document.forms, but real browser
DOMs apparently have forms directly on the document object.


John

Re: HTML DOM parser?

[email protected] (John J. Lee) wrote in message news:<87el04yyh [email protected] >...[color=blue]
> Paul Rubin <http://[email protected] nvalid> writes:
>[color=green]
> > [email protected] om writes:[color=darkred]
> > > Here is a quick example of using automation with IE
> > > # This is a sample of automating IE using Python.[/color]
> >
> > Thanks, I should have said I'm running under gnu/linux and I was
> > hoping for a standalone solution (some of the ones suggested sound
> > worth looking into). Even connecting up Python to Mozilla sounds
> > awfully heavyweight.[/color]
>
> PyKDE is less hassle, I think. It's certainly heavyweight, though.
> Probably more lightweight still is HttpUnit on Jython. I haven't used
> either, but I have compiled PyKDE recently, and didn't run into
> problems (but if you're unlucky, you may have to compile Qt, KDE, sip
> and PyQt first!).
>
> I seem to have got a basic JavaScript wrapper working now (I'm using
> libjs from Mozilla's standalone spidermonkey distribution), bound 4DOM
> to it, and extracted & executed the script from a web page. Quite a
> lot more to do, though (browser-like interface of some sort,
> javascript: scheme URLs, implement window object, wiring up event
> attributes to the JS interpreter, getting the DOM actually working
> propertly, understanding what document.write does, trying to connect
> the DOM to my Python HTML form and HTTP cookies interfaces...).
>
> Anybody happen to know where JavaScript's document.some_f orm is
> documented? Official W3C DOM has document.forms, but real browser
> DOMs apparently have forms directly on the document object.
>
>
> John[/color]

Try here: http://msdn.microsoft.com/library/de...ence_entry.asp