Add a function `htmlOrXmlCoarseParse` to extract content from html or xml format string. by zlx19950903 · Pull Request #19600 · ClickHouse/ClickHouse

zlx19950903 · 2021-01-26T06:57:55Z

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Mentioned in A function to extract text from HTML and XML #18454
add function htmlOrxmlCoarseParse;
support <script></script> parse;
support <style></style> parse;
support <![CDATA[]]> parse;
support white space collapse;
support any <content> format parse;
HyperScan to support SIMD;
Everything is done in a single pass.

Usage:

SELECT htmlOrXmlCoarseParse('<script>hello </script><![CDATA[world]]>')
world

…o guojiantao/htmlCoarseParse

zlx19950903 · 2021-01-27T10:17:49Z

I got a question about fast test. My source code depends on a third party named hyperscan. When I build locally, USE_HYPERSCAN will be set default 1. But in the fast test, it is set to be 0. Why does it occurs? This is a picture of cmake_log.txt output by fast test.

And this picture is cmaked locally:

abyss7 · 2021-01-27T10:58:23Z

@zlx19950903 because hyperscan is an optional dependency in our code, which means that you have to conditionally exclude your code if hyperscan is not available (i.e. checking the related macro). Your tests and code should compile and run fine in other checks.

…o guojiantao/htmlCoarseParse

zlx19950903 · 2021-01-28T04:57:07Z

@zlx19950903 because hyperscan is an optional dependency in our code, which means that you have to conditionally exclude your code if hyperscan is not available (i.e. checking the related macro). Your tests and code should compile and run fine in other checks.

If i exclude hyperscan from my source code, my function does not work. How about the query test? Shoule I also delete the unit test about my function?

ucasfl · 2021-01-28T07:06:55Z

src/Functions/htmlOrXmlCoarseParse.cpp

+
+    DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override
+    {
+        if(!isString(arguments[0]))


Style error, need a space between if and (.

…o guojiantao/htmlCoarseParse Conflicts: tests/queries/0_stateless/arcadia_skip_list.txt

abyss7 · 2021-01-29T11:53:22Z

@zlx19950903 because hyperscan is an optional dependency in our code, which means that you have to conditionally exclude your code if hyperscan is not available (i.e. checking the related macro). Your tests and code should compile and run fine in other checks.

If i exclude hyperscan from my source code, my function does not work. How about the query test? Shoule I also delete the unit test about my function?

It's expected to not work. There is an array of skipped tests inside docker/test/fasttest/run.sh - add your test here.

…o guojiantao/htmlCoarseParse

* master: (160 commits) Make Poco HTTP Server zero-copy again (ClickHouse#19516) Fixed documentation ccache 4.2+ does not requires any quirks for SOURCE_DATE_EPOCH Add a function `htmlOrXmlCoarseParse` to extract content from html or xml format string. (ClickHouse#19600) Reinterpret function added Decimal, DateTim64 support Add test Update InterpreterSelectQuery.cpp Improved serialization for data types combined of Arrays and Tuples. Improved matching enum data types to protobuf enum type. Fixed serialization of the Map data type. Omitted values are now set by default. Log stdout and stderr when failed to start docker in integration tests. Added comment Don't backport base commit of branch in the same branch (ClickHouse#20628) Fix fasttest retry for failed tests Dictionary create source with functions crash fix Added error reinterpretation tests Update run.sh Updated documentation fix subquery with limit Rename untyped function reinterpretAs into reinterpret ignore data store files Support vhost ...

* master: (153 commits) Add gdb to fasttest image Make Poco HTTP Server zero-copy again (ClickHouse#19516) Use fixed version for aerospike Fixed documentation ccache 4.2+ does not requires any quirks for SOURCE_DATE_EPOCH Add a function `htmlOrXmlCoarseParse` to extract content from html or xml format string. (ClickHouse#19600) Reinterpret function added Decimal, DateTim64 support test/stress: fix permissions for clickhouse directories test/stress: improve backtrace catching on server failures test/stress: use clickhouse builtin start/stop to run server from the same user Add test Update InterpreterSelectQuery.cpp Improved serialization for data types combined of Arrays and Tuples. Improved matching enum data types to protobuf enum type. Fixed serialization of the Map data type. Omitted values are now set by default. Log stdout and stderr when failed to start docker in integration tests. Added comment Don't backport base commit of branch in the same branch (ClickHouse#20628) Fix fasttest retry for failed tests Dictionary create source with functions crash fix Added error reinterpretation tests Update run.sh ...

alexey-milovidov · 2021-02-26T18:52:21Z

I believe this function can run faster.

This is on 80 vCPU machine:

:) SELECT count() FROM minicrawl WHERE NOT ignore(htmlOrXmlCoarseParse(content))

SELECT count()
FROM minicrawl
WHERE NOT ignore(htmlOrXmlCoarseParse(content))

→ Progress: 1.58 million rows, 143.44 GB (19.86 thousand rows/s., 1.80 GB/s.)

  18.86%  clickhouse              [.] runSheng
  14.56%  clickhouse              [.] roseRunProgram
  14.15%  clickhouse              [.] fdr_exec_teddy_msks4_pck
  11.27%  clickhouse              [.] roseCatchUpNfas
   6.28%  clickhouse              [.] handleSomInternal
   3.87%  clickhouse              [.] roseCatchUpAll
   3.75%  clickhouse              [.] nfaQueueExecToMatch
   3.70%  clickhouse              [.] roseFloatingCallback
   3.32%  clickhouse              [.] std::__1::vector<DB::(anonymous namespace)::HxCoarseParseImpl::SpanInfo, std::__1::allocator<DB::(anonymous namespace)::HxCoarseParseImpl::SpanInfo> >::push_back
   2.99%  clickhouse              [.] ZSTD_decompressSequences_bmi2
   2.77%  clickhouse              [.] DB::(anonymous namespace)::HxCoarseParseImpl::spanCollect
   1.15%  clickhouse              [.] roseNfaBlastAdaptor
   1.00%  clickhouse              [.] memcpy
   0.97%  clickhouse              [.] run_accel
   0.88%  clickhouse              [.] roseRunProgram_l
   0.85%  clickhouse              [.] roseNfaAdaptor
   0.83%  clickhouse              [.] nfaQueueExec
   0.78%  clickhouse              [.] shuftiExec
   0.73%  clickhouse              [.] handleSomExternal
   0.55%  clickhouse              [.] DB::deserializeBinarySSE2<4>
   0.51%  clickhouse              [.] DB::(anonymous namespace)::FunctionHtmlOrXmlCoarseParse::executeImpl
   0.45%  clickhouse              [.] nfaExecSheng_Q2

For max_threads = 1, the speed is just about 50 MB/sec. Looks obnoxiously slow.

guojiantao added 2 commits January 26, 2021 12:20

add html and xml coarse parse

2163516

add test file

40e14b9

robot-clickhouse added doc-alert pr-feature Pull request with new product feature labels Jan 26, 2021

abyss7 added the can be tested label Jan 26, 2021

abyss7 self-assigned this Jan 26, 2021

Merge branch 'master' of https://github.com/ClickHouse/ClickHouse int…

c97fb10

…o guojiantao/htmlCoarseParse

guojiantao added 6 commits January 27, 2021 19:21

add conditional check: hyperscan

6874085

fix style error

ed9d050

add conditional check

ec788bb

Merge branch 'master' of https://github.com/ClickHouse/ClickHouse int…

0768db9

…o guojiantao/htmlCoarseParse

bug fix

782a261

delete unit

4edca83

guojiantao added 3 commits January 28, 2021 14:18

typos check fix

c819cc3

add unit test

93013ec

style check fix

fd85eee

ucasfl reviewed Jan 28, 2021

View reviewed changes

guojiantao added 4 commits January 28, 2021 15:52

conflict fix

19406a8

fix build error: case style

fb5e758

Merge branch 'master' of https://github.com/ClickHouse/ClickHouse int…

de78dbb

…o guojiantao/htmlCoarseParse Conflicts: tests/queries/0_stateless/arcadia_skip_list.txt

acradis_skip test fix

5ee1f3b

guojiantao added 2 commits January 31, 2021 15:31

LINT error fix

5db99c1

Merge branch 'master' of https://github.com/ClickHouse/ClickHouse int…

aebfd09

…o guojiantao/htmlCoarseParse

abyss7 changed the title ~~Add a function htmlOrCoarseParse to extract content from html or xml format string.~~ Add a function htmlOrXmlCoarseParse to extract content from html or xml format string. Feb 9, 2021

abyss7 and others added 2 commits February 9, 2021 16:01

Remove comments

d1ad3a2

Merge remote-tracking branch 'origin/master' into zlx19950903/master

c51405e

abyss7 merged commit c92e613 into ClickHouse:master Feb 18, 2021

alexey-milovidov mentioned this pull request Feb 26, 2021

21.3 release checklist #20182

Closed

alexey-milovidov mentioned this pull request Feb 27, 2021

Rewrite extractTextFromHTML function #21292

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a function `htmlOrXmlCoarseParse` to extract content from html or xml format string.#19600

Add a function `htmlOrXmlCoarseParse` to extract content from html or xml format string.#19600
abyss7 merged 20 commits intoClickHouse:masterfrom
zlx19950903:master

zlx19950903 commented Jan 26, 2021

Uh oh!

zlx19950903 commented Jan 27, 2021

Uh oh!

abyss7 commented Jan 27, 2021

Uh oh!

zlx19950903 commented Jan 28, 2021

Uh oh!

ucasfl Jan 28, 2021

Uh oh!

abyss7 commented Jan 29, 2021

Uh oh!

alexey-milovidov commented Feb 26, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

zlx19950903 commented Jan 26, 2021

Uh oh!

zlx19950903 commented Jan 27, 2021

Uh oh!

abyss7 commented Jan 27, 2021

Uh oh!

zlx19950903 commented Jan 28, 2021

Uh oh!

ucasfl Jan 28, 2021

Choose a reason for hiding this comment

Uh oh!

abyss7 commented Jan 29, 2021

Uh oh!

alexey-milovidov commented Feb 26, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants