Is pre-compile pattern string in regexp_match operation #11146

Activity

1. zhuliquan

   added

   enhancementNew feature or requestNew feature or request

   on Jun 27, 2024
2. alamb commented on Jun 27, 2024

   

   alamb

   on Jun 27, 2024

   Contributor

   More actions

   Hi @zhuliquan -- I believe you are correct that the regexp is re-compiled for each batch. This is mentioned as one of the use cases for #8051

   I think it would be a nice improvement to look into pre-compiling the regexp somehow, though last time I checked the only benchmark we have (one of the clickbench queries) compiling the regexp was not a significant consumer of time
3. zhuliquan commented on Jun 28, 2024

   

   zhuliquan

   on Jun 28, 2024

   ContributorAuthor

   More actions

   Hi @zhuliquan -- I believe you are correct that the regexp is re-compiled for each batch. This is mentioned as one of the use cases for #8051

   I think it would be a nice improvement to look into pre-compiling the regexp somehow, though last time I checked the only benchmark we have (one of the clickbench queries) compiling the regexp was not a significant consumer of time

   I think should pre-compiled when create binary-physical-expr, instead of evaluating record_batch.
   

   datafusion/datafusion/physical-expr/src/expressions/binary.rs

   Lines 60 to 66 in 57280e4

   	pub fn new(
   	left: Arc<dyn PhysicalExpr>,
   	op: Operator,
   	right: Arc<dyn PhysicalExpr>,
   	) -> Self {
   	Self { left, op, right }
   	}
4. alamb commented on Jun 28, 2024

   

   alamb

   on Jun 28, 2024

   Contributor

   More actions

   I think should pre-compiled when create binary-physical-expr, instead of evaluating record_batch.

   I think this makes sense to me. Thank you

   One thing to look into might be to replace the regexp match operator with a function (and have the function's implementation store the precompiled regexp)

   One tricky part might be serialization (in datafusion-proto). We can probably handle it via an extension codec or something.