Skip to content

Force ScriptRanges to be evaluated as ScriptArrays before popping context#324

Closed
NeilMacMullen wants to merge 6 commits intoscriban:masterfrom
NeilMacMullen:item-322-capture-local
Closed

Force ScriptRanges to be evaluated as ScriptArrays before popping context#324
NeilMacMullen wants to merge 6 commits intoscriban:masterfrom
NeilMacMullen:item-322-capture-local

Conversation

@NeilMacMullen
Copy link
Copy Markdown
Contributor

This was the highest place I could find to do this. Unfortunately I don't have a good enough understanding of the codebase to figure out whether it's going to kill performance in some way.

I did consider adding a mode flag to TemplateContext (AllowVariableCapture) which would allow users to opt in to the slower-but-more-powerful syntax but that seems very much like a last resort.

Also, this does not address #320 It seems the that issues has the same root cause - deferred evaluation of the ScriptRange after the context has been popped but I couldn't track down the return path for that particular scenario.

@xoofx
Copy link
Copy Markdown
Member

xoofx commented Jan 26, 2021

I'm worried that this could be already too late (e.g if your stuffs is inside a loop that is using a loop variable)... Couldn't you do this in the evaluation of ScriptReturnStatement also?

public override object Evaluate(TemplateContext context)
{
var result = context.Evaluate(Expression);
context.FlowState = ScriptFlowState.Return;
return result;
}

The case you cover might be good enough for simple function (e.g f(x) = x | array.filter ...)

@xoofx
Copy link
Copy Markdown
Member

xoofx commented Jan 26, 2021

I did consider adding a mode flag to TemplateContext (AllowVariableCapture) which would allow users to opt in to the slower-but-more-powerful syntax but that seems very much like a last resort.

This would require quite some code to handle this scenario, definitely not trivial as it would change many parts in TemplateContext. Let's keep that for a 4.0 version if someone has enough willing to bring such a change and that it doesn't introduce massive performance issues...

@NeilMacMullen
Copy link
Copy Markdown
Contributor Author

Hmm - you are right to be concerned.. this code still throws a missing variable error although only in the specific case when the loop variable 'a' is used. (x and y are both fine). Modifying Evaluate in ScriptReturnStatement doesn't affect it and neither does modifying ScriptLoopStatementBase.

func call_array(p)
  x =true
  for a in [true]
  y=true
  ret  [1,2,3] | array.filter @(do;ret a;end)
end
end
call_array true

This would require quite some code to handle this scenario, definitely not trivial as it would change many parts in TemplateContext

Well, there is a brute-force approach of checking the mode flag in ArrayFunctions.ApplyFunction and doing the conversion to ScriptRange there but agree it's to be avoided if possible.

Gettting late here but may take another look tomorrow....

@NeilMacMullen
Copy link
Copy Markdown
Contributor Author

Please don't merge yet - I have found another couple of test cases to fix up

@NeilMacMullen
Copy link
Copy Markdown
Contributor Author

NeilMacMullen commented Jan 27, 2021

@xoofx I think this PR now handles all the cases I can think of (but maybe you can think of some constructions that would still be problematic?)

Probably the most controversial change is in TemplateContext.Variables where we now backtrack over the Loop-contexts in order to handle the case where a local function is declared inside a loop and wants access to the loop variable. Since invoking the function pushes a new context on the stack we need to look further backwards in order to find the loop variable. I can't see that this would affect performance/correctness for existing code but...

As a nice bonus these changes also appear to fix #320 in passing.

var items = _currentLocalContext.Loops.Items;
for (int i = count - 1; i >= 0; i--)
//seach back through the stack of contexts in case we are
//looking for a loop variable from with a local function invocation
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a correct fix, better remove it.

Capturing a context of variables would require to collect all the context at the time a function call is captured (e.g like the filter) and to reintroduce this context when it is evaluated.

Here you are using the current context, not the one that should have been captured by the function call.

As I said, doing a proper capture is way more involving, and I would prefer that we don't handle that for now because it is complicated to get it right.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To illustrate the issue, imagine that you have something like that:

func f(x); for y in [1..10]; ret [1,2,3] | array.each @(do; ret x + y; end;); end; end;
z = f(1)
for w in z
  w
end

when evaluating z in the following for, the function @(do; ret x + y; end;); will be indirectly called. But the context of x and y will have been lost. In order to support that, Scriban would have to capture entirely the context of evaluation of variables (coming from the for loop inside the f(x) function, + the f(x) function itself). Doing this is quite challenging and I didn't introduce it in Scriban in the 1st place for this reason.

Copy link
Copy Markdown
Contributor Author

@NeilMacMullen NeilMacMullen Jan 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without this change, it's still possible to capture a loop variable like this (assignment to a temporary variable)..

for a in [1]
b=a
fn =@(do; ret b ;end}
ret fn
end

which is good enough for the scenarios I care about.

Copy link
Copy Markdown
Member

@xoofx xoofx Jan 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, ok, but it would introduce non wanted behavior, because here you start to look back to contextStack, meaning that you would start to access the for loop also of a calling function, for which a capture should not be supported.

e.g:

func f(x); for y in x; ret g(y); end; end;
func g(w); for z in x; ret y; end; end;

With your change, I would think that ret y would be able to access the context of the calling loop y, which wouldn't be what we want.

I don't think it can be solved like that. Scriban doesn't support the concept of closure and capture variables. It requires a different way to resolve variable, which is beyond the scope of this fix for now.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To illustrate the issue, imagine that you have something like that:

Yes agreed, this is considerably more complicated than can be handled by the changes I have made. The motivation for these changes is to support this kind of thing:

{{
#library code

func take(a,b)
 if (string.contains a b)
   ret a
 else 
   ret null
 end
end

func takeLines(lines,text)
  ret array.filter lines @(do;ret take($0,text);end))
end
}}

{{
#user code - having defined the functions the syntax for filtering lines is now quite clean....
model = ["log line","sensor a","another log line"]
lines =model  |  takeLines  "sensors" 
}}

Without being able to capture any variables, the only way to implement "takeLines" appears to be to use text evaluation...

{{func takeLines(lines,text)
str = "{{f=  @(do;ret take($0,\"" + text +"\");end)}}"
object.eval_template str
ret (array.filter lines @f)
end}}

or perhaps to give up the array iterators completely and just use for loops...

I understand if you don't want a "partial" fix but as you say a full one really requires a fundamentally different approach. I'll back out the TemplateContext.Variable change and associated tests in any case.

if (IsInLoop)
{
var count = _currentLocalContext.Loops.Count;
var count = _currentLocalContext.Loops.Count;
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should not be changed I think

@xoofx
Copy link
Copy Markdown
Member

xoofx commented Feb 1, 2021

Thanks, merged separately to fix issue #320

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants