Force ScriptRanges to be evaluated as ScriptArrays before popping context#324
Force ScriptRanges to be evaluated as ScriptArrays before popping context#324NeilMacMullen wants to merge 6 commits intoscriban:masterfrom
Conversation
|
I'm worried that this could be already too late (e.g if your stuffs is inside a loop that is using a loop variable)... Couldn't you do this in the evaluation of scriban/src/Scriban/Syntax/Statements/ScriptReturnStatement.cs Lines 39 to 44 in 4c8b64b The case you cover might be good enough for simple function (e.g |
This would require quite some code to handle this scenario, definitely not trivial as it would change many parts in TemplateContext. Let's keep that for a 4.0 version if someone has enough willing to bring such a change and that it doesn't introduce massive performance issues... |
|
Hmm - you are right to be concerned.. this code still throws a missing variable error although only in the specific case when the loop variable 'a' is used. (x and y are both fine). Modifying Evaluate in ScriptReturnStatement doesn't affect it and neither does modifying ScriptLoopStatementBase.
Well, there is a brute-force approach of checking the mode flag in ArrayFunctions.ApplyFunction and doing the conversion to ScriptRange there but agree it's to be avoided if possible. Gettting late here but may take another look tomorrow.... |
|
Please don't merge yet - I have found another couple of test cases to fix up |
|
@xoofx I think this PR now handles all the cases I can think of (but maybe you can think of some constructions that would still be problematic?) Probably the most controversial change is in TemplateContext.Variables where we now backtrack over the Loop-contexts in order to handle the case where a local function is declared inside a loop and wants access to the loop variable. Since invoking the function pushes a new context on the stack we need to look further backwards in order to find the loop variable. I can't see that this would affect performance/correctness for existing code but... As a nice bonus these changes also appear to fix #320 in passing. |
| var items = _currentLocalContext.Loops.Items; | ||
| for (int i = count - 1; i >= 0; i--) | ||
| //seach back through the stack of contexts in case we are | ||
| //looking for a loop variable from with a local function invocation |
There was a problem hiding this comment.
This is not a correct fix, better remove it.
Capturing a context of variables would require to collect all the context at the time a function call is captured (e.g like the filter) and to reintroduce this context when it is evaluated.
Here you are using the current context, not the one that should have been captured by the function call.
As I said, doing a proper capture is way more involving, and I would prefer that we don't handle that for now because it is complicated to get it right.
There was a problem hiding this comment.
To illustrate the issue, imagine that you have something like that:
func f(x); for y in [1..10]; ret [1,2,3] | array.each @(do; ret x + y; end;); end; end;
z = f(1)
for w in z
w
end
when evaluating z in the following for, the function @(do; ret x + y; end;); will be indirectly called. But the context of x and y will have been lost. In order to support that, Scriban would have to capture entirely the context of evaluation of variables (coming from the for loop inside the f(x) function, + the f(x) function itself). Doing this is quite challenging and I didn't introduce it in Scriban in the 1st place for this reason.
There was a problem hiding this comment.
Without this change, it's still possible to capture a loop variable like this (assignment to a temporary variable)..
for a in [1]
b=a
fn =@(do; ret b ;end}
ret fn
end
which is good enough for the scenarios I care about.
There was a problem hiding this comment.
Oh, ok, but it would introduce non wanted behavior, because here you start to look back to contextStack, meaning that you would start to access the for loop also of a calling function, for which a capture should not be supported.
e.g:
func f(x); for y in x; ret g(y); end; end;
func g(w); for z in x; ret y; end; end;
With your change, I would think that ret y would be able to access the context of the calling loop y, which wouldn't be what we want.
I don't think it can be solved like that. Scriban doesn't support the concept of closure and capture variables. It requires a different way to resolve variable, which is beyond the scope of this fix for now.
There was a problem hiding this comment.
To illustrate the issue, imagine that you have something like that:
Yes agreed, this is considerably more complicated than can be handled by the changes I have made. The motivation for these changes is to support this kind of thing:
{{
#library code
func take(a,b)
if (string.contains a b)
ret a
else
ret null
end
end
func takeLines(lines,text)
ret array.filter lines @(do;ret take($0,text);end))
end
}}
{{
#user code - having defined the functions the syntax for filtering lines is now quite clean....
model = ["log line","sensor a","another log line"]
lines =model | takeLines "sensors"
}}
Without being able to capture any variables, the only way to implement "takeLines" appears to be to use text evaluation...
{{func takeLines(lines,text)
str = "{{f= @(do;ret take($0,\"" + text +"\");end)}}"
object.eval_template str
ret (array.filter lines @f)
end}}
or perhaps to give up the array iterators completely and just use for loops...
I understand if you don't want a "partial" fix but as you say a full one really requires a fundamentally different approach. I'll back out the TemplateContext.Variable change and associated tests in any case.
| if (IsInLoop) | ||
| { | ||
| var count = _currentLocalContext.Loops.Count; | ||
| var count = _currentLocalContext.Loops.Count; |
|
Thanks, merged separately to fix issue #320 |
This was the highest place I could find to do this. Unfortunately I don't have a good enough understanding of the codebase to figure out whether it's going to kill performance in some way.
I did consider adding a mode flag to TemplateContext (AllowVariableCapture) which would allow users to opt in to the slower-but-more-powerful syntax but that seems very much like a last resort.
Also, this does not address #320 It seems the that issues has the same root cause - deferred evaluation of the ScriptRange after the context has been popped but I couldn't track down the return path for that particular scenario.