SINDARIN revision: Variable scoping in scan

[Christian Speckner]

The current way of WHIZARD handling local variable assignments makes it impossible to export information out of loops --- assigning a value to a global instance of a variable will create a local copy which is lost when the loop is finished. Now, I remember #323, and it didn't strike me as an issue back then, but after having enough time to forget about this peculiarity, it actually bit me when trying to evaluate an integral several times and calculating mean and standard deviation by something like

real sum = 0
real sum2 = 0
scan int i = (1 => 10 /+ 1) {
   integrate (test)
   sum  = sum + integral (test)
   sum2 = sum2 + integral (test) ^ 2
}
real mean = sum / 10
real sdev = sqrt (sum2 / 10 - mean^2)

Now, I can work around this by putting whatever I want to do after the loop has finished into a conditional in the loop body, I think that our semantics for reassigning variables may be a bit counterintuitive :) Now that new variables have to be declared using the type var = ... construct, it might be a better choice for variable references within loops to access the global instance unless a local variable of the same name has been declared explicitly.

[kilian]

I'd have expected that somebody would observe this sooner ;-)

1. It's not a bug, it's a feature.
2. It's part of a bigger plan.

Since some time, I was thinking about a revision of Sindarin. This is the part relevant for the issue:

• Variables in the 'scan' body are strictly local.
• I will also revert the current (undocumented) behavior: local variables should _not_ retain their values between iterations.
• The scan body is allowed to be executed in parallel.
• Variables can be exported to the environment _after_ scan has completed. There will be an 'export' command for that.
• There will be array/list/etc. container variables, needed as 'export' targets.

However:

• There will also be a serial loop, probably 'while <condition> do <commands> enddo'. This is executed in the global environment, it is analogous to 'if .. then .. endif'.

Comments?

In principle, the Sindarin revision will get a new engine. However, I may implement part of the functionality now if there is need. I'm reluctant to implement arrays, as that is a major undertaking. What is probably easy is the 'while' loop, and exporting 'summary' variables like in the original example.

I should add that there are currently pending issues concerning physics, hence I don't push for language enhancements now ...

[Christian Speckner]

I'd have expected that somebody would observe this sooner ;-)

I wonder why it didn't strike me when we discussed #323 :)

1. It's not a bug, it's a feature.

Beauty lies in the eye of the beholder ;)

2. It's part of a bigger plan.

Sounds very reasonable to me. In particular, parallel loops directly in WHIZARD would be very elegant and powerful for parameter scans or for the generation of large event samples. However, I think the suggested (and the current) scoping behavior is both counterintuitive and complicates more mundane tasks considerably, therefore I'd suggest to introduce a serialized "for" loop with the canonical semantics in addition to the parallel construct and the "while" loop.

Anyway, I don't think this must be addressed right now. If anything, I'd opt for a "while" and a "for" loop, but I don't think it is important enough to take resources better used elsewhere. In addition, if we agree on syntax and semantics, I could implement them myself if in case I find myself with an unexpected gift of free time ;)

[kilian]

Agreed on 'for'. Suggested syntax and semantics:

scan [<type>] <var> = <list> { <script> }

• local scope
• <var> can't be modified in <script>
• executed in parallel if supported, number of iterations predetermined
• global variables can be set ('export' directive), but only after completion of the scan

for [<type>] <var> = <list> do <script> enddo

• global scope
• <var> can't be modified in <script>, number of iterations predetermined
• executed serially

while <condition> do <script> enddo

• global scope
• executed serially

Right now, all this is theory, but I'm in the process of setting up the final 'sindarin' concept (taking care of not breaking current code).

[Christian Speckner]

This is precisely what I had in mind. An interesting addition I had originally thought about was to allow the declaration of variables local to the scope which override any globals of the same name (as in C or Perl), but I guess this is a pain (at least in the current framework) which is not worth the gain.

Slightly unrelated: If you are working on designing a revision of sindarin, something I have been thinking about for a long time are namespaces; we are starting to have a proliferation of predefined variables (including the model parameters), and there is a growing chance of clashes :)

[kilian]

Namespaces: Yes, that's accounted for. Actually, models and their parameters have peculiar properties which make Sindarin somewhat interesting as a language ...

[Juergen Reuter]

I completely lost track about this ticket. What is the status? Still valid?

[Juergen Reuter]

My remark from 11 months ago is still valid????

[Juergen Reuter]

As there has been work on the variables setup in WHIZARD, what exactly is the status of this one here?

[kilian]

The ticket is still valid. In Sindarin, there should be a means for exporting values out of a local environment. This doesn't exist, still.

What did improve: local scoping correctly applies to model parameters in the same way as to other variables, as it was originally intended.

[kilian]

part of #544