Restore full functionality of 2.1.1.

[kilian]

I started revising the modules that handle process libraries and processes. Several problems with the old code make this necessary.

The first check-in with work-in-progress is r4027.

Goals:

• Handle matrix elements of arbitrary kinds (internal, external, hand-written, etc.) by a common process object.
• Multiple components of a process (some of them may be automatically generated, such as dipole subtractions).
• Remove unnecessary hard-coded assumptions, such as the use of VAMP for integration.
• Separate interface from implementation. For instance, all OMega-specific stuff is now in a separate module that extends a couple of abstract generic types.
• All new code should be covered by unit tests from the beginning, as far as it makes sense.

The original motivation was:

For various applications (in particular, NLO), a process has multiple subprocesses with kinematics and event samples that are partly common, partly separate. The process object has to be extended such as to properly handle those.

[Juergen Reuter]

Somehow this ticket slipped through... Is that not done when we merge CSs branch?

[kilian]

No. We agreed that I'll review CS's mods and maybe rewrite the process object in a more transparent way.

[Juergen Reuter]

Ok, that changes priorities then...

[Juergen Reuter]

Would be great, WK, if you could revisit this. Because: then, we could go and merge this with the trunk while I'm visiting CS in Freiburg in 2 weeks.

[Juergen Reuter]

Actually, this becoming one of the major stumbling stones for the project! It should be resolved at best before the next meeting!!!

[kilian]

This issue is directly connected to the recent merge of the NLO infrastructure. The process object and its components have to be properly (re)structured, incorporating the NLO additions and allowing for present and future extensions. This task requires a major rewrite of (part of, at least) the process-object related code.

Until this is done, don't expect the system to function properly. Currently, WHIZARD compiles, and basic tests and processes do work, but the decay and structure-function parts are defective.

[kilian]

As long as #468 is under construction, this has to be postponed again.

[kilian]

Wrong ticket rated down, correction ...

[Juergen Reuter]

Any progress here ... ???

[kilian]

Since you're asking:

I have agreed (with myself) on a new, much better structured process record. The functionality won't really change, but it is tedious to go through all subobjects and sub-subobjects to implement that change. I'm working with a temporary GIT repository locally, as ThO suggested. This means that I will touch the svn repository only once in a while. If anybody has urgent unrelated changes to whizard.nw, I can pull them in and keep up to date.

[Juergen Reuter]

Progress? ...

[Juergen Reuter]

Happy New Year. Any progress? This is getting terribly urgent!

[Juergen Reuter]

At least with gfortran 4.8.0 there is an ICE :( Will test today 4.5.x, 4.6.x, 4.7.x

[Juergen Reuter]

This is only the next step.

[Juergen Reuter]

Puh, this looks like a mess now. Looks like that this can be combined with #327 now.

[kilian]

Not yet. My current work is admittedly too slow, but still localized.

[Juergen Reuter]

Seems that the DPG meeting stalled the progress :(

[kilian]

Replying to jr_reuter:

Seems that the DPG meeting stalled the progress :(

Well - no. Stay tuned ...

[Juergen Reuter]

Process and process handling is done in OO form. Now, everything has to be reenabled and all features have to be reimplemented or switched on again (incl. proper testing!).

[Juergen Reuter]

Seems we are getting closer ... but SINDARIN steering is not yet back!

[kilian]

With r4272, Sindarin input can be parsed again. I will continue by resurrecting a few commands that implement minimal functionality: integration of a test process.

[kilian]

[read the fine print: right now, ALL Sindarin commands are still deactivated, so don't try ...]

[Juergen Reuter]

A bit more has been reactivated now, but still no trivial things doable. JRR will step in at the earliest feasible point (for structure functions and event formats).

[Juergen Reuter]

Most of the code is functional again, though there still is a segmentation fault when reading in a process defintion in the SINDARIN file. Now, it is important to restore the full functionality of 2.1.1.

[Juergen Reuter]

So how far are we now!?

[kilian]

r4413: Finally, we can compute integrals for real processes. Still far from full functionality. Here is a sketchy priority list:

1. Reactivate more of VAMP, in particular event generation and Grid I/O
2. Structure functions, at least for LHC processes
3. Event I/O

This may qualify for an alpha version.

4. Reactivate Sindarin constructs (many are currently disabled)
5. Recovery tools (re-reading of input, MD5 sums etc.)
6. More features of 2.1.1

Another milestone. Maybe release as beta with reduced functionality.

7. Remaining structure functions
8. Decays and correlations

Qualify for beta or experimental release.

9. Possibly new features: process containers etc.
10. Complete features, tests etc.

Release.

[Juergen Reuter]

1. almost done. 6a: Analysis. 8a. Shower and Pythia-Interface incl. MLM-Matching.

[kilian]

status 13/08/22:

1. Reactivate VAMP: done
2. Structure functions for LHC: PDF builtin done, LHAPDF w.i.p.
3. Event I/O: raw format done, other formats not yet.

4. Reactivate Sindarin: various variables and commands; control structures not yet.
5. Recovery tools: raw event files done, MD5 sums not yet.

[kilian]

status 28/08: we are moving to the stage where we list things that don't work (yet), instead of listing a few features that do work. The next step is splitting off the subset of features that must be present in an alpha version. We are close, actually.

[Juergen Reuter]

The following commands should be reinstated for 2.2.0:

• beam_polarization
• close_out (#542)
• exec
• expect
• fac_scale
• matrix_element_test (maybe rename)
• open_out (cf. #542)
• polarized (cf. #518)
• printf
• quit
• ren_scale
• reweight
• scale
• scan
• selection
• show
• stable (cf. #518)
• unpolarized (cf. #518)
• unstable (cf. #518)
• slha
• weight

To be deleted:

• load
• printd
• histogram_writer, plot_writer
• CS's NLO customs

JRR will delete, WK will re-create ...

[Juergen Reuter]

load and printd gone in r4718.

[Juergen Reuter]

In r4720, histogram_writer and plot_writer and CS NLO customs are gone. They will be reimplemented by WK and CW, respectively. There are still some leftovers which we will have to discuss like get_analysis_tags etc.

[Juergen Reuter]

This includes a final check of the attic code and whether everything is in the program ...

[Juergen Reuter]

Ok. let me start a list of things to be discussed:

1. grove_weights

[ 2. (equiv. of) core_interaction_write_summary (summary of quantum number states). Done in r5580. ] [ 3. process_status_counter: passed cuts, etc. Done in r5590. ] [ 4. handling of negative weights (was bookkeeping up to now, I would say). Partially done in r5498. Still, the vamp_sample_grids calls have to know about the option. Finally done in r5500, r5501. ] [ 5. check use of 'use tao_random_numbers'. Done in the meanwhile. ] [ 6. check lots of unused 'use XXX' in commands.f90] [ 7. ?polarized_events flag: only used (afaik) for the simulation message. (was actually also for setting the FM_XXX_HELICITY mode, done in r5263] [ 8. FORBIDDEN_ENDINGS check and file taken from file_list for analysis framework (in cmd_write_analysis_compile). This should be checked and reinstated (done in r5249)] [ 9. same for the error tags and the comment tags (though there is a separate ticket #542)] [10. name expression in cmd_write_analysis has to be reinserted (separate ticket #604)] [11. Several occasions where gfortran 4.6.3/4 bugs have been evaded. All that can be reverted as we are considering versions >= 4.7.1 now anyhow. -- discussed this in Wuerzburg -> we leave the safeguards.] [12. Remove undefined noweb chunks] [13. Reinsert MAX_TRIES_FOR_SINGLE_EVENT apparently is in, event to be set as user] [14. libraries variable in main.f90] [15. check for "call mask_term_signals"]

16. Chech that relocation scripts are still working properly ...
17. Discuss and check the rationale of this ticket above ...

[ 18. What about the subroutine particle_set_fill_hepmc_event ? JRR already some time ago recognized that this is now the subroutine hepmc_event_from_particle_set. However, it appears in eio_hepmc.f90. Question is what we do with the test (issue nr. 20). In r5593, I deleted the old particle_set_fill_hepmc_event and moved particle_to_hepmc and hepmc_event_from_particle_set to particles.f90 instead. Still have to think about nr. 20] [ 19. Dito for particle_init_hepmc ... This has been done in r5346.]

20. Dito for hepmc_test ...

[21. A lot of chunk (commented out stuff) in integrations.f90 ... separate ticket #610] [22. Reinstate the "event normalization" message (also in the default case) (?)]

23. Check numerical noise in weight adaptation (32 vs. 64bit)

[ 24. Check "gw", "g", "w", "" options -> separate ticket #622 now. Done in r5562.] [ 25. Check timer_start, timer_end in integration_results.f90. Removed in r5583.]

26. Discuss the "JRR: WK please check" points in the code
27. recompile_library
28. phs_forest_evaluate_momenta (still XXXed out)
29. particle_set_extract_interactions (seems to have been used for reconstructing the hard matrix element from an event via simulation_event (read), simulation_recover_process, event_recover_process, process_recover_kinematics, hard_interaction_recover_kinematics. Seems to be superseeded (?) by the process_instance_recover(_XYZ) functions called from e.g. simulation_rescan ???

[Juergen Reuter]

This message

|=============================================================================|
| vamp: parameter mismatch, discarding grid file 'beam_setup_1p.vg'

appears also for a new integration. It is just cosmetics ... But maybe we could do something about it.

[kilian]

The re-implementation of the graph command might still be incomplete, should check this.

[Juergen Reuter]

Several maintenance steps: beam_fmt eliminated, process_library_check_name_consistency no longer needed (checked), also I introduced a WHIZARD bug message for the case that the mci_integral is not known.

[Juergen Reuter]

Next step: veto against flavor sums with different masses has been reenabled.

[Juergen Reuter]

I actually checked that special MD5 sums for SF parameters are not needed. Changing them correctly leaves the PHS untouched, but the VAMP grids are newly built. Also, min_calls_per_bin and min_calls_per_channel have been reactivated.

[kilian]

I'll see through this list tomorrow. Points left open should also be discussed on Monday, if possible.

[Juergen Reuter]

All remarks above are closed, a.t.m. the only left-over is

A check of the relocation scripts.

[Juergen Reuter]

Left-over went into a separate ticket. Closing