Disassemble monolithic process_integration.nw file

[Juergen Reuter]

Development under a slow file system like AFS becomes a huge burden!

[Juergen Reuter]

I'd really like to proceed with this task in order to get it done. Could we start typing a structure into the ticket on how to group the parts?

[kilian]

I'd rather do this gradually. Right now, it must not interfere with #646. After merging in that branch, I'll see what to do first to clean up the mess ...

[Juergen Reuter]

Sorry, but it is a PAIN IN THE ASS to work with this monster file. HOW LONG WILL YOU DELAY THIS AGAIN AND AGAIN AND AGAIN!?!?!?

[Juergen Reuter]

sorry to say this again, but work has become almost impossible.

[kilian]

After some quick P0 fixes, this is a regular major item, therefore P1.

[kilian]

Urgent tickets gone, it's time to begin this task. I'll first have a look at the contents of the misc folder, part of which comes from the main whizard.nw source.

[Juergen Reuter]

Main task now.

[kilian]

This is now under way. The plan is to split the old whizard.nw into several individual noweb sources which represent separate parts of the package.

Currently (r6156, the first revision that appears to work again), I have split off basic tools and utilities. The next step is to separate whizard objects (actual data and procedures) from sindarin objects (their handlers as exposed to the user interface). The idea is that sindarin objects should depend on whizard objects, but not vice versa. To achieve this, some types/modules have to be further disentangled and split into sindarin interface and whizard implementation.

A further step to reduce dependencies even more should be to create facade modules that expose a lightweight public interface of a set of related modules.

[kilian]

Status r6199: various lower-level modules moved to separate subdirs. More to follow.

Specifically, abstract types with multiple concrete implementations should be separate. We expect to add new implementations in the context of distinct projects.

A key was to separate the look-up part of models (parameters, fields) from the I/O part.

The actual core (processes and such) is still entangled. Thinking about this. Eventually, the Sindarin part (variables, expressions, commands etc.) should become completely independent from the calculational part, but this is not yet trivial to do. It might be useful to convert expressions (eval trees) to an abstract type, since they are present all over the place.

[Juergen Reuter]

The last sentence seems to point to the contents of #544.

[kilian]

Replying to jr_reuter:

The last sentence seems to point to the contents of #544.

Yes and no. The abstraction I was hinting at would also shield the Whizard core from the implementation details of expressions (and variables?). This is not as ambitious as #544. However, such an abstraction would be welcome when I refactor this implementation.

[kilian]

Some preliminary work for further module separation (Sindarin objects vs. WHIZARD internals) in model objects, r6311.

A sensible next step could be introducing an abstract event object, so the event I/O modules no longer depend on the event implementation. This is probably to be postponed until after upcoming release.

[Juergen Reuter]

What is remaining here? Taking the sf_XXX and eio_XXX into separate directories each?

[kilian]

Replying to jr_reuter:

What is remaining here? Taking the sf_XXX and eio_XXX into separate directories each?

Yes, these are the next steps. I'll check where else the solution that I chose for eio_XXX applies (isolating test dependencies). There are more modules that I want to remove from whizard.nw if possible (notably, mci_XXX and phs_XXX).

You may wait with the LCIO improvements until after eio_lcio has been moved (soon). The rest can be done as time allows.

[kilian]

eio_XXX modules moved by r6362. LCIO is open for further improvements.

[kilian]

After exporting the various NLO modules, the first iteration is complete. The remaining whizard.nw has become rather lean.

The issue is not finished, next steps will be

1. shift various parts of Christian's code in the WHIZARD work flow and, in passing, try to disentangle them from whizard core
2. export the event-transform base and its implementations (namely, decay and shower).

This is on a somewhat longer time scale, therefore ranking the ticket down.

[kilian]

Recent progress: detached the variables module (introducing yet another abstract type) and moved it towards the top in the hierarchy. The cross-dependency between Whizard data and Sindarin structures is now essentially confined to the whizard-core module set. Nevertheless, this should be further disentangled if possible.

[kilian]

Don't count on progress here before the upcoming release, so ranking it down here. If the higher-prio tickets happen to be fixed rather soon, I might still do something.

[Juergen Reuter]

As WK cannot violate causality to work on this, we shift it to the next minor release version.

[Juergen Reuter]

has to be finished right now.... blocks all steps all the time....

[kilian]

Replying to jr_reuter:

has to be finished right now.... blocks all steps all the time....

Well - after some obvious disentanglements have been achieved via abstraction, there are various possibilities how to proceed. Which part does block what?

BTW, the compilation time during a complete rebuild won't change at all, this is determined by the ability of the compiler to optimize across modules. What can improve is the compilation time during a partial rebuild.

[Juergen Reuter]

Rebuilding entire WHIZARD trees for release validation adding up additional time. As long as this ticket is not settled, I will demand an additional day for release preparations pushing all our deadlines one day ahead...

[kilian]

As agreed on the phone, the next item is to revise dependencies, such that moving a module will trigger a correct rebuild. Let's fork a new ticket #704. For the next short-term release, I don't plan moving modules.

[kilian]

The next steps depend on what's the plan of #544.

[kilian]

Replying to kilian:

The next steps depend on what's the plan of #544.

Actually, re-implementing the Sindarin base goes in parallel to the more immediate projects, so:

As discussed with CW: Currently, the central processes module depends on the FKS etc. implementation, which is not desirable. Introducing abstract objects that control component combination, the processes module should be cleaned, the dependency can be inverted, and eventually the modules up to processes can be split from the current whizard core.

[Juergen Reuter]

I consider the disentanlgement here a more urgent task than the SINDARIN revision.

[kilian]

Replying to jr_reuter:

I consider the disentanlgement here a more urgent task than the SINDARIN revision.

The main obstacle here is actually the entanglement of Sindarin variables and expressions with process objects. So a re-implementation of the Sindarin base is the obvious way to go.

Another obstacle is NLO-specific code in the generic process objects. I'll do something along these lines, now that we have working examples.

It may also be worthwhile to abstract process objects in the context of events, but this is a lot, and I don't want to interfere in the shower refactoring even more.

In any case, we agreed on a longer timeline for the next release, so fundamental things can be tackled.

[Juergen Reuter]

I see, well the shower refactoring should be more or less done by this week.

[Juergen Reuter]

The only modules left in whizard-core are: event_transforms.f90 hadrons.f90 decays.f90 shower.f90 events.f90 eio_raw.f90 user_files.f90 rt_data.f90 dispatch.f90 process_configurartions.f90 compilations.f90 integrations.f90 event_streams.f90 simulations.f90 expr_tests.f90 commands.f90 and the WHIZARD C interface. whizard.f90 main.f90

To me, the first 6 might be a natural block to split off, but the rest seems to be the true core. If we do agree on a policy here, we could finish this ticket. The 'core' is anyhow now only 33k instead of originally 230k lines. Comments?

[kilian]

Regarding the first 6, yes, they should go in their own subdir. Volunteers?

The rest will be reorganized again in the context of the Sindarin revision. No action now.

Now, as the actual core I'd consider the processes.f90 module and friends, which have been moved already. We may revise the subdir names at some point.

[Juergen Reuter]

I will do this then later today. Will call it 'transforms' though it is a bit more than that. Names are definitely not ideal, but these build system changes are particularly annoying, as they always cause delays for several hours. So we should reduce them to an absolute minimum, and preferrably settle things in a stable state as soon as possible.

[Juergen Reuter]

As of r6819, the event transform modules and the event API modules have been moved to src/transforms. The whizard-core.nw is now only 27k lines long. Besides maybe a final renaming of the src/XXX and one final splitting of whizard-core, I would suppose that this ticket is done. The restructuring of the expressions and variables as well as the disentanglement of the NLO code from processes might change things a little bit, but that is not part of this ticket, there are dedicated ones. So how long do we keep this one open here?

[kilian]

After removing all test-related code from the default build (#730), there is no urgent task here.

I see some benefit in detaching references to NLO matrix elements and FKS subtraction from the WHIZARD core, possibly abstracting some of the new data structures introduced there. This has to be investigated further.

[kilian]

Reviving this task.

The plan is to disentangle processes.f90 further. Dependencies on implementations of MCI, PHS, and all kinds of NLO-related stuff have crept in and should be removed step by step.

First step in r7170: remove trivial MCI implementation dependencies.

[Juergen Reuter]

Next steps will be after the release of 2.2.8.

[kilian]

to be continued after #772 and the preparations for #420.

[Juergen Reuter]

The most important (and maybe only important) thing left here is the refactoring of the processes / process_stacks setup within the process_integration.nw file.

[Juergen Reuter]

This has become joint work of CW and WK now, as a refactoring process for the NLO infrastructure (and finally then a refactoring for the state matrices and evaluators.

[Juergen Reuter]

The next step is CW's NLO refactoring.

[Juergen Reuter]

After the third refactoring (after 1.95 -> 2.0.0, 2.1.1 -> 2.2.0 now 2.3.1 -> 2.4.0) for the NLO/core/process part and the proper ordering of the chunks thereafter, everything has been done here. Closing.

[Juergen Reuter]

Milestone renamed

[Juergen Reuter]

Milestone renamed