Out of memory in phase space generation

[ohl]

For a certain process for ttbar + 1 Jet (i.e. q,g -> b bbar l+ l- nu nubar q for q = u,d,c,s,U,D,C,S the phase space generation is big enough to trigger the "out of memory" failure!

[kilian]

Can't reproduce. With this .sin file

alias q = u:d:c:s:U:D:C:S
alias lp = E1:E2
alias lm = e1:e2
alias nu = n1:n2
alias nubar = N1:N2
process ttj = q,g => b, bbar, lp, lm, nu, nubar, q

ms = 0  mc = 0  me = 0  mmu = 0
sqrts = 2000
integrate (ttj)

memory consumption peaks at 1GB. O'Mega needs about 200M, gfortran uses 500M (with -O3), phase space close to 1G.

It appears, however, that the memory used during phs setup is not freed properly: if I run as above, memory stays at 1G. If I create phase space and use it in a separate run, Whizard uses not more than 250M. This should be looked at (in a separate ticket?), but since peak memory is not affected, it doesn't look urgent.

1G is still manageable even for 32bit, but phase space really emerges as a bottleneck in more complicated processes. I might look for a more memory-efficient algorithm, but propose to postpone this for now. Opinions?

[Juergen Reuter]

As discussed with WK on the phone, we postpone an implementation of a more efficient algorithm at least to 2.0.5.

[Juergen Reuter]

This is really severe and prevents one from using big LHC processes. Ranking up.

[Juergen Reuter]

This becomes more and more severe an issue. FB reports about BSM processes (in the NMSSM) where no 2->6 process on 32bit allows for phase space generation.

[Juergen Reuter]

Some new values to add here: for a 2->8 process in the SM the phase space generation is now 31 GB large! In general, for 2->8 SM or 2->6 BSM processes, the phase space generation needs a week or so.

[Juergen Reuter]

Had to kill the processes with memory exceeding 50 GB! This really is show-stopper. BSM processes do not work at all.

[Juergen Reuter]

Reranked after discussion with WK.

[Juergen Reuter]

This is for sure on a longer time scale, moving it.

[Juergen Reuter]

This has become much more urgent now: the following semi-simple example is already borderline:

model = SM_CKM
alias j = u:d:c:s:b
alias J = U:D:C:S:B
process tth_had = e1, E1 => b,b,B,B,j,j,J,J
compile
sqrts = 2500
ms = 0
mc = 0
mb = 0
integrate (tth_had)

Phase space generation is running for 2.5 days now, at the moment at a size of 19 GB.

[Juergen Reuter]

Is there any news on that one here?

[Juergen Reuter]

Milestone renamed

[Juergen Reuter]

During the last Wuerzburg WHIZARD meeting end of 2016 TO confirmed that the memory seems still growing but we have no further data points. Shall we close the ticket for now, and (re)open again if we experience any more problems?

[Juergen Reuter]

Milestone renamed

[Juergen Reuter]

With the fast_wood/cascades2 method now having been merged, shall we close this issue? Or do we claim to find possible memory leaks in the (old) cascades implementation?

[Juergen Reuter]

Update from MU: he also sees this growth in virtual memory when the CASCADE2 (fast_wood) method is applied. So the 'out of memory' issue is not yet solved. MU promised to look into this.

[Juergen Reuter]

After discussion with MU and WK, MU is convinced that this cannot be further improved within the cascades2 algorithm at the moment. The problem is the setup of phs_trees/phs_forests. The advice is for users in case of memory problems to use the cascades2 framework. Only when users will then run (again) into out of memory issues we will reopen again. closing for now.