attempt to simplify and reduce memory for multiringbuffer #2523

ahnitz · 2019-02-26T22:44:18Z

No description provided.

titodalcanton · 2019-02-27T15:22:28Z

This is being tested.

titodalcanton · 2019-02-27T15:24:54Z

I immediately got this:

2019-02-27 07:23:44,326 node742 0 background and zerolag coincs
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/tito/projects/pycbc/live/multidetector/env/lib/python2.7/site-packages/mpi4py/__main__.py", line 7, in <module>
    main()
  File "/home/tito/projects/pycbc/live/multidetector/env/lib/python2.7/site-packages/mpi4py/run.py", line 196, in main
    run_command_line(args)
  File "/home/tito/projects/pycbc/live/multidetector/env/lib/python2.7/site-packages/mpi4py/run.py", line 47, in run_command_line
    run_path(sys.argv[0], run_name='__main__')
  File "/usr/lib64/python2.7/runpy.py", line 240, in run_path
    return _run_module_code(code, init_globals, run_name, path_name)
  File "/usr/lib64/python2.7/runpy.py", line 82, in _run_module_code
    mod_name, mod_fname, mod_loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/tito/projects/pycbc/live/multidetector/env/bin/pycbc_live", line 573, in <module>
    coinc_results.append(c.add_singles(results, data_reader))
  File "/home/tito/projects/pycbc/live/multidetector/env/lib/python2.7/site-packages/pycbc/events/coinc.py", line 1116, in add_singles
    ifos=valid_ifos)
  File "/home/tito/projects/pycbc/live/multidetector/env/lib/python2.7/site-packages/pycbc/events/coinc.py", line 1058, in _find_coincs
    coinc_results['background/time'] = numpy.array([self.background_time])
  File "/home/tito/projects/pycbc/live/multidetector/env/lib/python2.7/site-packages/pycbc/events/coinc.py", line 838, in background_time
    time *= len(self.singles[ifo]) * self.analysis_block
TypeError: object of type 'MultiRingBuffer' has no len()

titodalcanton · 2019-02-27T15:37:52Z

Can we just rename MultiRingBuffer.num_elements to __len__?

ahnitz · 2019-02-27T16:16:40Z

I've renamed it so it isn't len either, as it really was something else. It was supposed to be the number of time steps that have been added to the buffer. This maxes out at the max buffer size, but before this we need to know how many have been taken to correctly know the total background time.

pycbc/events/coinc.py

titodalcanton · 2019-02-28T16:25:39Z

New error:

Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/tito/projects/pycbc/live/multidetector/env/lib/python2.7/site-packages/mpi4py/__main__.py",
 line 7, in <module>
    main()
  File "/home/tito/projects/pycbc/live/multidetector/env/lib/python2.7/site-packages/mpi4py/run.py", line
 196, in main
    run_command_line(args)
  File "/home/tito/projects/pycbc/live/multidetector/env/lib/python2.7/site-packages/mpi4py/run.py", line
 47, in run_command_line
    run_path(sys.argv[0], run_name='__main__')
  File "/usr/lib64/python2.7/runpy.py", line 240, in run_path
    return _run_module_code(code, init_globals, run_name, path_name)
  File "/usr/lib64/python2.7/runpy.py", line 82, in _run_module_code
    mod_name, mod_fname, mod_loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/tito/projects/pycbc/live/multidetector/env/bin/pycbc_live", line 573, in <module>
    coinc_results.append(c.add_singles(results, data_reader))
  File "/home/tito/projects/pycbc/live/multidetector/env/lib/python2.7/site-packages/pycbc/events/coinc.p
y", line 1118, in add_singles
    updated_indices = self._add_singles_to_buffer(results, ifos=valid_ifos)
  File "/home/tito/projects/pycbc/live/multidetector/env/lib/python2.7/site-packages/pycbc/events/coinc.p
y", line 939, in _add_singles_to_buffer
    self.singles[ifo].add(trigs['template_id'], data)
  File "/home/tito/projects/pycbc/live/multidetector/env/lib/python2.7/site-packages/pycbc/events/coinc.p
y", line 578, in add
    self.buffer[i] = numpy.concatenate([self.buffer[i], v])
ValueError: all the input arrays must have same number of dimensions

Apparently v is a numpy.void and does not like to be concatenated with an empty array.

titodalcanton · 2019-02-28T20:11:26Z

@ahnitz after the commits I just did, the analysis is running without errors. Could you double check that I did not mess anything up?

ahnitz · 2019-02-28T20:13:26Z

@titodalcanton Nothing glaringly obvious at least.

titodalcanton · 2019-03-01T15:37:32Z

After several hours of running, the memory usage appears still limited to several Gb. Yay!

titodalcanton · 2019-03-01T17:32:57Z

@ahnitz is this still a work in progress or just needs to be tested? Is the result expected to be identical to the previous version? If so, that is an easy test.

ahnitz · 2019-03-01T18:31:04Z

It just needs to be tested. Results should be identical.

…

On Fri, Mar 1, 2019, 18:32 Tito Dal Canton ***@***.***> wrote: @ahnitz <https://github.com/ahnitz> is this still a work in progress or just needs to be tested? Is the result expected to be identical to the previous version? If so, that is an easy test. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2523 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACGrRrqkGcU5bpr3mgIyM6xTmCpdgv39ks5vSWRKgaJpZM4bTSs3> .

titodalcanton

Looks good and tests are underway.

titodalcanton · 2019-03-02T04:00:31Z

After running on my fake frames with injections, the exact same set of HDF and coinc XML files is generated, with identical file names, with respect to a run done with v1.13.4. However, I compared a few coinc XML files in detail and they are not identical.

Most of the differences are tiny changes in the SNR time series and PSDs, which look like numerical rounding in the least significant digits. I was a bit surprised to see even such a tiny difference, but whatever.

However in one case (coinc-1169265807.xml.gz for self reference) the coinc_inspiral:combined_far value went from 3.491173665171772e-05 to 0.0001112754634923875. The coinc_inspiral:snr is equal to 6 significant digits. I do not remember seeing combined_far change in a couple other coinc files. What do you think?

ahnitz · 2019-03-02T13:46:40Z

Hmmm.. hard to say. One issue is that pycbc live can be less numerically stable or long running than a standard analysis. This is because it makes choices to redo or reuse parts of the analysis (like the psd) depending on ift it's changes sufficiently. This can mean that a different PSD could be used for a stretch of data. One way to reduce this (for testing only) would be to force it to always recalculate the psd, i.e. set the threshold very low or to zero.

…

On Sat, Mar 2, 2019 at 5:00 AM Tito Dal Canton ***@***.***> wrote: After running on my fake frames with injections, the exact same set of HDF and coinc XML files is generated, with identical file names, with respect to a run done with v1.13.4. However, I compared a few coinc XML files in detail and they are not identical. Most of the differences are tiny changes in the SNR time series and PSDs, which look like numerical rounding in the least significant digits. I was a bit surprised to see even such a tiny difference, but whatever. However in one case (coinc-1169265807.xml.gz for self reference) the coinc_inspiral:combined_far value went from 3.491173665171772e-05 to 0.0001112754634923875. The coinc_inspiral:snr is equal to 6 significant digits. I do not remember seeing this field change in a couple other coinc files. What do you think? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2523 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACGrRvtTOqg9ol-4SK4-agBMsC4C9tN5ks5vSfdfgaJpZM4bTSs3> .

-- Dr. Alexander Nitz Max Planck Institute for Gravitational Physics (Albert Einstein Institute) Callinstrasse 38 D-30167 Hannover, Germany Tel: +49 511 762-17097

titodalcanton · 2019-03-04T03:09:49Z

Here is a comparison of the FARs for the coinc events generated for the injection run. X shows basically time (event index), y shows new / old FAR. So FARs tend to be systematically larger now.

ahnitz · 2019-03-04T21:55:26Z

@titodalcanton Yeah that's disconcerting. I think what I'll do is put together a test script with these two version and feed them random numbers to see what happens. One of them must have a bug, I think.

ahnitz · 2019-03-04T22:47:29Z

@titodalcanton Based on my test (see pycbc chat for more details) I think this is the right implementation and fixes a bug in the current version that is hit when the buffer resizes. My suggestion is that we merge this patch. What do you think?

titodalcanton · 2019-03-05T01:11:21Z

Ok I think I get what the problem with the old class is. Let's merge and keep testing this, also given how much memory the old approach requires.

attempt to simplify and reduce memory for multiringbuffer

c388138

ahnitz added the work in progress label Feb 26, 2019

ahnitz requested a review from titodalcanton February 26, 2019 22:44

update

c077b50

Small fix for logging

ceb3fb2

titodalcanton reviewed Feb 28, 2019

View reviewed changes

pycbc/events/coinc.py Outdated Show resolved Hide resolved

Fixes

6f48b65

titodalcanton added 2 commits February 28, 2019 11:28

More fixes

6340b3e

Buuuuuugs

4d98525

Improve docstring

6600895

titodalcanton removed the work in progress label Mar 1, 2019

titodalcanton approved these changes Mar 1, 2019

View reviewed changes

titodalcanton mentioned this pull request Mar 1, 2019

Improve PyCBC Live logging, help and metadata #2538

Merged

titodalcanton added this to the ER14 milestone Mar 4, 2019

titodalcanton merged commit a3de734 into gwastro:master Mar 5, 2019

titodalcanton mentioned this pull request Mar 13, 2019

memory leak in pycbc live #2430

Closed

titodalcanton added the low latency label Nov 22, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

attempt to simplify and reduce memory for multiringbuffer #2523

attempt to simplify and reduce memory for multiringbuffer #2523

ahnitz commented Feb 26, 2019

titodalcanton commented Feb 27, 2019

titodalcanton commented Feb 27, 2019

titodalcanton commented Feb 27, 2019

ahnitz commented Feb 27, 2019

titodalcanton commented Feb 28, 2019

titodalcanton commented Feb 28, 2019

ahnitz commented Feb 28, 2019

titodalcanton commented Mar 1, 2019

titodalcanton commented Mar 1, 2019

ahnitz commented Mar 1, 2019 via email

titodalcanton left a comment

titodalcanton commented Mar 2, 2019 •

edited

Loading

ahnitz commented Mar 2, 2019 via email

titodalcanton commented Mar 4, 2019

ahnitz commented Mar 4, 2019

ahnitz commented Mar 4, 2019

titodalcanton commented Mar 5, 2019

attempt to simplify and reduce memory for multiringbuffer #2523

attempt to simplify and reduce memory for multiringbuffer #2523

Conversation

ahnitz commented Feb 26, 2019

titodalcanton commented Feb 27, 2019

titodalcanton commented Feb 27, 2019

titodalcanton commented Feb 27, 2019

ahnitz commented Feb 27, 2019

titodalcanton commented Feb 28, 2019

titodalcanton commented Feb 28, 2019

ahnitz commented Feb 28, 2019

titodalcanton commented Mar 1, 2019

titodalcanton commented Mar 1, 2019

ahnitz commented Mar 1, 2019 via email

titodalcanton left a comment

Choose a reason for hiding this comment

titodalcanton commented Mar 2, 2019 • edited Loading

ahnitz commented Mar 2, 2019 via email

titodalcanton commented Mar 4, 2019

ahnitz commented Mar 4, 2019

ahnitz commented Mar 4, 2019

titodalcanton commented Mar 5, 2019

titodalcanton commented Mar 2, 2019 •

edited

Loading