Img bkg tests #104
Img bkg tests #104
Conversation
…iness_instance
|
I managed to make the two tests included work. See here: https://portal.nersc.gov/project/lsst/descqa/v2/?run=2018-04-11_1&catalog=focal_plane_0_test and here: https://portal.nersc.gov/project/lsst/descqa/v2/?run=2018-04-11_14&test=quick_bkg_pk Right now, the configurations just pull images from one directory (same as in the case of the instance catalog). Ideally, we would like that the program goes over all the visits available and repeats these tests. A different approach may be just create 10-15 config files with different visits, randomly selected. I think this might be sufficient. |
|
Hi Javier,
The one thing that is very confusing in your set-up is that you call
this quick-bkg.
Quick background is a setting that is used in PhoSim and has a specific
meaning and we are not using it currently. I think this could really
confuse people. Is there a better word/description for this?
Thanks!
…On 4/11/18 4:01 AM, fjaviersanchez wrote:
I managed to make the two tests included work. See here:
https://portal.nersc.gov/project/lsst/descqa/v2/?run=2018-04-11_1&catalog=focal_plane_0_test
and here:
https://portal.nersc.gov/project/lsst/descqa/v2/?run=2018-04-11_14&test=quick_bkg_pk
Right now, the configurations just pull images from one directory
(same as in the case of the instance catalog). Ideally, we would like
that the program goes over all the visits available and repeats these
tests. A different approach may be just create 10-15 config files with
different visits, randomly selected. I think this might be sufficient.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#104 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AMQ9jGkLx85FcLeYvuewvV4ZWXFwPv0Eks5tncZ6gaJpZM4TPKVr>.
|
|
Should I worry that the amplitude in Test 33 and R34 is not the same?
Sorry, I think I am not fully sure what I am looking at ... Thanks again!
…On 4/11/18 4:01 AM, fjaviersanchez wrote:
I managed to make the two tests included work. See here:
https://portal.nersc.gov/project/lsst/descqa/v2/?run=2018-04-11_1&catalog=focal_plane_0_test
and here:
https://portal.nersc.gov/project/lsst/descqa/v2/?run=2018-04-11_14&test=quick_bkg_pk
Right now, the configurations just pull images from one directory
(same as in the case of the instance catalog). Ideally, we would like
that the program goes over all the visits available and repeats these
tests. A different approach may be just create 10-15 config files with
different visits, randomly selected. I think this might be sufficient.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#104 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AMQ9jGkLx85FcLeYvuewvV4ZWXFwPv0Eks5tncZ6gaJpZM4TPKVr>.
|
|
@katrinheitmann Thanks for the comments. I think that you shouldn't be worried because I think that the raft 34 is on the edge of the focal plane so you expect some vignetting and that's the power that we are seeing. The raft that should be compatible with test 30 is R22. This can be changed in the configuration file. Rafts 12, 13, 21, 23, 31, 32, and 33 should also be fine/close. I chose the worst one... Please correct me if I am wrong @SimonKrughoff @jchiang87 |
|
Raft 11 is also one of the central 9 rafts. |
|
@katrinheitmann I am suspecting of a bug in the power-spectrum code because I checked with a central raft and the power is still really high... https://portal.nersc.gov/project/lsst/descqa/v2/?run=2018-04-12_1&test=pk_img |
|
Fixed: https://portal.nersc.gov/project/lsst/descqa/v2/?run=2018-04-12_10&test=pk_img (the blue line is perfectly below the orange because I am using the file from test 9 to check that the code is working). And this is checking a different exposure: https://portal.nersc.gov/project/lsst/descqa/v2/?run=2018-04-12_12&test=pk_img |
|
@fjaviersanchez I realized there are too many different things in this PR... can you do the following?
Thanks! |
|
Thanks @yymao. I implemented the changes that you suggested. By the way, I would suggest not to run this test on the full focal plane without rebinning (it takes a very long time to run). |
|
@fjaviersanchez I've proposed some more changes by submitting a PR to your PR: fjaviersanchez#1 |
|
@fjaviersanchez many thanks for bearing with me. I can merge this. The remaining question is whether we need to add some sort of validation data for this test? |
|
Thank you @yymao. I would say so. Let me please check for a good sample or model. |
|
@yymao I think that this is PhoSim's cleanest run to get a good estimation of the power spectra: |
|
@fjaviersanchez that could work. What's the precision that you'd like the PSD to be validated? We could also fit a functional form and use the fitted PSD to validate? |
There was a problem hiding this comment.
@fjaviersanchez thanks for the commits. I made one comment about the score calculation. Also, I just ran this PR and think the validation data do not look right. Can you take a look? You can find the DESCQA run here
descqa/ImgPkTest.py
Outdated
| if count: | ||
| total_chi2 /= count | ||
| ndof = len(self.validation_data['k']) - 1 | ||
| score = 1 - chi2.cdf(total_chi2, ndof) |
There was a problem hiding this comment.
Probably no 1 - here, since the passing criterion is passed=(score < 0.95)?
score = chi2.cdf(total_chi2, ndof)
|
@yymao the new validation data should be more representative of most of the images that we'll analyze since I am including sources (the previous one had no sources in the image, that's why it had very low power). I also corrected the bug with the score. Thanks! |
|
Thanks @fjaviersanchez! This looks great. Here's latest DESCQA run. I'm merging this now. |
This pull request tries to incorporate tests on images. Documentation about the tests is still missing and some changes in the validation data are still needed but, hopefully, I'll have this working by tomorrow.