A branch or a new repo for sharing DC2 analysis notebooks?

[yymao]

Many people now have tried out the tutorial notebooks and have their notebooks and/or analysis code that are not yet at a stage of a standalone package/repo, but are potentially useful for the broader DC2 users so that we are not reinventing wheels.

The question is how to collect these notebooks. If the notebook is well documented and is of the "tutorial grade", we can just put them in DC2_Repo/Notebooks. For other notebooks, should we collect them in a new branch of this repo, or should we have a new DC2_analysis (or DC2_users or DC2_results) repo?

(cc @katrinheitmann @evevkovacs @wmwv @joezuntz @idellant )

[jchiang87]

I strongly prefer having a separate repo for notebooks (including the current tutorials) and results. We've been using the DC2_Repo for Runs 1.1p and 1.2p to track software versions and the work that needs to be done at each stage of the production. I think those items should be kept separate from the more user-oriented view of the DC2 project that the notebooks and results would represent. So I'd suggest a new DC2_analysis or DC2_results repo. And we might consider renaming the current DC2_Repo to something like DC2_production so that we don't have to say things like "It's in the DC2_Repo repo"

[yymao]

+1 for renaming DC2_Repo to DC2-production and have a new DC2-analysis repo

[salmanhabib]

Agree with @jchiang87.

[idellant]

I also think it's better to keep them separate.

[evevkovacs]

Yes, I agree.

[rmandelb]

I agree about having analysis (including the current tutorials) separate, so the repositories are DC2-production and DC2-analysis (or something like that). Just to make sure we are on the same page, the second of these will primarily include generic tutorials or code to do something that lots of people need to do, like matching between various catalogs and learning how to relate and plot quantities -- but once we start getting into project-specific activities, those will be encouraged to move into the repo associated with the relevant project, yes?

How do you classify notebooks and plots associated with validation? Is that considered part of production (=stay in this repo) or should it go with the more user-focused notebooks etc.? Many validation tests are in DESCQA (including a plan for some of the post-DM validation), but e.g. there's at least one recent issue on DC2 validation in this repo.

[yymao]

In my mind, I think DC2-analysis should contain data access/analysis tutorials, quick analysis code and notebooks from users that contain some quick analysis but don't belong to any analysis repos yet. The idea is to lower the barrier for people to share what they did with DC2, and have a place the people can use as a starting point when they want to do DC2 analysis. For projects that have its own repos, the code/notebooks should go to their respective repos (but maybe they'll want to mention those analysis repos in some readme file in DC2-analysis).

Sill in my mind, I think validation should go to DC2-production. Here by validation I mean (1) the discussion of potential issues, (2) what need to be validated, and (3) validation code that are not part of DESCQA. The potential workflow could be:

• A posts a quick analysis notebook in DC2-analysis. B notices the result is bit unexpected and opens an issue in DC2-production with a link to that notebook in DC2-analysis. After some discussion people found the root of the issue and decided to have this being a validation test, so C works with A to port part of the notebook as validation script in to DC2-production.

• D thinks we need to validate some feature X, and looks in DC2-analysis but finds no one has done something along that line. D opens an issue in DC2-production. After discussion people think this validation test should be done in DESCQA. D opens an issue in DESCQA to add a new test.

[yymao]

It sounds like we have reached a consensus here 🎉 I can go ahead to create DC2-analysis (or @rmandelb do you want to do that since you're the Analysis Coordinator?) and move the tutorials from here to DC2-analysis. (I propose that we also impose a consistent naming scheme for the tutorial notebooks now that we are moving them. Thoughts, @wmwv @EiffL?)

I don't have the permission to rename this repo to DC2-production. I think either @cwwalter or @drphilmarshall can do so?

[cwwalter]

I made it, so I can do it. Are there any implications to changing the name of a repo that everyone is using? Do we have to tell users anything?

In the analysis repo we might want to have more substructure than just a sub-directory for notebooks. We could split it up for tutorials etc.

[yymao]

According to https://help.github.com/articles/renaming-a-repository, after renaming, almost everything will continue to work as long as no one creates a new LSSTDESC/DC2_Repo repo.

[cwwalter]

I notice some repos with something like: DC2-production and others with DC2_production. I have vague memories I choose underscore for some reason I knew about a long time ago. Do we have a preferred pattern?

[yymao]

I don't think DESC has a convention. On GitHub, I think hyphen is more widely used than underscore (see for example, GitHub's repos: https://github.com/github).

[cwwalter]

OK thanks.

:)

[cwwalter]

DC2_Repo has been renamed to DC2-production.

[drphilmarshall]

On your local command line, you can make the switch with

git remote set-url origin https://github.com/LSSTDESC/DC2-production.git

or

git remote set-url origin git@github.com:LSSTDESC/DC2-production.git

but as Yao says, the old repo name will work for a while.

https://help.github.com/articles/changing-a-remote-s-url/

[yymao]

This can now be closed with the new DC2-analysis repo https://github.com/LSSTDESC/DC2-analysis that @rmandelb just created 🎉!