Evaluation, Reproducibility, Benchmarks Meeting 21

Minutes of meeting 21

Date: 27th June 2023

Present: Carole, Annika, Nicola

TOP 1: Metrics reloaded implementation

• The repository is in a good shape but the tutorials and documentation can be expanded (probably starting from August)
• Organize proper testing:
  • Carole will write a list of things to be tested
  • We will have two Google Docs:
    • One sheet with commenting rights: People put their names to the topics they would be interested to test
    • Second sheet in which those people get edit rights to describe potential issues
  • Annika promotes the testing at the MONAI MIDL Meetup (13 July 23)
  • We will acknowledge testers in the implementation paper and on github

TOP 2: Next steps

• Enable easy challenge data access via MONAI
  • Work on a standard benchmarking format
• Best practices/link initiatives
• Reproducibility in general
• Potential new members:
  • Ask someone from grand-challenge.org
  • Ask someone from Mauricio’s group
  • TODO for next meeting: Discuss potential open call for new members
  • May have alternating meetings with core group or extended group

MISC

• We will try to find another meeting slot for the July meeting
• We will skip the August meeting
• Carole will be on leave between October and February