Competition

MIDRC mRALE Mastermind Challenge

Organized by challenge-organizer - Current server time: June 8, 2023, 5:52 a.m. UTC

Current

Training
April 26, 2023, 3 p.m. UTC

Next

Validation
June 10, 2023, 7 p.m. UTC

End

Competition Ends
July 10, 2023, 9 p.m. UTC

MIDRC horizontal logo

 

MIDRC mRALE Mastermind Challenge:

Developing AI to predict COVID Severity on Chest Radiographs

Brought to you by the MIDRC Grand Challenges Working Group and MedICI 

Welcome to the starting line!

 

NEW! Informational webinar on Tuesday June 6 at 10am Eastern Time.

Are you registered but unsure how to get started? Or not yet registered but interested in participating? Have challenge specific questions? This webinar is for you! Please join us for this 30-minute webinar with the opportunity to ask questions. Registration is not required, just drop in.

 

The MIDRC mRALE Mastermind Challenge is organized in the spirit of cooperative scientific progress that benefits the common good and health outcomes for all. Your contributions could help advance the diagnosis, treatment, and prognosis of COVID-19.

The goal of this Challenge is to train an artificial intelligence/machine learning (AI/ML) model in the task of predicting COVID severity in terms of mRALE score [Li et al] from frontal-view portable chest radiographs (CXRs) obtained within 2 days of a positive COVID-19 test. This Challenge provides annotated training data, i.e., chest radiographs and their mRALE scores as assigned by a team of experts. The data for model validation and testing will not be made available for download, however. Instead, you will submit containerized code of your trained model for inference on this Challenge platform against the validation and test datasets. This Challenge uses Docker as a containerization solution.

Prizes

First and second place teams: MIDRC support to take your method/model through the FDA regulatory process. 

Cash prizes through 7th place!

1st Place - $ 15,000
2nd Place - $ 8,000
3rd Place - $ 7,000
4th - 7th Place - $ 5,000 each

See the "Terms and Conditions" for constraints on receiving MIDRC support through regulatory approval and on receiving cash prizes. Also note that many MIDRC investigators and their labs are excluded from competing per our conflict of interest policy. 

All participants with valid submissions in the Challenge test phase will receive contributor credit on Challenge manuscripts and acknowledgment at AAPM and RSNA 2023 annual meetings as well as on midrc.org.

We are excited to be conducting this Challenge and hope you will participate! We encourage you to invite your colleagues and friends to create a team. Please make sure to familiarize yourself with the Challenge by navigating through the links (on the left) and tabs (at the top) and read the “Terms and Conditions” before signing up. You will need to agree to these "Terms and Conditions" as part of the sign-up process.

Remember to check out the Tutorials and the mRALE Mastermind GitHub repo to start building your model. Please take advantage of the opportunity to make practice submissions during the training phase!

Important Dates

Challenge period: 5/15– 7/10/2023:
  • 5/1 Registration is now open!
  • 5/15 Training phase begins  
  • 6/10 Validation phase opens; registration closes   
  • 7/1 Test phase opens
  • 7/10 Test phase closes; challenge complete
  • 7/12 Winners announced/contacted 
  • 7/23 – 7/27 Meeting: AAPM 2023 Annual Meeting, Houston TX; recognition/winners public announcement
Note that the Challenge Platform times at the top of this page are in the UTC timezone (EST+5)

Li et al reference: Matthew D. Li, Nishanth T. Arun, Mishka Gidwani, Ken Chang, Francis Deng, Brent P. Little, Dexter P. Mendoza, Min Lang, Susanna I. Lee, Aileen O’Shea, Anushri Parakh, Praveer Singh, and Jayashree Kalpathy-Cramer, Radiology: Artificial Intelligence 2020 2:4, https://doi.org/10.1148/ryai.2020200079

 

 

 

 

MIDRC mRALE Mastermind Challenge;

Challenge Logistics 

The Challenge has 3 phases: (1) a training phase (which includes practice submissions so that you can familiarize yourself with the platform and code containerization), (2) a validation phase, and (3) a test phase. 

Set Team Name

Please click your profile at the top right corner where your screen name is to set the "Team name" in your individual profile settings.

Training Phase 

During the Challenge training phase, you are encouraged to use the provided annotated training cohort (see the "Get Data" link under the "Participate" tab) for model development and training on your local hardware. At this time, you also have the opportunity to perform practice submissions. For this purpose, a small set of ‘practice’ chest radiographs in DICOM format are available for download directly from the Challenge platform. Submissions to the Challenge platform during the training phase will perform inference on this very limited number of ‘practice’ DICOM images. This is intended to allow you to troubleshoot general issues with Docker submissions, reading DICOM images, and to verify that your algorithm's output by case is the same locally as on the Challenge platform. This small dataset is not intended for training. Please use this opportunity to "test drive" the submission process and resolve any issues you encounter to minimize potential problems during later phases. Only limited technical assistance will be available after the training phase. There is a maximum of 20 practice submissions per team. 

Validation Phase 

During the Challenge validation phase, you will submit your trained model(s) to the Challenge platform to perform inference and evaluation on the unpublished validation set (which will not be available for download). The validation phase allows you to (1) further familiarize yourself with the containerization and submission process of your code and trained model(s) and (2) to fine-tune your model(s). A Leaderboard will be available during the validation phase to promote friendly competition.There is a maximum of 10 submissions per team in this phase. 

Test Phase 

During the Challenge test phase, you will submit your most promising model(s) to the Challenge platform for inference and evaluation on the unpublished test set (which will not be available for download). There is a maximum of 3 submissions per team in this phase.  

More Details and Tutorials 

Please go to "Challenge Details" (left) to learn more about this Challenge. The "Tutorials" link on the left provides valuable information on Docker and includes tutorial videos on Docker installation, using Docker, as well as on downloading data from data.midrc.org, and the submission process to this Challenge platform. Please also review the "Terms and Conditions" (left) for additional important information.  

Questions and Help 

Discussion of the Challenge on the Forum ("Forums" tab) is encouraged. The Forum should be used for any questions you may have about this Challenge so that Challenge organizers can help if needed, and other participants can benefit from questions and answers. 

Ready to participate? 

Once you are ready to participate, go to the "Participate" tab and log in or register for an account when prompted. You are required to agree to the Challenge "Terms and Conditions" to access the Challenge platform so please read them beforehand (see link on the left). 

MIDRC mRALE Mastermind Challenge;

Challenge Details 

Challenge Task

The task is to predict COVID-19 severity in terms of mRALE score from portable chest radiographs (CXRs) obtained within 2 days of a positive COVID test. The acronym mRALE stands for modified RALE score which, in turn, stands for Radiographic Assessment of Lung Edema. This grading scale was originally validated for use in pulmonary edema assessment in acute respiratory distress syndrome and incorporates the extent and density of alveolar opacities on chest radiographs. The grading system is relevant to COVID-19 patients as the chest radiograph findings tend to involve multifocal alveolar opacities, and many hospitalized COVID-19 patients develop acute respiratory distress syndrome. To obtain an mRALE score, each lung is assigned a score for the extent of involvement by consolidation or ground glass/hazy opacities (0 = "none"; 1 = "≤ 25%"; 2 = "25%–50%"; 3 = "51%–75%"; 4 = ">75%" involvement). Each lung score is then multiplied by an overall density score (1 = "hazy", 2 = "moderate", 3 = "dense"). The sum of scores from each lung is the mRALE score. Thus, a normal chest radiograph receives a score of 0, while a chest radiograph with complete consolidation of both lungs receives the maximum score of 24.  

Reference: Matthew D. Li, Nishanth T. Arun, Mishka Gidwani, Ken Chang, Francis Deng, Brent P. Little, Dexter P. Mendoza, Min Lang, Susanna I. Lee, Aileen O’Shea, Anushri Parakh, Praveer Singh, and Jayashree Kalpathy-Cramer, Radiology: Artificial Intelligence 2020 2:4, https://doi.org/10.1148/ryai.2020200079

 

Chest Radiograph Expert Annotations

The CXR exams have been annotated in terms of left lung/right lung extent of involvement and density from which the mRALE score has been calculated. Note that for frontal CXR, the left lung is displayed on the right and vice versa. For completeness, and potential use in model training, the individual left/right lung assessments are included in the annotation file (see "Get Data") with self-explanatory column headers and the encoding for involvement: 0 = "none"; 1 = "≤ 25%"; 2 = "25%–50%"; 3 = "51%–75%"; 4 = ">75%" involvement, and for overall density:1 = "hazy", 2 = "moderate", 3 = "dense". 

The estimated mRALE score is the only output your AI/ML model needs to provide. Other output scores are not allowed. 

Model Training Material

You can find helpful material, including annotations and a file manifest for downloading training data from data.midrc.org, on our mRALE Mastermind Challenge GitHub repo.

Performance Metrics

The primary performance metric to rank submissions is quadratic-weighted kappa. Submissions will be ranked using the primary performance metric. A statistically significant difference in performance between the winner and runners-up is not required to "win" the Challenge. Only performance on the test set will be used to rank submissions. A secondary performance metric, prediction probability concordance (PK), will be used to break ties, if needed. 

Output of Your Model/Algorithm

The output of your model should be an estimated mRALE score (a single score per CXR image), which is an intereger score between 0 (normal) and 24 (the most severe).  

Formatting the Output of Your Model

The output of your method should be provided in a single comma-separated CSV file with image name in the first column and the corresponding output mRALE score in the second column.  

* Make sure the header and rows are in this specific format: 

fileNamePath,score [coming soon]

<dicom-name-1>.dcm,<integer mRALE score between 0 and 24> 

<dicom-name-2>.dcm,<integer mRALE score between 0 and 24> 

<dicom-name-3>.dcm,<integer mRALE score between 0 and 24> 

... 

etc. 

The Challenge Platform Specs

The system specifications are as follows: 

 
 Azure VM Name 
vCPU 
RAM (GB) 
 Temp Storage SSD (GB) 
 GPU 
GPU Memory (GB) 
 Max uncached disk throughput: 

IOPS/MBps 
 Max NICs 
Standard_NC6s_v3 
6 
112 
736 
1 
16 
20000/200 
4 

 

Note that internet connectivity is not provided within the Challenge platform. All necessary code, model weights, and library requirements need to be provided in your submission. GPU will only be available during the validation and test phases, not for the practice submissions during the training phase. 

Submissions to the Challenge Platform

You need to supply a zip archive that contains a Dockerfile, all necessary code, and a trained model to allow the Challenge platform to build a Docker Image to run and evaluate your model on the practice cases, validation, or test sets, depending on the Challenge phase, respectively. Example zip archives suitable for submission are provided in the "Starting Kit" (go to the "Participate" tab, then to "Files"). Each trained model needs to be submitted in its own zip archive. It is important to note that all model training and fine-tuning needs to be performed on your own hardware. The Challenge platform only performs inference using trained models submitted in the required format. 

There is no performance assessment for the practice submissions using the practice data.The performance of your model(s) will be reported back to you and shown on the Leaderboard in the validation phase. For the test phase, performance will be reported after conclusion of the Challenge.  

In the test phase, a description of your model and training data (plain text or Word file) needs to be included in your zip archive submission for your submission to be considered a valid submission, i.e., for its performance to be reported back to you and to be part of the Challenge.  

Local Computer Requirements

It is advisable to have Docker installed on your local computer so you can check locally how your code runs within a Docker Image. Go to https://docs.docker.com/ to learn more about how to install Docker on your own computer. The videos at the "Tutorials" link (left) provideadditional information.  

Docker Images will be built and run on the Challenge platform with Docker version 20.10.13 and above, so, if possible, a local install of Docker should be that version or higher. 

Sharing of Code and Trained Models

It is highly encouraged that you allow MIDRC to make your code and trained model(s) public on the MIDRC GitHub (see "Terms and Conditions").  

Summary Tables

 
 Challenge data 
Data available for download 
Data available on platform for inference 
Number of CXR exams 
Training  
Yes 
Yes
~2,000 
Practice submission 
Yes 
Yes 
12 
Validation 
No 
No
 ~200 
Test
No 
No
~800 

 

Challenge phase 
Leaderboard 
 GPU available 
 Maximum number of submissions 
Maximum size of zip archive submissions 
 Maximum submission run time on platform 
Training
N/A 
N/A 
999
15.0 GB
1000 seconds 
Validation 
Yes 
Yes 
10 
15.0 GB
2500 seconds
Test
No 
Yes 
3 
15.0 GB
5000 seconds

 

Summary of Important Points

  1. All model training and fine-tuning needs to be performed on your own hardware. The Challenge platform submission system should be used for inference only using a trained model during the Docker practice submission period and for the validation and test phases of the Challenge. 

  1. The validation and test data will not be made available to participants and both sets contain unpublished data (to be made publicly available in time after conclusion of this Challenge). 

  1. It is highly encouraged to practice submitting a Docker archive to the platform during the Challenge training phase for inference on the small set of ‘practice’ cases on the Challenge platform. Technical help will not be available to your team in later phases of the Challenge if your team did not participate in the practice submissions. 

  2. CXRs for the Challenge validation and test datasets 

a) are portable CXRs in the anteroposterior (AP) view
b) were obtained within 2 days of a positive COVID test 
c) are in DICOM format only. Any potential conversion from DICOM to a different image format must be performed within your submitted Docker container.
d) pertain to adults (no pediatric exams)
e) represent a single CXR per subject (patient) 

For more details see the "Get Data" link under the "Participate" tab. 

  1. Within the Challenge platform, Docker submissions will not be allowed to access the internet (e.g., downloading pre-trained ImageNet weights will not be possible). Everything needed to successfully run your model needs to be included in your submitted Docker container. 

  1. GPU will be available on the Challenge platform only during the validation and test phases. 

  1. Submissions that exit with an error, i.e., submissions that fail, do not count towards the maximum number of submissions allowed. 

MIDRC mRALE Mastermind Challenge; 

Tutorials 

Installing Docker:

Using Docker Tutorials:

Practice with the Demo: 

Please note that not all tutorial videos below have sound at this time.

  1. 'How to' Step 1: Sign Up and Login
  2. 'How to' Step 2: Choose a Team
  3. 'How to' Step 3: Training Data, Docker Image Building and Testing
  4. 'How to' Step 4: Uploading Submissions to the Challenge Platform
  5. 'How to' Step 5: Running your Submission on the Challenge Platform 

Missed our informational webinar from June 6th?

  • Watch the video, which includes a short introduction and a live demonstration on using Docker and the submission process. 

Downloading Data and Cohort Building at data.midrc.org 

MIDRC mRALE Mastermind Challenge;

Terms and Conditions 

By participating in this Challenge, each participant agrees to the following: 

Participants 

  • Participation in this Challenge acknowledges the educational and community-building nature of the Challenge and commits participants to conduct consistent with this spirit for the advancement of the medical imaging research community. 

  • For more background information on the expected conduct of participants, see this article  for a discussion of lessons learned from the LUNGx Challenge, which was sponsored by SPIE, AAPM, and NCI. 

  • Anonymous participation is not allowed. 

  • Participants from the same research group, company, or collaboration are required to participate as a team, i.e., form a team within the Challenge platform.  

  • Individual participants should form a single-user team.  

  • Team size is limited to 8 participants. 

  • Participants may only join one team. 

  • Entry by commercial entities is permitted but must be disclosed. 

  • No conflict of interest may exist for any team to be considered in the final ranking as per the MIDRC Grand Challenge Conflict Policy. 

  • All participants must attest that they are not directly affiliated with the labs of any of the Challenge organizers or major contributors. 

  • Registration after the deadline will not be considered for the Challenge (see ‘Important dates’ at the bottom of the “Overview” page). 

AI/ML Methods 

  • Participants are strongly encouraged to agree to MIDRC making their code and trained model(s), including weights, publicly available after completion of the Challenge.  In order for a participating team to win cash prizes, it is a requirement to allow MIDRC to make the team's code and trained model weights publicly available. 

  • As part of the registration process, participants will select one of two options: 
* Upon Challenge completion, our team agrees that our trained model(s) and Docker submission(s) WILL be made public by MIDRC. 
* Our team wishes to participate in the Challenge, but we do NOT wish for our submission(s) and trained model(s) to be made public by MIDRC.  

Note: If the top 2 performing teams decide they'd like MIDRC support for going through the FDA regulatory process of their model (see the section on performance evaluation), this will preclude any requirement of making code publicly available and any publication of results by the Challenge organizers will be done in such a way as not to harm any potential commercialization prospects. 

  • Descriptions of participants’ methods and results may become part of presentations, publications, and subsequent analyses derived from the Challenge (with proper attribution to the participants) at the discretion of the organizers. While methods and results may become part of Challenge reports and publications, participants may choose not to disclose their identity and remain anonymous for the purpose of these reports. 

  • Only fully automated methods are acceptable for the Challenge. It is not possible, and not allowed, to submit manual annotations or interactive methods. 

  • Using transfer learning/fine-tuning of models pretrained on general-purpose datasets (e.g., ImageNet) is allowed. 

Submissions to the Challenge 

  • Once participants make a submission within the test phase of the Challenge, they will be considered fully vested in the challenge, so that their performance results will become part of any presentations, publications, or subsequent analyses derived from the Challenge at the discretion of the organizers. Withdrawal at this point is not allowed but participants can choose to have their results reported anonymously in these presentations and publications. 

  • For submissions in the test phase, participants will be required to disclose a description of their methods and training data used. Without this, a submission will be considered invalid. In other words, a description of the method/model (plain text or Word file) needs to be included in your zip archive submission in order for a submission to be considered a valid submission, i.e., for its performance to be reported back and to be part of the Challenge.  

  • Each Challenge phase has a maximum number of submissions allowed per team (see the “Challenge Details” page). Submissions that result in errors flagged by the Challenge platform will be labeled “Failed” and do not count towards the maximum number of submissions allowed. After the maximum number of submissions for a team is reached, the Challenge system will not accept further submissions to the applicable Challenge phase. 

 Performance Evaluation, Ranking, Prizes, and Participant Credit

  • Performance on the test dataset, i.e., performance in the test phase of the Challenge, will be used to rank submissions and determine the Challenge placement of participating teams.

  • The primary performance metric will be used to rank submissions. 

  • Raking of submissions will be performed using the value for the primary performance metric, without taking into account statistical significance of any differences in performance among submissions. Thus, it is not required to demonstrate a statistically significantly better performance than other submissions to "win" the Challenge.  

  • A secondary performance metric will be used to break ties, if needed. 

  • The highest-performing submission of a participating team will determine the team's ranking within the Challenge.

  • The top 2 performing teams will have the opportunity to receive support from MIDRC through the FDA regulatory process of their method through the evaluation of their method using the MIDRC sequestered (non-public) Data Commons, provided that their best submission substantially, and statistically significantly, outperforms random guessing.

  • Cash prizes will be awarded to the top 7 teams as outlined on the "Overview" page, provided that these teams agree to MIDRC making their code and trained models public and that their best submission substantially, and statistically significantly, outperforms random guessing. The top 2 teams do not need to make their code/model public if they want to take this code/model through the regulatory process with MIDRC support. 

  • All participants with a valid submission in the test phase will receive contributor credit on Challenge publications, acknowledgement at AAPM and RSNA 2023 annual meetings as well as on midrc.org.

  • In the validation phase, the performance of your submission will automatically be put on the Leaderboard. There is no Leaderboard in the test phase. 

  • See the "Challenge Details" page under the "Learn the Details" tab for the performance metrics used in this Challenge.

Help

  • Technical help will be provided by Challenge organizers as much as possible if problems arise during the submission process. However, help to teams in the validation and test phases of the Challenge will be limited and may not always be possible. Teams that did not participate in the practice submissions during the training phase will receive the lowest priority and may not receive any help. 

  • It is imperative to read the “Challenge Logistics” and “Challenge Details” pages (links in the menu on the left) and we also suggest that you visit the “Tutorials” page (link in menu on the left) prior to attempting to make any submissions. 

  • The “Forums” tab should be used to communicate all questions or issues regarding the Challenge. 

Training

Start: April 26, 2023, 3 p.m.

Description: Training phase: create models and upload your docker images in preparation for the test phase.

Validation

Start: June 10, 2023, 7 p.m.

Description: Validation phase: create models and upload your docker images in preparation for the test phase. Also perform test runs on validation data.

Final

Start: July 1, 2023, 3 p.m.

Description: Test phase

Competition Ends

July 10, 2023, 9 p.m.

You must be logged in to participate in competitions.

Sign In