MSlesion Segmentation
Automatic Evaluation of Online Submissions
1. Introduction
The MS lesion 2008 segmentation challenges accepts a participant's submission by online. A participant downloads data set from NITRC, processes the data using the own algorithm, and uploads the segmentation result to http://www.ia.unc.edu/MSseg/submit.php after registering team name and password. Once the segmentation is uploaded, an automatic evaluation is performed and its score is posted to the result page (http://www.ia.unc.edu/MSseg/results_table.php).
2. Data Processing
The zip file uploaded by a participant is stored at '/var/www/vhosts/ia/MSseg/results/incoming/post_submission'. However, the evaluation cannot be performed at the web server because the server is not computationally powerful (small memory and slow cpu) to execute the evaluation program. Another workstation is required to run the evaluation program. In addition, the web server is not allowed to access to intra-workstations due to security reasons. Therefore, rather than pushing the download data to the computing workstation, the workstation should fetch the user data from the web server. The fetching should be MANUALLY triggered. At this moment, 'ares.ia.unc.edu' is working as a computation server because only the ares server has 'LaTeX' installation, which is required to produce a 'pdf' version of the evaluation report.
3. Evaluation Script
In order to initiate the evaluation from 'ares', a manager executes '/NDRC/MSlesion/results/processing/executeEvaluation.py'. This python script will download data from the web server, unzip into a subdirectory, execute '/NDRC/MSlesion/results/tool-bin/linux64/EvalLesionSegmentation_linux64', generate a PDF report, and upload the report back to the web server.
1) Downloading evaluation data
The list of team names to evaluate is provided from http://www.ia.unc.edu/MSseg/list_unevaluated_data.php list_unevaluated_data.php . The script fetches data from mySQL database. In the database, 'mslesionrankings' table keeps track of users' uploading. If 'evaltime' is null, or 'evaltime' is older than 'uploadtime', the php script will print those team names. After parsing the script's output, 'executeEvaluation.py' python script sends an authenticated http request to the server and download zip files into '/NDRC/MSlesion/results/processing/onlineEval'.
2) Segmentation Evaluation
'The downloaded zip files are evaluated using 'EvalLesionSegmentation_linux64' that produces '.csv' files into '/NDRC/MSlesion/results/processing/onlineEval/[TEAM_NAME]'.
3) Generating report
The report is generated in PDF format using 'pdflatex'. At this time, only 'ares' server is able to run 'pdflatex', which means the python script must run on 'ares' server.
4) Uploading report
'The generated report is uploaded back to the web server via the php script named, 'http://www.ia.unc.edu/MSseg/upload_evaluated_report.php'. The script accepts a post request and stores the uploaded pdf file into '/var/www/vhosts/ia/MSseg/reports' after updating 'mslesionrankings' table.
4. User management
A user can reset or change their password via 'http://www.ia.unc.edu/MSseg/register.html'. Once the password is reset, an announcement email is sent to the user using the sender account 'mslesion2008@gmail.com' whose password is '!mslesion2008!'. In order to send an email, PHPMailer class must be available, which is currently located at /var/www/vhosts/ia/MSseg/phpmailer. The change of password is done by the script '/var/www/vhosts/ia/MSseg/reset_password.php'.
5. Summary
In order to evaluate a new submission, a manager simply does followings:
- ssh ares
- cd /Human/NDRC/MSlesion/results/processing
- ./executeEvaluation.py
If there is no need for the execution of evaluation (because it takes longer time), use '-s' flag. This flag lets the script skip the evaluation and perform only the report generation.
ex) ./executeEvaluation.py -s
Addendum
- UNC_test1_Case02 was removed (issue with motion in the data, bad quality)
- not all teams submitted all datasets => skewed evaluation
