EvalAI Submission Failed Status

Hi ARCLab team,

Thank you for hosting this competition!

I wanted to open a thread to discuss the errors I am getting when testing submissions to EvalAI.

Using your code structure provided in ml_baseline, I formated a main.py script using the default RandomForest classifier to generate pickle files under the relative path ./trained_model/***.pkl. I then utilize the provided submission.py to run in docker, which creates the submission file in the docker path /submission/submission.csv

I use the following commands to build the DockerFile and run the submission.py. I am running Python 3.10.12 on WSL Ubuntu 22.04.3

docker build -t splid-submission:latest -f Dockerfile .
docker run --rm --name mysubmission -v /home/asaadat/mit-arclab-ai-competition/dataset:/dataset -v /home/asaadat/mit-arclab-ai-competition/submission:/submission splid-submission

After submission to the competition phase using evalai push splid-submission:latest --phase mit-competition-2164, I get the following Stderr output from EvalAI. I added a logger to output successful executions in submission.py and printed the output file after writing it to /submission/submission.csv.

2023-12-20T19:57:29+0000 - INFO - <module> - Starting utils.tabularize_data
2023-12-20T19:57:29+0000 - INFO - <module> - Loading pickle models
2023-12-20T19:57:35+0000 - INFO - <module> - Running test predictions for EW
2023-12-20T19:57:35+0000 - INFO - <module> - Running test predictions for NS
2023-12-20T19:57:35+0000 - INFO - <module> - Running utils.convert_classifier_output
   ObjectID  TimeIndex Direction Node Type
0         1          0        EW   SS   HK
1         1          0        NS   SS   HK
2        20          0        EW   SS   CK
3        20          0        NS   SS   CK
2023-12-20T19:57:35+0000 - INFO - <module> - Saving results csv: /submission/submission.csv
   ObjectID  TimeIndex Direction Node Type
0         1          0        EW   SS   HK
1         1          0        NS   SS   HK
2        20          0        EW   SS   CK
3        20          0        NS   SS   CK

Running the main ML classifier script and submission.py locally and with docker run works without errors.

As for the dataset folder, I copied over and renamed the phase_1 folder to dataset and, within that, the train folder to test, as this seems to match the format described in the ml_baseline code. Is this correct?

Please advise how to fix the EvalAI Failed status, even though no error is seemingly present in the Stderr output. Is there still an issue present in EvalAI submissions?

I think it is an issue for everyone right now - see here

Hi,

There is an ongoing issue with the EvalAI submission platform and we are working closely with the EvalAI team to resolve this issue.

We’ll let you know when that issue is fixed.

Sorry for the incoveniences.

Best,
Peng

Hi,

Please see this thread for an update on this issue.