Tasks

There are four challenge tracks:

Overall
Important

To submit to our competition, please use the Submit link. We ask that each team only sign up with one account, as this team name will be used for tracking. We also ask that there be only one team per organization (lab or corporation). If you believe there is a need to have more than two, please reach out to the organizers, either directly or through the forums.

Each task of the competition will take place in two phases: validation and test. For each phase datasets will be provided and people must submit a decoder, a set of encoded images and a selected docker container in the Submit link on the navigation.

The submissions run in a selected docker container by either running your executable or by first unzipping your zip file and running ./decode. At this point, your decoder should be able to decode the images or videos, including unzipping them, into a local subfolder (we recommended images/ or videos/ for easier debugging).

It is important that you submit a good working decoder in this phase or you will not be eligible in the test phase.

Also, you must have a paper ID and a paper submitted to DCC before the test phase begins.

Once the test phase begins, you must upload an exact decoder binary or zip file as you did in the validation phase. If they are not hashed to the same value, your decoder will be rejected and not ran.

We will also release the test data. You must encode this test data locally and be able to submit a previous decoder and file the represents the entirety of the compressed dataset.

To be prize eligible, you must submit to all the bit-rates of a track (all three image rate points and both of the video rate points). You are not limited to using the same decoder for all the rate points, but that decoder has to have been submitted in the validation phase.

Image compression
Task

The goal is to compress a corpus of images to three bitrates:

  • 0.3 bpp*
  • 0.15 bpp*
  • 0.075 bpp*

The winners will be chosen based on a human rating task. The raters compare the submitted images to each other, using the uncompressed image as a reference, and chose the preferred submission.

For guidance, several objective metrics will be shown on the leaderboard but not taken into account for prizes.

Validation set

The validation set is derived from the CLIC 2022 test set. We will have a similar amount of images and resolutions for the test set.

  • 30 images, collected from Unsplash.
  • Provided as png files.
  • Images resized such that the longer side is 2048.
  • Download here (130 MB)
Test set

The images are released using the Unsplash license. Please contact the organizers if you have objections to any of the released pictures.

Submission guidelines
  • The decoder size should not exceed 4GB.
Video compression
Task

The goal is to compress a corpus of videos to an average bitrate of

  • 0.05 mbps*
  • 0.5 mbps*

The winners will be chosen based on a human rating task. The raters compare the compressed videos to the uncompressed videos and chose the preferred codec.

For guidance, several objective metrics will be shown on the leaderboard but not considered for prizes.

Validation set

We provide a validation set that reflects the (hidden) test data. Validation set statistics:

Test set

Please contact the organiziers if you have objections to any of the released videos.

Submission guidelines
  • The decoder size should not exceed 4GB.
  • IMPORTANT: Submissions only need to compress 30 videos of the validation set, see above.
  • We provide the videos as mp4, but we will expect submissions to output three PNGs per frame: one for Y, U, and V channels. Please use the devkit for converting to PNGs, if you want to train on PNGs.
    • Or look at the Example Submission below.
Example submission
The devkit contains an example submission based on H264. It showcases the format expected by the server.
Image Perceptual Model

In the image perceptual model track, you need to design a model to evaluate the submissions of the image compression task, for instance by proposing a metric. Given a pair of images (one being the original and the other being a distorted image), your metric can generate a score. Methods A and B can be compared by generating two scores d(O, A) and d(O, B), where O is the original image. If d(O, A) < d(O, B), the metric prefers method A over method B.

In our image compression evaluation, human raters are presented with three images (O, A, B) and asked to pick one of A or B. The winner of the perceptual model challenge is chosen based on how often each submission matches these binary decisions of the raters.

Development kit

The development kit is provided as an example of what is expected from participants. It is in no way intended to contain data representative of the final challenge simply because that is not possible. The final test set will be created from the files uploaded by the participants in the compression challenge, and as a result it is impossible for us to provide data which will match that distribution in the validation set.

To get the devkit, use:

git clone https://github.com/spacl-ua/CLIC-devkit.git

Validation set
  • Note: The validation dataset is the same as last year's.
  • Download here (2.1 GB)
Test set

The full details of the tasks are contained in the README file of the repository.

The 2021 test data may be used to train new models for this year's competition. Please note that the link above contains pointers to the raw data (unfiltered) which we collected from raters. Therefore, it is possible that certain pairs of images might have been rated both ways (i.e., one rater might have disagreed with the other). It is up to the participants to do any data cleanup that might be necessary.

Video Perceptual Model *NEW*

In our new video perceptual model track you will need to design a model to predict MOS scores on the video compression task. Given a video, your method will need to generate a predicted mean opinion score (MOS) in the interval [1,5], as is generated by Microsoft's P.910 implementation on CLIC videos. The evaluation of your model will be the MOS over the entire dataset. Similar to the Image Perceptual Model track, teams will upload a CSV containing with each line containing two fields <video_name>,<mos score> where the video name will have the full subpath, for example libx264_10/0d49152a92ce3b843968bf2e131ea5bc5e409ab056196e8c373f9bd2d31b303d.mp4,5.0.

The metric used to rank the winners is Kendall’s tau-b. The confidence interval of the P.910 results will be used to account for potential ties. Pearson’s Correlation Coefficient (PCC) and Spearman’s Rank Correlation Coefficient (SRCC) will also be computed for reference.

Training data
Test set
  • Download test set here (43.8 GB). Please note that the video compression task has changed bitrates this year, so developing a robust perceptual metric is important.

* bpp: bits per pixel. One pixel consists of three color components.
* mbps: megabits per second. 0.05 mbps = 50 000 bits per second.

Sponsored by