Rules of Grand Prize Team

1. Creation of Grand Prize Team

Team Gravity and Team Dinosaur Planet have decided to create the Grand Prize Team.

The home page of Grand Prize Team is: http://www.grandprizeteam.com/

The leader of Grand Prize Team is: Gábor Takács from team Gravity.

2. The goal of Grand Prize Team

The goal of Grand Prize Team (GPT) is to win the Netflix Grand Prize, by combining the predictions of many teams. Grand Prize Team is open to anybody for joining. The earlier you join, the larger your contribution to the GPT will likely be, and thus your share from the Grand Prize will be bigger. You can find the rules for calculating the share of Grand Prize Team members in this section.
Note, that a 0.0001 improvement in Grand Prize Team's Quiz RMSE is approximately equivalent to a 6666 USD share of the 1 Million USD Grand Prize. (The reason for the word "approximately": if Grand Prize Team wins the 1 Million USD with more than a 10% improvement over Cinematch, then the contribution of a 0.0001 improvement is lower).

3. Definitions, terminology

• founders of Grand Prize Team: team Gravity and team Dinosaur Planet
• Grand Prize Team members: the teams who are in the Netflix Prize contest and who have chosen to add their participating team members to the Grand Prize Team.
• Grand Prize Team members' leaders: the leaders of the teams that are Grand Prize Team members.
• Grand Prize Team participants: the people comprising the teams that are Grand Prize Team members.
• Grand Prize Team applicant: a team that wishes to become a member of the Grand Prize Team, but has not been chosen as member yet.

4. How to join Grand Prize Team?

1. The leader of your team has to register himself/herself on our home page, and accept the rules, and give the name of his/her team in the Netflix Prize contest.
2. The leader also declares, that he/she is the leader of the team with the same name in the Netflix Prize competition (his/her Netflix Prize team), and that all team members of his/her Netflix Prize team also accepts these rules, and intend to be member of Grand Prize Team.
3. Once you have accepted the rules, you can upload probe and qualify files. If you just want to know how much improvement you would get on Grand Prize Team with your predictions, you can upload only the probe prediction files, wait and see how much the expected improvement is, and then upload your qualifying files if you so choose.
4. We also provide an option for everybody to upload only probe predictions and see how much is the improvement -- without registration nor any obligation.
5. Once you have accepted the rules and uploaded the probe and qualifying files, you become an applicant. There may be more teams that are waiting to see how much they can improve on Quiz. The organizers of Grand Prize Team choose the team with the most promising predictions (see this section for further details). If it is your team and the Quiz improvement of your predictions is sufficient (again, see this section), you are chosen to be the member of Grand Prize Team. Then you are not an applicant anymore, but rather you are a Grand Prize Team member!
6. Once you are the member of Grand Prize Team, you guarantee that you will collaborate with other Grand Prize Team members maximally to win the Grand Prize. That is, if Netflix announces Grand Prize Team as the winner of the Grand Prize, each member of Grand Prize Team must follow the rules of Netflix Prize to claim the Grand Prize: you sign the documents and contracts sent by Netflix, as do others, send the appropriate source codes that generated your predictions to Netflix (it must reproduce exactly your probe and qualify predictions), and send your methods' description to Netflix.
   See the Netflix Prize rules about ``reproducing'': "The judges sole activity during verification will be to ensure that the provided algorithm description and source code could reasonably have generated the prediction sets submitted."
   Team leaders have to ensure that in case of any problem, they have a deputy that can do the required tasks.
   The membership in Grand Prize Team is not exclusive, that is, team members are allowed to be members of other Netflix Prize teams.
7. Once we have chosen an applicant to be the member of Grand Prize Team, we maximally represent it, but do not register its participants into the Netflix Prize contest (however, there is one exception, see this section).
   Reason: the addition of each team member causes that Grand Prize Team disappears from the leaderboard until the added participants accept the rules. If somebody fails to accept the rules within a short time, then this is a clear disadvantage for the Grand Prize Team. If somebody completely fails to accept the Netflix Prize rules then Grand Prize Team cannot compete in the Netflix Prize competition. This rule ensures the presence of Grand Prize Team on the leaderboard.
   In case of winning the Grand Prize, we disclose the name of our team members towards Netflix, and at the option of each Grand Prize Team member, to the public.
8. Upon explicit request from the leader of an applicant team or Grand Prize Team member, there is an exception on the registration rule by registering the team's leader (the person, who is the leader of the corresponding Netflix Prize team), if all of the following conditions are met:
   • the team's prediction improves/improved Quiz RMSE by at least 0.0001
   • in case of an applicant team: the submission improves the Grand Prize Team results by at least a certain amount (see later what is meant by a certain amount), or in other words: the team's prediction gets accepted.
   • team follows the following protocol: negotiate a 1 hour interval with Grand Prize Team leader in e-mail, when both the applicant team's leader and Grand Prize Team leader will be online. When that time arrives, Grand Prize Team leader registers the team leader on http://www.netflixprize.com/team-edit page. When the e-mail sent by Netflix arrives, team leader shall accept the rules of the Netflix Prize contest as soon as possible. If applicant fails to accept the rules, at Grand Prize Team leader's sole discretion Grand Prize Team leader can accept the rules of Netflix Prize contest on behalf of the team leader, since all Netflix Prize contestants have already accepted the rules of the Netflix Prize contest. However, in this case, on Grand Prize Team leader's sole discretion, applicant will be considered as a quitting team. This implies, that when Grand Prize Team's improvement over Cinematch is above or equal to 10%, the team will be penalized according to this paragraph; team's predictions will be removed from Grand Prize Team's predictions; and team's points will be set to zero, and also any other issue holds that is mentioned in this section.

5. How to submit probe and qualifying files

We have discussed this issue briefly already in this section. Once your team registered on our home page and accepted the rules, then you can do the following:

• Upload your probe predictions.
• See how much improvement you can give to Grand Prize Team.
• Decide whether you send qualifying predictions.

Once Grand Prize Team is available to send a new submission, we choose the most promising set of predictions (see later what is meant on most promising).

• If it improves the Grand Prize Team results by at least a certain amount (see later what is meant by a certain amount), and the set of predictions comes from a Grand Prize Team applicant, the applicant becomes a Grand Prize Team member.
• If it improves below the certain amount, the applicant remains an applicant.
• The certain amount:
  In the beginning, we expect from each new prediction to improve Probe RMSE by at least 0.0001. If the improvement is below 0.0001, the predictions are considered individually. We use forward feature selection to select the best predictions, therefore, feel free to upload many (let's say 100) predictions.
  We expect that the improvement on Quiz RMSE (Quiz improvement) is at least 90% of the improvement on Probe RMSE (Probe improvement). If lower, it is considered individually, whether the submission will be included into Grand Prize Team's submission set.
• In Grand Prize Team submissions, we extend the set of predictions with only one Grand Prize Team member's/applicant's predictions. That makes clear which team improved the submission.
• The most promising set of predictions:
  In the beginning, promise is equal to the Probe improvement.
  However, members whose new predictions regularly give lower Quiz improvement than Probe improvement, i.e. the predictions are overlearned, will be considered individually. This means, that we can decrease their promise (being not equal to Probe improvement). Grand Prize Team members/applicants must accept the decision of Grand Prize Team leader.
  However, if many applicants regularly give lower Quiz improvement than Probe improvement, we will change the rules (see also this section).
  We calculate the promise of a set of predictions in two steps: first we calculate how the predictors altogether improve on the Probe RMSE. This promise is then modified with a score associated to the team (which score is related to the former overlearnings of the team) providing the set of predictions. Before we submit, we select the most promising set of predictions, based on the previous definition of promise. We apply forward feature selection on the most promising set of predictions, which chooses a subset of it. This forward feature selection procedure may change the promise. This new promise is the one that will be compared to the Quiz RMSE improvement.
• After submitting the new set of predictions, the Quiz RMSE on the leaderboard tells us, how much improvement is achieved due to the addition of the new set of predictions. According to the above rules, there are two possibilities:
  • the new set of predictions will be included into Grand Prize Team's submission set:
    • If the new set of predictions is provided by an applicant: applicant becomes a Grand Prize Team member, which means that it is included in the Grand Prize Team (but not on the Netflix Prize page, see this rule). The initial number of points of this Grand Prize Team member is proportional to the improvement in Quiz RMSE, see this rule.
    • If the new set of predictions is provided by a Grand Prize Team member, the member increases its share in Grand Prize Team.
  • the new set of predictions will not be included into Grand Prize Team's submission set: The submission may have decreased Grand Prize Team's Quiz RMSE score, thus, the leaderboard does not reflect the actual score of Grand Prize Team. The actual score will be displayed on the homepage of Grand Prize Team.

5.1 Format of probe and qualifying files

6. How to quit Grand Prize Team

There are two cases:

• When Grand Prize Team's improvement over Cinematch is below 10%:
  Grand Prize Team members may decide to quit Grand Prize Team, without any obligation.
• When Grand Prize Team's improvement over Cinematch is greater than, or equal to 10%:
  Grand Prize Team members may decide to quit Grand Prize Team. However, in that case, the quitting team will be penalized by its share, i.e. it has to pay that money to Grand Prize Team (See this section about shares.). That money will be distributed among the team members (including the quitting team), according to their shares (again, see this section). Grand Prize Team leader removes the predictions of the quitting team. If the actual blending method of Grand Prize Team was provided by the quitting team, Grand Prize Team stops using that method, and start using an available blending method (a method the providing team of which is still a member of Grand Prize Team).

7. Using the home page

• Registration is done by accepting the rules.
• You will be informed about the results on our home page.
• You will see which Grand Prize Team members/applicants give the most promising result.
• Before submitting, we will let the Grand Prize Team members know, which new set of predictions will be submitted. We also post the md5 checksum of the included probe and qualifying prediction files and the md5 checksum of the submission file.
• We inform team leaders about the results in email as well.

8. What happens when Grand Prize Team reaches 10%?

We keep collecting better and better predictions, until Netflix chooses Grand Prize Team as being eligible for winning the Grand Prize or the contest ends.

9. What happens when Netflix chooses Grand Prize Team as being eligible for winning the Grand Prize?

In this case, Grand Prize Team and its members must cooperate with Netflix: signing documents, sending source code, sending description, etc. We expect Grand Prize Team members to send their source code and description directly to Netflix. Recall that an eligible algorithm must be originally developed or implemented (e.g., must not violate or infringe on any applicable law or regulation or third-party rights); must be written in English; and must not require any third-party software or licenses, payment on the part of Netflix, or otherwise prevent Netflix from exercising its rights for the algorithm. If a member provided a blending algorithm that has been used in one submission with success (i.e., Grand Prize Team leader decided to use that blending method, see this section) --regardless of being used in the winning submission or not--, then you should also send its source code and description to Netflix and to Grand Prize Team leader. This is important for the reproducibility of each submission.
Everybody must fully cooperate in order to claim the Grand Prize.

10. What happens when Grand Prize Team wins the Grand Prize?

Grand Prize Team leader calculates the shares of Grand Prize Team members, and asks Netflix to transfer the calculated amount of money according to the instructions of the Grand Prize Team members' leaders. Grand Prize Team leader will give each member a report of exactly how much money was going to be distributed to each member.
Recall the following Netflix Prize rule: Awarded Prizes will be distributed among the Participants and affiliated organizations according to the instructions of the team leader. Grand Prize Team publishes the probe and qualify files with their md5 checksum, which took part (i.e. input for blending, or the blended result) in the winning submission or any previous submission of Grand Prize Team. Grand Prize Team also publishes the blending methods with their source code to enable everybody to reproduce the submissions, thus, to audit the calculation of shares. With these published materials, Grand Prize Team members can check whether everything went fine during the contest.

11. What are the shares of Grand Prize Team members

In case of winning the Grand Prize, the shares are the following:

• Gravity and Dinosaur Planet: 33.33%
• Other Grand Prize Team members: 66.67%

12. Using a team's own blending algorithm

Grand Prize Team is open to receiving new blending methods. These will be considered individually. We expect source code.
Submissions: once a promising blending method is provided, we can decide that instead of submitting an extended set of predictions, we vary the blending method.
Consequently, a Grand Prize Team applicant may turn its status into Grand Prize Team member not only by sending a good set of predictions, but also by sending a good blending algorithm.
There are basically two kinds of how a blending method may be used:

1. case A: We replace the current blending method with the new one.
2. case B: We blend the set of current predictions of Grand Prize Team (both Probe and Qualifying predictions) with the new blending method, and extend the set of current predictions with the blended result as a new prediction. In this case, we do not change the blending method.
   Example: suppose that Grand Prize Team is using ridge regression, and currently has 100 predictions, and someone submits a slow but powerful Neural Network blender. We may choose to blend the 100 predictions with the Neural Network blender, getting a new prediction (both Probe and Qualifying). Now Grand Prize Team has 101 predictions to submit. We compare the Quiz RMSE of ridge regression of 100 columns against the Quiz RMSE of ridge regression of 101 columns.

13. Responsibility

We do not disclose the probe and qualify prediction files until winning the Grand Prize, except to Netflix. After winning the Grand Prize, we publish the probe and qualify files with their md5 checksum. See this section. We also publish the blending methods (at least the source code) used in any Grand Prize Team submission, to ensure the repeatability of the results.
Recall the following Netflix Prize rule: At the end of the Contest, ... qualifying sets will become available upon request at the Machine Learning Archive
Gravity and Dinosaur Planet --the founders of Grand Prize Team-- declares, that they will never use the prediction files uploaded by teams to decrease their RMSE on leaderboard.

14. Removing predictions

In order to prevent heavy overlearning, we may remove some predictions from the set of predictions. This is executed when the removal of the columns causes at most a 0.00002 increase(!) in the Quiz RMSE per column. For example, we can remove 5 or more than 5 columns, when the removal increases the Quiz RMSE by at most 0.0001. Another example: we can remove 10 or more than 10 columns, when the removal increases the Quiz RMSE by at most 0.0002. We cannot remove 4 columns when Quiz RMSE increases by 0.0001.
If we remove a set of columns, and then later we decide to use it again (because it improves the results), we give 0 points for the second inclusion to the team providing those columns, because those points have been added to the team's points for the first inclusion.

15. Grand Prize Team as a member of another team

Grand Prize Team can create a unified team with another Netflix Prize team, or can become a part of another Netflix Prize team, if Grand Prize Team leader and the majority of the Grand Prize Team members agree on this, where the weight of the votes of the Grand Prize Team members are equal to the points of them (see this section on the number of points).

If such a team wins the competiton, the shares of Grand Prize Team members measured in percentage are relative to the share of Grand Prize Team in that team, and not relative to the Grand Prize.

16. Changing the rules

We reserve the rights to change the rules, if all members agree with the change. In this case, we will require applicants to accept the new rules.
When rules are changed, Grand Prize Team members will be informed via e-mail. They should accept/reject the new version in two work-days (work days in US), in e-mail. Not answering the email implies the acceptance of rules.
When rules are changed, but we are awaiting the answer of Grand Prize Team members, we require newly registered users to accept both the current and the to-be-accepted version of the rules. This is to ease the change from the old version of rules to the new version: since in this case we do not have to ask new applicants. If someone cannot accept both versions of the rules, it can wait until the new version will be accepted/rejected, and then register.

17. General Contest Terms and Conditions

The same rules holds as in the Netflix Prize contest's rules, section "General Contest Terms and Conditions", paragraph "A. Delivery" on Dec 10, 2008.
Moreover, The same rules holds for Grand Prize Team founder as in the Netflix Prize contest's rules, section "General Contest Terms and Conditions", paragraph "G. No Warranty" on Dec 10, 2008, for Netflix.
If you use this site, you are responsible for maintaining the confidentiality of your account information and your password. You agree to accept responsibility for all activities that occur under your account or password. You represent that you are of sufficient legal age to use this site and to create binding legal obligations for any liability you may incur as a result of the use of this site. You understand that you are financially responsible for all uses of this site by you and those using your login information.
We do our best to attain our goals, but we expressly disclaim any and all reliability for damages or loss of data resulting from any of our acts on behalf of the Grand Prize Team, either directly or indirectly, intentional or as an unintentional consequence of act. We are not liable to any damages caused by viruses or spywares. Though, we back up the received predictions, blending softwares and descriptions, we assume that the submitting team does also so, and can provide copies of submitted predictions, blending softwares and descriptions in case of data loss at the Grand Prize Team.

• v1: rules_v1.html
  Initial version.
• v2: rules_v2.html
  Important changes: we can add Grand Prize Team member's leader to the team-edit page (see this); when rules changes, there is a 2-work-day limit for accepting the new version (see this).
• v3: rules_v3.html
  We updated this section which now reflects the initial submission of Grand Prize Team (which contains the predictors of team Gravity and team Dinosaur Planet).
  We updated this section: Grand Prize Team leader can add blending methods by blending all or a subset of the existing columns, and extending the set of Grand Prize Team predictors with the new, blended result as a predictor.
• v4: rules_v4.html
  We do not ask Netflix to remove rejected sumbissions anymore since this is unworkable (this section and this section were changed).
• v5: rules_v5.html
  Grand Prize Team can be a member of another Netflix Prize team (if the majority and the leader agrees), see this section.
• current version: rules.html