The new Community Priority Evaluation (CPE) guidelines prepared by the Economist Intelligence Unit (EIU), and published by ICANN are now past their feedback period. We, at Radix, believe that ICANN has received feedback from approximately 10 stakeholders, and I for one, am looking forward to those being published.

In light of the fact that none of the comments that ICANN received have been made public yet, I decided to blog about my multiple concerns with the new guidelines. Sparing a thought for the not-so-involved reader, I have limited my rant to some of the more important issues.

These also form the crux of our feedback submitted to ICANN along with a suggested red-lined version of the actual guidelines that can be found here.

#1: New guidelines cannot supersede the AGB

Of primary concern to us are certain communications between CTAG and ICANN, which we believe sought to modify the process and scoring guidelines in the AGB and relax the CPE standards.?

One can infer from Mr. Craig Schwartz’s recent letter to Ms. Christine Willett (VP, gTLD Operations) and correspondence on the NTAG list that a closed-door meeting was held in Durban where community TLD applicants potentially lobbied ICANN to amend the Guidebook and relax the CPE standards. ?

Hundreds of applicants have relied on the AGB and paid tens of millions of dollars in applying for specific strings. The publication of these new guidelines for feedback must not lead to a re-litigation of the Guidebook criteria. A change in the Guidebook at this stage would significantly change the rights of applicants and the value of their investments. Applicants would have made different decisions if the CPE criteria were different. ?

The CPE has been built upon a foundation of guiding principles that the community applicants seek to upset. A qualified community application eliminates all directly contending standard applications, regardless of how well qualified the latter may be. This is a fundamental reason for enforcing very stringent qualification requirements for a community-based application. It is for these reasons that we firmly believe that the AGB must be, in effect, sacrosanct.?

We believe that this concern was likely voiced by other applicants too. As a result, the CPE teleconference held on 10th September, 2013 started with Christine issuing a clarification that ICANN did not intend that the new guidelines would change anything in the AGB. We certainly hope that the intention shows up more clearly in EIU’s next draft of the guidelines.?

#2: A detailed process for the Community Priority Evaluation is still missing

The limited CPE process as defined in the current guidelines document provides for applications to be evaluated by two evaluators working independently to score applications. These will then be reviewed by members of the core project team to ensure consistency of approach across all applications.?

The process, when published, should clearly address scenarios where the evaluators and core project team members have conflicting views and specify which opinion would be binding. Additionally, there should be more detail on the documentation that will accompany each evaluation result. This, along with some clarity on how interactions between applicants and panels (public vs. private) will be managed as well as expected timelines for the complete process (incorporating the effects of GAC Advice and dispute resolution) would be a good start.

#3: Recommendations for Training and Process

ICANN and EIU must institute processes that will ensure consistency in CPE decisions.

Training: Follow-on documents from EIU should describe training and scenario testing to ensure that panels arrive at similar conclusions given identical data sets. Training should conclude when consistency is attained. Training should also include exposure to documents describing development of the community processes that include foundations in the guidebook. ICANN staff involved in the development should brief EIU panelists to describe the rationale for the current CPE standards.

Process: The process should include collaboration among panelists or evaluation of applications by the same evaluation panel (with several members) so that results are consistent. QA and other reviews should review rationale as well as scores to ensure the right scores were made for the right reasons. Rationale must be required for each score that relates to specific criteria and definitions in the Guidebook.?

Christine did mention on the CPE teleconference that scoring decisions would be accompanied with a justification for a particular score. While we find that encouraging, we can only hope that there will be consistency in the justifications provided across panels and across applications.

#4: The guidelines should be clearer and must not weaken the AGB criteria or change the application

In many cases, the new Guidelines, taken as written, seem to weaken the Guidebook criteria, either through vagueness or an inadvertent change to the intent. The new Guidelines should serve to retain the existing Guidebook criteria and provide additional clarity in a way that accomplishes the intent of the elaborate policy making process.?

While the EIU document provides additional information for evaluators, the information is vague in some places, and will undoubtedly lead to inconsistent results. An example where an explanation in the new guidelines creates uncertainty is:?

“‘Delineation’ also refers to the extent to which a community has the requisite awareness and recognition from its members.”
[How should the evaluator gauge whether a community has the “requisite awareness and recognition”?]

The use of Internet searches employed by evaluators is also of concern. Internet searches should be used only to the extent of verifying claims of the applicant. However information gathered from an Internet search should not be used towards modifying any claims/statements made in the application. It is important to limit the evaluators’ discretion in this area.

Which brings us to the next concern; the document omits significant portions of the AGB definitions and replaces them with general questions to be answered by panels. We are concerned that the EIU should not use a different standard than the published one.

Conclusion

While we are aware that the guidelines are well-intentioned, we have requested that care must be taken to ensure that the Guidebook intent is realized in the evaluation. Otherwise, the reliance placed by applicants upon the Guidebook processes and standards would have been misplaced. Those processes and standards were developed after long, intensive community discussion. Individual interests should not be allowed to change the agreed-upon implementation of the new gTLD policy at this late date. I rest my case!