Network Working Group A. Doria (editor) Internet-Draft ETRI Expires: January 12, 2006 H. Alvestrand Cisco Systems B. Carpenter IBM Zurich Research Laboratory July 11, 2005 Experience with the General Area Review Team draft-doria-genart-experience-00.txt Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on January 12, 2006. Copyright Notice Copyright (C) The Internet Society (2005). Abstract The General Area Review team has been doing Late Reviews of Internet Drafts since 2004. This draft discusses the experience and the lessons learned in the first 18 months of this process. Doria (editor), et al. Expires January 12, 2006 [Page 1] Internet-Draft GenART July 2005 Table of Contents 1. Discussion Venue . . . . . . . . . . . . . . . . . . . . . . . 3 2. What is GenART . . . . . . . . . . . . . . . . . . . . . . . . 3 3. Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 4. GenART Reviews . . . . . . . . . . . . . . . . . . . . . . . . 3 4.1 IESG Telechat Review Process . . . . . . . . . . . . . . . 3 4.2 IETF LC Assignments . . . . . . . . . . . . . . . . . . . 5 4.3 Form of review . . . . . . . . . . . . . . . . . . . . . . 6 4.4 Archiving of reviews . . . . . . . . . . . . . . . . . . . 7 5. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 6. Impressions . . . . . . . . . . . . . . . . . . . . . . . . . 7 6.1 Reviewers' Impressions . . . . . . . . . . . . . . . . . . 8 6.2 One General Area Director's Impressions . . . . . . . . . 9 6.3 One GenARt Secretary's Impressions . . . . . . . . . . . . 10 7. Needed Improvements . . . . . . . . . . . . . . . . . . . . . 11 8. Applicability . . . . . . . . . . . . . . . . . . . . . . . . 11 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 11 10. Informative References . . . . . . . . . . . . . . . . . . . 12 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . 12 A. IANA considerations . . . . . . . . . . . . . . . . . . . . . 12 Intellectual Property and Copyright Statements . . . . . . . . 13 Doria (editor), et al. Expires January 12, 2006 [Page 2] Internet-Draft GenART July 2005 1. Discussion Venue Discussion of this proposal is intended to place on the IETF mailing list in the absence of a better home. 2. What is GenART The General Area Review team was created personally by the former General Area Director and retained by the current one. It has no official role in the IETF standards process, except as a set of individuals entitled, like everyone, to comment on Internet-Drafts. Its Secretary, and the team of volunteer reviewers, serve at the invitation of the General AD. The original secretary was Avri Doria, recently replaced by Mary Barnes, who previously served as a reviewer. The current General AD was also previously a reviewer. 3. Goals The original and continuing goal of the GenART team was, and is, to offload some of the burden from the General Area AD of IESG reviews. The load for the bi-weekly IESG reviews is often quite large; occasionally there are 20 drafts or more scheduled for discussion in a single telechat. GenART was based on a model that had proved productive in the Ops Directorate: Quick review close to telechat time, to advise the AD on issues that remain serious. By having a trusted group of reviewers read and evaluate the drafts, the General Area AD would be able to concentrate on those drafts where there was an concern expressed by the reviewer. 4. GenART Reviews 4.1 IESG Telechat Review Process The process for reviewing documents when they appear on the IESG agenda: o The nearly final IESG meeting agenda generally appears on Thursday night, one week before the IESG telechat. The GenART secretary uses this as the input for the assignment process. o For documents reviewed at Last Call, a new review is only asked for if the document is revised. In this case the reviewer, if possible the person who did the last call review, only needs to check that any open issues were resolved. Often the draft will not have changed between IETF LC and the IESG telechat review. o The Secretary names a reviewer per document, more or less randomly. This assignment process is currently very labor intensive and is completely manual (although much of it could, in Doria (editor), et al. Expires January 12, 2006 [Page 3] Internet-Draft GenART July 2005 theory, be automated). 1. Remove everyone who has announced they are on vacation, or is otherwise unavailable for reviewing, from the resource list. 2. All previously reviewed drafts are assigned to the reviewer who reviewed them previously, assuming that reviewer is available. Otherwise, these documents are assigned to a new person in the process described below. 3. Make note of the groups of drafts that are being grouped as a single ballot. If the load is not too great, an attempt is made to assign all the documents in a single ballot to a single reviewer. Sometimes, given the length or number of documents, this is unreasonable and the group is split between several reviewers. 4. Make a note of whether any of the reviewers is involved with the draft as either an author or the chair of the WG. This is sometimes difficult to get right and occasionally, given the fact that most GenART reviewers are active IETF participants, sometimes a reassignment is required once the original assignments are announced. 5. At this point, a very modified round robin process is used to make the remaining review assignments. The process is approximately as follows: + Begin the round robin from where the previous week's assignments stopped. + If someone has indicated a restricted load, e.g. they have only agreed to review one draft per cycle, skip them once that load is met. + For every two repeat reviews someone has, skip them in one cycle of the round robin. The reason for equating two repeat reviews to a single new review is that there is an assumption that the task of re-reviewing is lighter then the process of reviewing. Note: if someone is being assigned a returning document as if for the first time, it is treated the same as any new assignment. + After the round robin is completed, then the secretary looks at the list and if it looks very imbalanced with someone having far too much work to do then some load shifting may be done. There is no attempt made to actually equalize the load as the length and complexity of the drafts is not taken into account in this process. (Thus, a reviewer could end up with a couple of hundred page documents, but this is statistically rare.) o It should be mentioned that assignment is never made according to a reviewers technical specialty. Even though it happens, when, for example, routing drafts fall on routing experts or MIBs fall on MIB doctors, it is coincidental. To the reviewer, the choice looks random. Doria (editor), et al. Expires January 12, 2006 [Page 4] Internet-Draft GenART July 2005 o Once the assignments are made, the web pages that list the reviews and the assignments are posted. If the reviewers notice any problems or conflict of interest, a bargaining process, shifting documents from one reviewer to another, takes place. o Once the review has been completed the reviewer sends the review to the GenART list. o The secretary takes the reviews, sometimes edits them for format, records the review on the web pages and gives each review a short synopsis. o The reviews are them posted on the public web page (this may occur after the review has been read by the AD, or even after the IESG telechat). o If the AD concludes that the concerns raised by the reviewer warrant placing a DISCUSS comment on the document, the AD will do so, and the DISCUSS must be resolved before the document advances. Usually, the reviewer will be involved in the resolution process. o Occasionally, and more often of late, reviewers have been mailing reviews to authors, ADs or WGs. This part of the process has been voluntary, and sometime resulted in confusion among the authors and working groups. Reviewers have begun appending a description of GenART and the purpose of the review on these occasion to lessen the confusion. o All reviews need to be completed and posted before the Thursday telechats. For the reviews to be useful to the General area AD the earlier the posting the better. It should be noted, that this allows the reviewers only 4 working days and the weekend, for those who work on the weekend, to complete their reviews. The secretary generally needs to work late on the Wednesday night before the telechat to record all the information. In fact the secretary's job usually requires night work (depending on time zone effects). It also requires a responsive Internet connection, even when on travel. 4.2 IETF LC Assignments While the original process was meant only for late reviews before the IESG telechat, it was decided to include IETF Last call reviews. In some ways this proved to be an overloading of the process and has presented difficulties. Obviously the greatest utility for IETF LC would be achieved by having the reviews completed before the end of the last call. However due to the secretary's schedule and the difficulty of assigning an IETF LC review as soon as it was announced, these assignment were often made midway through the the LC timing, and sometimes even after the IETF LC ended. Within GenART practice to date, the IETF LC assignments took on a different aspect as they were treated more as early warning of pending IESG reviews then as IETF LC Doria (editor), et al. Expires January 12, 2006 [Page 5] Internet-Draft GenART July 2005 requirements. While there were many occasions on which an IETF LC review was done before the end of the last call, these occasions were treated as happy coincidences and not as a requirement. The nature of the IETF LC assignments has remained an issue for the GenART as the General Area AD would prefer for them to completed by the end of the IETF LC. There have been frequent conversations on the GenART mailing list about the nature and timeliness of IETF LC reviews. IETF LC reviews were assigned on a strictly round robin basis with the exception that an IETF LC document would not be assigned to someone who was scheduled to be on vacation for the entire period of the LC. Otherwise a separate list of the reviewers was maintained and each assignment was done in order. 4.3 Form of review Rather than invent new guidelines, the GenART requirements for the form of a review stole liberally from draft-carpenter-solutions-sirs-01, making adaptations for the special "late, quick review" case and the nature of the General area's concerns. Each review must start with a summary statement chosen from or adapted from the following list: o This draft is ready for publication as a [type] RFC, where [type] is Informational, Experimental, etc. (In some cases, the review might recommend publication as a different [type] than requested by the author.) o This draft is basically ready for publication, but has nits that should be fixed before publication. o This draft is on the right track but has open issues, described in the review. o This draft has serious issues, described in the review, and needs to be rethought. o This draft has very fundamental issues, described in the review, and further work is not recommended. o Unfortunately, I don't have the expertise to review this draft. The length of a review can vary greatly according to circumstances, and it is considered acceptable for purely editorial comments to be sent privately if it's obvious that the document needs substantial revision. All substantive comments, however, must be included in the public review. Wherever possible, comments should be written as suggestions for improvement rather than as simple criticism. Explicit references to prior work and prior IETF discussion should be given whenever possible. Reviewers are asked to review for all kinds of problems, from basic Doria (editor), et al. Expires January 12, 2006 [Page 6] Internet-Draft GenART July 2005 architectural or security issues, Internet-wide impact, technical nits, problems of form and format (such as IANA Considerations or incorrect references),and editorial issues. Since these reviews are on documents that are supposed to be finished, the review should consider "no issue too small" - but should cover the whole range from the general architectural level to the editorial level. All reviews should apply generally agreed IETF criteria, such as: o [RFC1958] The Architectural Principles of the Internet o [RFC3426] General Architectural and Policy Considerations o [RFC3439] Some Internet Architectural Guidelines and Philosophy o [NITS] The "I-D Nits" document maintained by the IESG o [RFC2223] Instructions to RFC Authors o [BCP26] Guidelines for Writing an IANA Considerations Section in RFCs o [RFC3552] Guidelines for Writing RFC Text on Security Considerations o As well as any other applicable architectural or procedural documents. It is considered important that reviews give precise references to such criteria when relevant to a comment. Of special interest to the GEN area (because it's no area's special interest) is: o Clear description of why the document or protocol is useful to the Internet Adherence to IETF formalities such as capitalized MUST/SHOULD (ID-Nits) o Useful and reasonable IANA considerations o Correct dependencies for normative references o That it's written in reasonably clear English 4.4 Archiving of reviews All reviews are archived and publicly visible. The archive can be found at: http://www.alvestrand.no/ietf/gen/art/gen-art.html 5. Results Over the last 18 months, the GenART has provided reviewing services to 2 ADs and has done over 450 publicly available reviews. Each of these reviews was been done in a short interval of about a week by a dozen or fewer reviewers. 6. Impressions This section is divided into 3 subsections, the impressions as gathered from the GenART review team, the impressions of the ADs for who they worked, and the impressions of one the secretaries of the GenART. Doria (editor), et al. Expires January 12, 2006 [Page 7] Internet-Draft GenART July 2005 6.1 Reviewers' Impressions The following list of comments are excerpted and edited, but without attribution, from comments sent in by the reviewers of GenART in response to the request: "We'd like to ask you each to write a few lines about your personal experience and lessons learned as a Gen-ART reviewer." o We really do find problems, but we don't find problems with most documents. o Comments seem to be in three areas: editorial/grammar, editorial/ what-the-heck-does-this-mean, and actual problems. I'm seeing fewer reviews in the first category, which is a good thing. o Since we're not a documented step in the process, a fair number of authors/editors don't know what to do with us. Arguing is rare, but negotiating is more common, and it's pretty frequent that someone asks "do I rev the document now or wait until the telechat?" Adding the boilerplate explaining who we are, why we are annoying this particular document, etc. seems to help, but it would be nice to agree on a particular boilerplate. o It is becoming rarer that we hear back "these guys have suffered enough, I'm voting no objection" (I'm remembering an LDAP document that had been around so long it had 2119 referenced AS A DRAFT - some people suffered a lot). o The direct assignment of reviews is necessary and effective. It does not matter much as far as I can tell what scheme is used to actually do the assignment. o Folks are very open to the reviews that come out of GenART. This somewhat surprised me because I have seen resistance to outside reviews in other cases. o The improvements that have come about (for example one of my latest, the sipping conference draft - whatever the outcome) have made a big difference to the comprehensibility and usability of the documents - and provide a useful incentive to keeping going. o Some form of review like this is desperately needed. While most of the stuff we see is good, every once in a while really bad errors have made their way all the way to IESG vote. o Reading this stuff is interesting. I like having a reason to read a wide range of materials. o I am more than convinced that this can be and is a valuable process. It is IMO a pity that SIRS and so on did not take off, because this late stage reviewing is a poor substitute for doing the same thing at a much earlier stage. Very few of the drafts that have come past my screen are truly fully ready for IESG review. It is actually a joy to find the occasional nugget that is both well written and is a proper technical job, such that the review really can say 'This is ready'. Doria (editor), et al. Expires January 12, 2006 [Page 8] Internet-Draft GenART July 2005 o I have certainly found the process intellectually stimulating! It encourages me to take a wider interest in what is going on in the IETF, but consumes a fair bit of time to do a proper job, and requires a very wide knowledge to be able to properly catch the cross-area implications: I hope (believe!) that this is something that one gets better at with experience and doing a few of these reviews. o There are probably a very limited pool of people who have both the time and the inclination to keep on doing these reviews. It does require a fair bit of dedication. o It is difficult to avoid correcting the English, even if that is not really the point: Often really bad English (whether as a result of non-mother tongue authors with limited grasp or mother tongue authors using informal language) obscures/corrupts what is being said or just makes it impossible to read. o Mostly authors welcome the comments: I think most of them understand the concept of 'ego-free reviewing' and we have generally been constructive rather than destructive. o Part of the job of gen-art is to think the unthinkable from another point of view, to challenge (apparently undocumented) assumptions and apply experience from other fields. 6.2 One General Area Director's Impressions It's essential. The reviewing load for the IESG DOES NOT SCALE. On a single fortnight example, the IESG had 21 drafts on the agenda. It is just impossible, and no wonder we sometimes miss serious issues. So I think a distributed review team with o(30) trusted reviewers needs to be institutionalized. I suspect that will need to be formalized in a BCP sooner or later - with their reviews having a formal position in the standards process, and the expectation that the whole IESG truly reviews all documents being relaxed. I think we've learned that polite, well reasoned, constructive reviews are very positively received by authors and WGs. Dismissive reviews are counter-productive. And reviews sent in private eventually show up in public, so it's better to go public at the start. I believe that there is a bit of a problem in the way we handle LC reviews right now. Sometimes the timing works out well (the LC review comes in good time, and the draft gets revised prior to hitting the IESG agenda). But quite often the LC review never happens, or happens so late that the document still has serious Doria (editor), et al. Expires January 12, 2006 [Page 9] Internet-Draft GenART July 2005 issues when it hits the agenda. The other problem I see is a big detail - between late Thursday or early Friday when the secretary sends out the assignments, and Wednesday when I like to start filling in ballots, there are only three work days (plus possible volunteer time over the weekend). Now even with only one document to review, that may be a real challenge. Sometimes, a lucky reviewer will get 130 pages (e.g. draft-ietf-nntpext-base-27). That doesn't compute. There are some mechanical issues. The process followed is far too manual. Everything needs to be robotic except for the judgment calls about which reviewer gets which draft. Similarly, the reviewer should be able to just paste the review into a web form, click, and it's sent off to everyone appropriate and posted to the review site. That, by the way, would be the same tool that we also need to support early review, WG Last Call review, etc. 6.3 One GenARt Secretary's Impressions Serving as the secretary of GenART was a worthwhile experience. From a personal point of view, it gave me an easy way to track all of the work going through the IESG review process and see how the work flowed through that process. Also, by reviewing and doing light editing on all of the reviews in order to create some degree of uniformity of presentation and to create the one line abstracts that go on the review web page, I had the opportunity to really get a survey of the work being approved by the IETF. One element of the GenART that makes it work is the immediacy of the review. I think that the fact that these are late reviews that have a particular advisory role allows a relatively small group of people to read and review. The nature of these reviews is informal, and originally the reviews were only intended for the General Area AD, though they were made public. During 2004 there was little if any interaction between authors and reviewers. There was some discussion during 2004 about trying to expand the role of GenART to a more formal, early review model, i.e to evolve it into a form of SIRS. I was against such a transformation because I felt it would risk something that worked. I believed that the risk was inherent in formalizing the reviews and in adding mechanisms for standardizing the review mechanisms that would resort from formalization. Another concern involves the interaction between reviewers and authors. As discussed above, recently, it has become the practice to send reviews to the authors with an explanation about the nature of GenART reviews. While it is clear that this has Doria (editor), et al. Expires January 12, 2006 [Page 10] Internet-Draft GenART July 2005 resulted in improved RFCs, it has also resulted in increased work load for the reviewers. The addition of IETF LC reviews was also a potential increase in work load for the reviewers. If these review assignments were only to be used as an early warning for a reviewer of a draft that would eventually be in IESG review, then I saw it as a helpful to the reviewers. If, on the other hand, they became an extra work load that need to be completed by the end of LC then I thought it was harmful. I think that GenART is an experiment that works but I believe it is fragile. As secretary I was often concerned about overburdening reviewers, and felt it my responsibility to keep them from burn out; though I was never sure how to achieve that given the work load. 7. Needed Improvements Automation of the manual processes of assignment, and of mailing and web-posting the resulting reviews seems, to be the main improvement needed. For example, the IETF Secretariat system generates an "Evaluation" message to the IESG whenever a ballot is initiated for a draft. These messages, and "IETF Last Call" messages, could be used to trigger assignment of reviews to reviewers, with only exceptional cases needing manual handling. A web tool could allow reviewers to upload their reviews and send mail to interested parties. 8. Applicability As implemented today, the process has no formal role in the IETF standards process. Whether the reviews appear during IETF Last Call or during IESG Evaluation, they have no distinguished status. But as trust in the review team has built, and as the team itself has learned to deliver reviews that are generally well received, they have had a significant impact on document quality and on timeliness. Rather than becoming a roadblock, they have (in general) allowed the General AD to feel more confident in reaching decisions and be more precise in resolving issues. A question that naturally arises is whether a formal or official role for such a team would bring even more benefits. 9. Acknowledgments Initial comments were received from the members of the GenART team and the experiences discussed in this document were derived from their hard work over the last 18 months as reviewers. We thank the reviewers of GenART: Mark Allman, Mary Barnes, David Black, Scott Brim, Elwyn Davies, Spencer Dawkins, Lakshminath Dondeti, Joel Halpern, John Loughney, Lucy Lynch, Michael Patton, and Suzannne Doria (editor), et al. Expires January 12, 2006 [Page 11] Internet-Draft GenART July 2005 Woolf 10. Informative References Authors' Addresses Avri Doria ETRI Lulea University of Technology Lulea SE Phone: +1 401 663 5024 Email: avri@acm.org Harald Alvestrand Cisco Systems Weidemanns vei 27 Trondheim 7043 NO Phone: Email: harald@alvestrand.no Brian E Carpenter IBM Zurich Research Laboratory Saeumerstrasse 4 8803 Rueschlikon Switzerland Phone: Email: brc@zurich.ibm.com Appendix A. IANA considerations As this is an informational document about an IETF process, there are no IANA considerations. However, one of the requirements for an IETF document is the inclusion of IANA considerations section. One of the areas that the IESG review includes, and thus the GenART review needs to include is a reading of the IANA section to make sure that it not only exists, but that it includes Doria (editor), et al. Expires January 12, 2006 [Page 12] Internet-Draft GenART July 2005 Intellectual Property Statement The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org. The IETF has been notified of intellectual property rights claimed in regard to some or all of the specification contained in this document. For more information consult the online list of claimed rights. Disclaimer of Validity This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Copyright Statement Copyright (C) The Internet Society (2005). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. Doria (editor), et al. Expires January 12, 2006 [Page 13] Internet-Draft GenART July 2005 Acknowledgment Funding for the RFC Editor function is currently provided by the Internet Society. Doria (editor), et al. Expires January 12, 2006 [Page 14]