Network Working Group R. Sparks
Internet-Draft Oracle
Intended status: Informational T. Kivinen
Expires: July 30, 2015 INSIDE Secure
January 26, 2015

Tracking Reviews of Documents
draft-sparks-genarea-review-tracker-00

Abstract

Several review teams ensure specific types of review are performed on Internet Drafts as they progress towards becoming RFCs. The tools used by these teams to assign and track reviews would benefit from tighter integration to the Datatracker. This document discusses requirements for improving those tools without disrupting current work flows.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on July 30, 2015.

Copyright Notice

Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.


Table of Contents

1. Introduction

As Internet Drafts are processed, reviews are requested from several review teams. For example, the General Area Review Team (Gen-Art) and the Security Directorate (Secdir) perform reviews of documents that are in IETF Last Call. Gen-art performs a follow-up review when the document is scheduled for an IESG telechat. These teams also perform earlier reviews of documents on demand. There are several other teams that perform similar services, often focusing on specific areas of expertise.

The secretaries of these teams manage a pool of volunteer reviewers. Documents are assigned to reviewers, taking various factors into account. For instance, a reviewer will not be assigned a document for which he is an author or shepherd. Reviewers are given a deadline, usually driven by the end of last call or a telechat date. The reviewer sends each completed review to the team's mailing list, and any other lists that are relevant for document being reviewed. Often, a thread ensues on one or more of those lists to resolve any issues found in the review.

The secretaries and reviewers from several teams are using a tool developed and maintained by Tero Kivinen. Much of its design predates the modern Datatracker. The application currently keeps its own data store, and learns about documents needing review by inspecting Datatracker and tools.ietf.org pages. Most of those pages are easy to parse, but the last-call pages, in particular, require some effort. Tighter integration with the Datatracker would simplify the logic used to identify documents ready for review, make it simpler for the Datatracker to associate reviews with documents, and allow users to reuse their Datatracker credentials. It would also make it easier to detect other potential review-triggering events, such as a document entering working group last call, or when a RFC's standard level is being changed without revising the RFC. Tero currently believes this integration is best achieved by a new implementation of the tool. This document captures requirements for that reimplementation, with a focus on the workflows that the new implementation must take care not to disrupt. It also discusses new features, including changes suggested for the existing tool at its issue tracker [art-trac].

For more information about the various review teams, see the following references

Gen-Art [Gen-Art] [RFC6385]
Secdir [Secdir]

2. Overview of current workflows

This section gives a high-level overview of how the review team secretaries and reviewers use the existing tool. It is not intended to be comprehensive documentation of how review teams operate. Please see the references for those details.

A team's secretary periodically (typically once a week) checks the tool for documents it has identified as ready for review. The tool has compiled this list from Last Call announcements and telechat agendas. The secretary creates a set of assignments from this list into the reviewer pool, choosing the reviewers in roughly a round-robin order. That order can be perturbed by several factors. Reviewers have different levels of availability. Some are willing to review multiple documents a month. Others may only be willing to review a document every other month. The assignment process takes exceptional conditions such as reviewer vacations into account. Furthermore, secretaries are careful not to assign a document to a reviewer that is an author, shepherd, responsible WG chair, or has some other already existing association with the document. The preference is to get a reviewer with a fresh perspective. The secretary may discover reasons to change assignments while going through the list of documents. In order to not cause a reviewer to make a false start on a review, the secretaries complete the full list of assignments before sending notifications to anyone. This assignment process can take several minutes, and it is possible for new last calls to be issued while the secretary is making assignments. The secretary typically checks to see if new documents are ready for review just before issuing the assignments, and updates the assignments if necessary. The issued assignments are sent to the review team list and are reflected in the tool. For those teams handling different types of reviews (Last Call vs Telechat for example), the secretary typically processes the documents for each type of review separately, and potentially with different assignment criteria. In Gen-art, for example, the Last Call reviewer for a document will almost always get the follow-up Telechat review assignment. Similarly, Secdir assigns any re-reviews of a document to the same reviewer. Other teams may choose to assign a different reviewer.

Reviewers discover their assignments through the announcement to the list or by looking at their queue in the tool. (Most reviewers only check the tool when they see they have an assignment via the list). A reviewer has the opportunity to reject the assignment for any reason. The secretary will find another volunteer for any rejected assignments. The reviewer can indicate that the assignment is accepted in the tool before starting the review.

The reviewer sends a completed review to the team's list, and any other lists relevant to the review. For instance, many last call reviews are also sent to the IETF general list. The teams typically have a template format for the review. Those templates usually start with a summary, describing the conclusion of the review. Typical summaries are "Ready for publication" or "On the right track, but has open issues". The reviewer uses the tool to indicate that the review is complete, provides the summary, and has an opportunity to provide a link to the review in the archives. (Note, however, that having to wait for the document to appear in the archive to know the link to paste into the tool is a significant enough impedance that this link is often not provided by the reviewer. The Secdir secretary manually collects these links from the list and adds them to the tool.)

Occasionally, a document is revised between when a review assignment is made and when the reviewer starts the review. Different teams can have different policies about whether the reviewer should review the assigned version or the current version.

3. Requirements

3.1. Secretary focused

o
A secretary must be able to see what documents are ready for review of a given type (such as a Last Call review)
o
A secretary must be able to assign reviews for documents that may not have been automatically identified as ready for a review of a given type. (This allows the secretary to work around errors and handle special cases, including early review requests.)
o
A secretary must be able to work on and issue a set of assignments as an atomic unit. No assignment should be issued until the secretariat declares the set of assignments complete.
o
It must be easy for the secretary to discover that more documents have become ready for review while working on an assignment set.
o
The tool should make preparing the assignment email to the team's list easy. For instance, the tool could prepare the message, give the secretary an opportunity to edit it, and handle sending it to the list.
o
A secretary must be able to easily see who the next available reviewers are, in order.
o
A secretary must be able to edit a Reviewer's availability, both in terms of frequency, not-available-until-date, and skip-next-n-assignments. (See the description of these settings in the Reviewer focused section.)
o
The tool should make it easy for the secretary to identify that a reviewer is already involved with a document. The current tool allows the secretary to provide a regular expression to match against the document name. If the expression matches, the document is not available for assignment to this reviewer. For example, Tero will not be assigned documents matching '^draft-(kivinen|ietf-tcpinc)-.*$'. The tool should also take any roles, such as document shepherd, that the Datatracker knows about into consideration.
o
The tool should make it easy for the secretary to see key features of a document ready for assignment, such as its length, its authors, the group and area it is associated with, its title and abstract, and any other personnel (such as the shepherd and reviewers already assigned from other teams) involved in the draft.
o
The tool must make it easy for the secretary to detect and process re-review requests on the same version of a document (such as when a document has an additional last call only to deal with new IPR information).
o
Common operations to groups of documents should be easy for the secretary to process as a group with a minimum amount of interaction with the tool. For instance, it should be possible to process all the of documents described by the above bullet with one action. Similarly, for teams that assign re-reviews to the same reviewer, issuing all re-review requests should be a simple action.
o
The tool must make it easy for the secretary to see the result of previous reviews from this team. In Secdir, for example, if the request is for a revision that has only minor differences, and the previous review result was "Ready", a new assignment will not be made.
o
The tools must make it easy for the secretary to see the recent performance of a reviewer while making an assignment (see Section 3.4). This allows the secretary to detect overburdened or unresponsive volunteers earlier in the process.
o
A secretary must be able to withdraw a review assignment.

3.2. Reviewer focused

o
A reviewer must be able to indicate availability, both in frequency of reviews, and as "not available until this date." The current tool speaks of frequency in these terms:
Assign at maximum one new review per week
Assign at maximum one new review per fortnight
Assign at maximum one new review per month
Assign at maximum one new review per two months
Assign at maximum one new review per quarter
o
Reviewers must be able to indicate that they should be skipped the next n times they would normally have received an assignment.
o
A reviewer must be able easily discover new review assignments. (The tool might send email directly to an assigned reviewer in addition to sending the set of assignments to the team's list. The tool might also use the Django Message framework to let a reviewer that's logged into the Datatracker know a new review assignment has been made.
o
Reviewers must be able to see their current set of outstanding assignments, completed assignments, rejected assignments. The presentation of those sets should either be separate, or, if combined, the sets should be visually distinct.
o
A reviewer must be able to reject a review assignment, optionally providing the secretary with an explanation for the rejection.
o
A reviewer must be able to indicate that have accepted and are working on an assignment
o
It should be possible for a reviewer to reject or accept a review either by using the tool's web interface, or by replying to the review assignment email.
o
A reviewer must be able to indicate that a review is complete, capturing where the review is in the archives, and the high-level review-result summary
o
It must be possible for a reviewer to clearly indicate which version of a document was reviewed. Documents are sometimes revised between when a review was assigned and when it is due. The tool should note the current version of the document, and highlight when the review is not for the current version.
o
It must be easy for a reviewer to submit a completed review.
-
The current workflow, where the reviewer sends email to the team list (possibly copying other lists) and then indicates where to find that review must continue to be supported. The tool should make it easier to capture the link to review in the list archives (perhaps by suggesting links based on a search into the archives).
-
The tool should allow the reviewer to enter the review into the tool via a web form. The tool will ensure the review is posted to the appropriate lists, and will construct the links to those posts in the archives.
-
The tool could also allow the reviewer to submit the review to the tool by email (perhaps by replying to the assignment). The tool would then ensure the review is posted to the appropriate lists.

3.3. Review Requester and Consumer focused

o
It should be easy for an AD or group chair to request any type of review, but particularly an early review, from a review team.
o
It should be possible for that person to withdraw a review request.
o
It must be easy to find all reviews of a document when looking at the document's main page in the Datatracker. The reference to the review must make it easy to see any responses to the review on the lists it was sent to.
o
It must be easy to find all reviews of a document when looking at search result pages, and other lists of documents such as the documents on a telechat agenda.

3.4. Statistics focused

o
It must be easy to see the following, across all teams, a given team, or a given reviewer, and independently across all time, or across configurable recent periods of time:
-
How many reviews have been completed
-
How many reviews are in progress
-
How many in progress reviews are late
-
How many completed reviews were late
-
How many reviews were not completed at all
-
Average time to complete reviews (from assignment to completion)
o
It must be easy to see, for all teams, for a given team, or for a given reviewer, across all time, or across configurable recent periods:
-
Total counts of reviews in each review state (done, rejected, etc.)
-
Total counts of completed reviews by result (ready, ready with nits, etc.)
o
Where possible, the above statistics should be visible as a time-series graph.
o
It should be possible view the correlation between completed reviews results with number of discuss positions taken on a given document. Future enhancements may allow ADs to indicate their position was informed by a given review to help make this view more meaningful.

4. Security Considerations

This document discusses requirements for tools that assist review teams. These requirements do not affect the security of the Internet in any significant fashion. The tools themselves have authentication and authorization considerations (team secretaries will be able to do different things than reviewers). None of these have been identified as non-obvious.

5. IANA Considerations

This document has no actions for IANA.

6. Acknowledgments

Tero Kivinen and Henrik Levkowetz were instrumental in forming this set of requirements and in developing the initial Django models in the appendix.

7. Changelog

7.1. 00

8. Informative References

, ", ", "
[Gen-Art]General Area Review Team Guidelines", Work in Progress , January 2015.
[RFC6385] Barnes, M., Doria, A., Alvestrand, H. and B. Carpenter, General Area Review Team (Gen-ART) Experiences", RFC 6385, October 2011.
[Secdir]Security Directorate", Work in Progress , January 2015.
[art-trac]Area Review Team Tool - Active Tickets", Work in Progress , January 2015.

Appendix A. A starting point for Django models supporting the review tool

from django.db import models
from ietf.doc.models import Document
from ietf.person.models import Email
from ietf.group.models import Group, Role

from ietf.name.models import NameModel

class ReviewRequestStateName(NameModel):
    """ Requested, Accepted, Rejected, Withdrawn, Overcome By Events, 
        No Response , Completed  """

class ReviewTypeName(NameModel):
    """ Early Review, Last Call, Telechat """

class ReviewResultName(NameModel):
    """Almost ready, Has issues, Has nits, Not Ready, 
       On the right track, Ready, Ready with issues, 
       Ready with nits, Serious Issues"""

class Reviewer(models.Model):
    """
    These records associate reviewers with review team, and keeps track
    of admin data associated with the reviewer in the particular team.
    There will be one record for each combination of reviewer and team.
    """
    role        = models.ForeignKey(Role)
    frequency   = models.IntegerField(help_text=
                                  "Can review every N days")
    available   = models.DateTimeField(blank=True,null=True, help_text=
                        "When will this reviewer be available again")
    filter_re   = models.CharField(blank=True)
    skip_next   = models.IntegerField(help_text=
                         "Skip the next N review assignments")

class ReviewResultSet(models.Model):
    """
    This table provides a way to point out a set of ReviewResultName 
    entries which are valid for a given team, in order to be able to 
    limit the result choices that can be set for a given review, as a 
    function of which team it is related to.
    """
    team        = models.ForeignKey(Group)
    valid       = models.ManyToManyField(ReviewResultName)

class ReviewRequest(models.Model):
    """
    There should be one ReviewRequest entered for each combination of 
    document, rev, and reviewer.  
    """
    # Fields filled in on the initial record creation:
    time          = models.DateTimeField(auto_now_add=True)
    type          = models.ReviewTypeName()
    doc           = models.ForeignKey(Document, 
                           related_name='review_request_set')
    team          = models.ForeignKey(Group)
    deadline      = models.DateTimeField()
    requested_rev = models.CharField(verbose_name="requested_revision",
                            max_length=16, blank=True)
    state         = models.ForeignKey(ReviewRequestStateName)
    # Fields filled in as reviewer is assigned, and as the review 
    # is uploaded
    reviewer      = models.ForeignKey(Reviewer, null=True, blank=True)
    review        = models.OneToOneField(Document, null=True, 
                                                   blank=True)
    reviewed_rev  = models.CharField(verbose_name="reviewed_revision", 
                                     max_length=16, blank=True)
    result        = models.ForeignKey(ReviewResultName)

Authors' Addresses

Robert Sparks Oracle 7460 Warren Parkway Suite 300 Frisco, Texas 75034 USA EMail: rjsparks@nostrum.com
Tero Kivinen INSIDE Secure Eerikinkatu 28 HELSINKI, FI-00180 FI EMail: kivinen@iki.fi