RDAP Mirroring Protocol (RMP)
draft-harrison-regext-rdap-mirroring-00

Abstract

The Registration Data Access Protocol (RDAP) is used by Regional Internet Registries (RIRs) and Domain Name Registries (DNRs) to provide access to their resource registration information. While most clients can retrieve the information they need on an ad hoc basis from the public services maintained by each registry, there are instances where local copies of those remote data sources need to be maintained, for various reasons (e.g. performance requirements). This document defines a protocol for transferring bulk RDAP response data and for keeping a local copy of that data up to date.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on August 5, 2019.

Copyright Notice

Copyright (c) 2019 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

• 1. Introduction
• 1.1. Requirements Language
• 2. RDAP Mirroring Protocol Implementation
• 2.1. Overview
• 2.2. File Definitions
• 2.2.1. Update Notification File
• 2.2.2. Snapshot File
• 2.2.3. Delta File
• 2.3. RDAP Objects
• 2.4. Serial Numbers
• 2.5. Server Use
• 2.5.1. Initialization
• 2.5.2. Publishing Updates
• 2.5.3. Consolidation
• 2.6. Client Use
• 2.6.1. Processing the Update Notification File
• 2.6.1.1. Initial
• 2.6.1.2. Subsequent
• 3. Operational Considerations
• 4. Security Considerations
• 5. Acknowledgements
• 6. IANA Considerations
• 7. References
• 7.1. Normative References
• 7.2. Informative References
• Authors' Addresses

1. Introduction

The Registration Data Access Protocol (RDAP) is used by Regional Internet Registries (RIRs) and Domain Name Registries (DNRs) to provide access to their resource registration information. For a client, this typically involves following the bootstrap process [RFC7484] to determine the base URL for the query, constructing an RDAP request, sending it, and then processing the response.

This mode of operation is appropriate for many use cases. However, some clients may need local access to the whole data set:

• their performance requirements may be such that the time required for sending/receiving HTTP requests to arbitrary remote servers is not acceptable;
• they may be conducting analysis of the data set as a whole; or
• they may be providing access to the data set in their own right, as an alternative to redirecting to the authoritative source for the data.

This document defines a protocol that can be used by a client to retrieve a local copy of a remote RDAP data set, as well as to maintain that local copy as further remote updates occur.

1.1. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

2. RDAP Mirroring Protocol Implementation

2.1. Overview

A registry that wants to make use of this protocol publishes an Update Notification File to a specified URL. That file in turn links to a Snapshot File and a series of Delta Files. The Snapshot File contains all of the registry's RDAP state as at a given point in time. The Delta Files contain changes that have been made to the registry's RDAP state since the Snapshot File was generated.

As further changes are made to the registry's RDAP state, the registry publishes new Delta Files, amends the Update Notification File to include links to the new Delta Files, and then republishes the Update Notification File. Periodically, the registry regenerates and republishes the Snapshot File, which in turn allows for older Delta Files to be removed from the Update Notification File.

All files in the protocol are signed by the server using JSON Web Signature (JWS) [RFC7515]. The JSON Web Key (JWK) [RFC7517] used by the server to sign the files is distributed out-of-band.

A client that wants to make use of this protocol needs to learn the server's Update Notification File URL and JSON Web Key out-of-band. Once these are known, the client retrieves the Update Notification File, validates its signature, and follows the links in it to retrieve the Snapshot File and Delta Files. It validates the signatures on these files, and then uses them to initialize its local state. It then records the serial number of the most-recently-issued Delta File, or of the Snapshot File if no Delta Files are present.

The client will then periodically retrieve the Update Notification File, determine the Delta Files that have been added since it was last retrieved by the client, retrieve those Delta Files, and update its local state accordingly.

A server may opt not to publish a Snapshot File in the Update Notification File. Such servers will only publish Delta Files in their Update Notification File, and must distribute the Snapshot File out-of-band.

2.2. File Definitions

2.2.1. Update Notification File

                
{
    "version": 1,
    "refresh": 3600,
    "snapshot": { "uri": "https://example.com/1/snapshot.json",
                  "serial": 1 },
    "deltas": [
        { "uri": "https://example.com/2/delta.json",
          "serial": 2 },
        { "uri": "https://example.com/3/delta.json",
          "serial": 3 },
    ]
}
                
                

Example Update Notification File:

• Update Notification Files MUST be well-formed JSON.
• The "version" attribute in the root element MUST be present, with a value of "1".
• A "refresh" attribute MAY be present. If it is present, it is an integer representing how long the client should wait (in seconds) after retrieving the Update Notification File before attempting to retrieve it again.
• A "snapshot" attribute containing a link to a Snapshot File MAY be present.
• A "deltas" attribute containing an array of links to Delta Files MUST be present. If no Delta Files have been published by the server, this array will be empty.
• The Delta File entries in the "deltas" attribute MUST be in serial number order, and the serial numbers MUST form a contiguous sequence.
• If a Snapshot File is included, its serial number MUST either be equal to that of one of the Delta Files, or one less than the smallest Delta File serial number.

2.2.2. Snapshot File

                
{
    "version": 1,
    "serial": 3,
    "defaults": { "port43": "whois.example.com",
                  ... },
    "objects": [
        { "id": "https://example.org/I1",
          "object": { "rdapConformance": [ "rdap_level_0" ],
                      "objectClassName": "ip network",
                      ... } },
        { "id": "https://example.org/D2",
          "object": { "rdapConformance": [ "rdap_level_0" ],
                      "objectClassName": "domain",
                      ... } },
        ...
    ]
}
                
                

Example Snapshot File:

• Snapshot Files MUST be well-formed JSON.
• The "version" attribute in the root element MUST be present, with a value of "1".
• The "serial" attribute in the root element MUST be present, with a value that is an unsigned 32-bit integer.
• An "objects" attribute containing an array of RDAP object (see [RFC7483]) and identifier pairs MUST be present. If no RDAP objects have been published by the server, this array will be empty.
• A "defaults" attribute MAY be present. For each object received from the server, the client should treat the object as also having each attribute from the "defaults" attribute, except where the object already contains an attribute with that name. This applies to objects received both before and after the Snapshot File is processed.

2.2.3. Delta File

                
{
    "version": 1,
    "serial": 4,
    "defaults": { "port43": "whois-2.example.org",
                  ... },
    "removed_objects": [ "1" ],
    "added_or_updated_objects": [
        { "id": "https://example.org/I3",
          "object": { "rdapConformance": [ "rdap_level_0" ],
                      "objectClassName": "ip network",
                      ... } },
        { "id": "https://example.org/D4",
          "object": { "rdapConformance": [ "rdap_level_0" ],
                      "objectClassName": "domain",
                      ... } },
        ...
    ]
}
                
                

Example Delta File:

• Snapshot Files MUST be well-formed JSON.
• The "version" attribute in the root element MUST be present, with a value of "1".
• The "serial" attribute in the root element MUST be present, with a value that is an unsigned 32-bit integer.
• A "defaults" attribute MAY be present. For each object received from the server, the client should treat the object as also having each attribute from the "defaults" attribute, except where the object already contains an attribute with that name. This applies to objects received both before and after the Delta File is processed.
• A "removed_objects" attribute containing an array of RDAP object identifiers MUST be present. If no RDAP objects have been removed since the previous Delta File was generated by the server, this array will be empty.
• An "added_or_updated_objects" attribute containing an array of RDAP object (see [RFC7483]) and identifier pairs MUST be present. If no RDAP objects have been added or updated since the previous Delta File was generated by the server, this array will be empty.

2.3. RDAP Objects

The base RDAP object definitions from [RFC7483] do not contain any mandatory attributes. For the purposes of this protocol, each RDAP object MUST include an "rdapConformance" attribute, so that a client can determine whether the object is one that it is able to process.

Each RDAP object is paired with an identifier (the "id" attribute in the parent object). The "id" attribute value MUST be a URI ([RFC3986]). Its value uniquely identifies an RDAP object within the data set, so as to support internal linking and for subsequent removal of the object by a Delta File.

In each RDAP object, each link to an RDAP object that is a member of the data set for which the server is providing mirroring MUST have as the value of its "href" attribute the identifier for that object (i.e. the value of the "id" attribute from the target's parent object). This includes self-references (i.e. links with the "self" relation type).

If a server includes a link object with the "self" relation type with each of its RDAP objects, then using the value of the "href" attribute of that link object as the identifier for each RDAP object is RECOMMENDED.

                
{
    "version": 1,
    "serial": 5,
    "removed_objects": [],
    "added_or_updated_objects": [
        { "id": "https://example.org/E5",
          "object": { "rdapConformance": [ "rdap_level_0" ],
                      "objectClassName": "entity",
                      "handle": "E5",
                      "links": [
                          { "rel": "self",
                            "href": "https://example.org/E5",
                            ... }
                      ],
                      ... } },
        { "id": "https://example.org/I6",
          "object": { "rdapConformance": [ "rdap_level_0" ],
                      "objectClassName": "ip network",
                      "links": [
                          { "rel": "self",
                            "href": "https://example.org/I6",
                            ... }
                      ],
                      "entities": [
                          { "handle": "E5",
                            "links": [
                                { "rel": "self",
                                  "href": "https://example.org/E5",
                                  ... }
                            ],
                            ... }
                      ],
                      ... } }
        ...
    ]
}
                
                

For example, a Delta File containing an entity object, along with an IP network object that links to it:

A server MAY omit from an object data that it returns as part of its corresponding public service response, when that data can be determined by reference to another object in the data set. In such cases, the server MUST include a "links" attribute containing a link object with a "self" relation, so that the target object can be resolved by the client.

2.4. Serial Numbers

Serial numbers in the files defined in this protocol are unsigned 32-bit integers. For the purposes of this protocol, the serial number arithmetic defined in [RFC1982] applies.

2.5. Server Use

2.5.1. Initialization

If a server is publishing a Snapshot File via the Update Notification File, then initialization is like so:

• generate an initial Snapshot File, with a serial number selected by the server;
• sign the Snapshot File using JWS, and publish the result using JWS Compact Serialization at a URL that is unique to this serial number;
• generate an initial Update Notification File, with a serial number equal to that of the Snapshot File, and containing a link to the Snapshot File; and
• sign the Update Notification File using JWS, and publish the result using JWS Compact Serialization.

To avoid doubt, any RDAP object that is part of the data set for which the server is providing mirroring, as well as being the target of a link contained within another RDAP object, MUST be present within the Snapshot File.

If a server is not publishing a Snapshot File via the Update Notification File, then initialization is like so:

• generate an initial Snapshot File, with a serial number selected by the server;
• sign the Snapshot File using JWS, and distribute the result to clients out-of-band using JWS Compact Serialization;
• generate an initial Update Notification File, containing no Snapshot File link or Delta File links, with a serial number equal to that of the Snapshot File (i.e. the one that will be distributed out-of-band); and
• sign the Update Notification File using JWS, and publish the result using JWS Compact Serialization.

2.5.2. Publishing Updates

The server periodically publishes changes that have been made to its RDAP state as Delta Files. The timing and frequency of publication is a local policy matter for the server. The process is like so:

• if a Delta File has been generated previously: generate a new Delta File, containing the changes that have been made to the RDAP state since the last Delta File was generated, with a serial number that is one greater than the serial number of the last Delta File;
• if no Delta File has been generated previously: generate a new Delta File, containing the changes that have been made to the RDAP state since the last Snapshot File was generated, with a serial number that is one greater than the serial number of the last Snapshot File;
• sign the Delta File using JWS, and publish the result using JWS Compact Serialization at a URL that is unique to its serial number;
• take the currently-published Update Notification File, increment its serial number, add a link to the new Delta File, and optionally perform the steps described in the "Consolidation" section below; and
• sign the Update Notification File using JWS, and publish the result using JWS Compact Serialization.

Delta Files MUST NOT remove an RDAP object that would cause a relative reference link within the client's local state to become unresolvable.

2.5.3. Consolidation

On publishing an update, the server may optionally consolidate the Snapshot File and Delta Files that it is publishing. The process is like so:

• if the server is publishing a Snapshot File: generate a new Snapshot File based on the server's current state with a serial number equal to that of the new Delta File, publish the new Snapshot File, and replace the link in the Update Notification File to the previous Snapshot File with a link to the new Snapshot File; and
• remove Delta Files from the Update Notification File that have become stale.

Whether a given Delta File is 'stale' is a local policy matter for the server.

2.6. Client Use

2.6.1. Processing the Update Notification File

2.6.1.1. Initial

The client downloads the signed Update Notification File using the URL provided by the server (out-of-band) and validates the signature against the server's JWK.

The client validates the signature of the Snapshot File against the server's JWK, and then uses that file to initialize its local state by adding all of the objects from the "objects" attribute. It then records the serial number of the Snapshot File. The signed Snapshot File is either accessible from the Update Notification File, or made available to the client out-of-band.

The client then processes each Delta File from the Update Notification File in order, from the Delta File with a serial number one greater than the client's recorded serial number, through to the Delta File with the largest serial number, in order to update its local state.

Processing a Delta File involves three steps:

• verify the signature against the server's JWK;
• for each entry in the "removed_objects" attribute, remove from the local state any object with an "id" attribute value equal to the entry;
• for each entry in the "added_or_updated_objects" attribute: if an object with the given values for the "id" attribute exists in the local state, then replace that object with the new object; otherwise, add the new object to the local state.

Once this is complete, the client records the serial number of the last Delta File that it processed.

2.6.1.2. Subsequent

The client downloads the signed Update Notification File using the URL provided by the server (out-of-band), and validates the signature against the server's JWK. If the frequency with which the client should do this has been suggested by the server via the "refresh" attribute, the client SHOULD honor that suggestion. If the "refresh" attribute is not present, retrieval frequency is a local policy matter for the client.

The client then processes each Delta File from the Update Notification File in order, from the Delta File with a serial number one greater than that which has been recorded, through to the Delta File with the largest serial number. Processing of the Delta Files is otherwise as per the instructions for initial processing.

If the Update Notification File retrieved by the client does not contain a Delta File with a serial number one greater than that which has been recorded, the client MUST delete all of its local state and reinitialize itself. If the Update Notification File contains a Snapshot File, then that Snapshot File can be used for reinitialization. If it does not, then a new Snapshot File must be located out-of-band.

3. Operational Considerations

A server may omit previously-published Delta Files from its Update Notification File as a matter of local policy. If a server is publishing its Snapshot Files out-of-band, then omitting a Delta File that a client needs will result in the client needing to perform an out-of-band action in order to reinitialize its state. Even if a server is linking to its Snapshot Files from the Update Notification File, reinitialization may be an expensive operation for a client. Servers should consider adopting local policy that limits the chance of reinitialization happening: for example, by using the "refresh" attribute value in the Update Notification File.

4. Security Considerations

[RFC7481] describes security requirements and considerations for RDAP generally. Those requirements and considerations also apply to the use of this protocol.

This protocol requires the use of JWS ([RFC7515]) and JWK ([RFC7517]), which in turn refer to JSON Web Algorithms (JWA) ([RFC7518]). Implementations MUST support ES256 as defined in JWA ([RFC7518], section 3.4) for signing and validating files in this protocol. Implementations MAY support other algorithms from the "JSON Web Signature and Encryption Algorithms" registry created by [RFC7518].

5. Acknowledgements

This protocol is largely a repurposing of the RPKI Repository Delta Protocol (RRDP) for RDAP. Much of the terminology (e.g. Update Notification File, Snapshot File, Delta File) is taken from that document, and the structure is also quite similar.

Experience with the Near Real Time Mirroring (NRTM) protocol, which serves a similar purpose for databases that are based on the Routing Policy Specification Language (RPSL), helped to inform this corresponding effort in RDAP.

6. IANA Considerations

TBD

7. References

7.1. Normative References

7.2. Informative References

Authors' Addresses