The Series Transfer Pattern (STP)Universität Bremen TZIPostfach 330440BremenD-28359Germany+49-421-218-63921cabo@tzi.orgEricssonTorshamnsgatan 23Stockholm16483Swedenklaus.hartke@ericsson.comInternet-DraftMany applications make use of Series of data items, i.e., an array of
data items where new items can be added over time. Where such Series
are to be made available using REST protocols such as CoAP or HTTP,
the Series has to be mapped into a structure of one or more resources
and a protocol for a client to obtain the Series and to learn about
new items.Various protocols have been standardized that make Series-shaped data
available, with rather different properties and objectives. The
present document is an attempt to extract a common underlying pattern
and to define media types and an access scheme that can be used right
away for further protocols that provide Series-shaped data.Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Task
Force (IETF). Note that other groups may also distribute working
documents as Internet-Drafts. The list of current Internet-Drafts is
at .
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 10 October 2020.
Copyright Notice
Copyright (c) 2020 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
() in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with
respect to this document. Code Components extracted from this
document must include Simplified BSD License text as described in
Section 4.e of the Trust Legal Provisions and are provided without
warranty as described in the Simplified BSD License.
Table of Contents
. Introduction
. Objectives
. A REST Series Transfer Pattern (STP)
. Basic collections
. Pagination and Observing linked lists
. The "cursor" pattern
. IANA considerations
. Security considerations
. Informative References
Acknowledgements
Authors' Addresses
Introduction(TO DO: Insert an extended form of the abstract first here, expanding
the reference to and in the process.)Examples for protocols that provide Series-shaped data are:
The Atom Syndication Format defines feeds as Series
of entries (links plus some metadata, which can often be much of
the content of an entry), where a feed is represented by a
collection resource that contains just a small number of the most
recent entries. By polling a feed, a client can contain a fresh
view of the Series, with a focus on recent items. If the client
does not poll often enough, it will miss items.
Messaging protocols such as XMPP or SIMPLE transfer series of what
is often called "Instant Messages". A publish/subscribe mechanism
allows a client to select sequences of messages that it is
interested in.
Mail servers that provide interactive access to stored messages
present a Series to their clients. Obviously, loss of messages is
frowned upon.
CoAP Observe allows a client to observe a resource as it changes;
the client can collect the changes into a Series. Observe is
focused on eventual consistency: a fresher data items simply
overwrites an older one. The present document uses the observe
pattern to build a more general Series Transfer Pattern.
Syslog is an interesting case of a Series Transfer.
,
,
,
,
.
An RTP stream can be viewed as an (somewhat extreme) case of a
Series; new items are just sent inside separate UDP packets. A
sequence number allows to detect (but not normally ask for
retransmission of) missing items. A timestamp as well as source
data (SSRC, CSRC) provide further common metadata that aid in the
processing of the Series items.
An example of an ad-hoc design of a series transfer protocol is
.
Server-sent events are a somewhat bizarre version of a series
transfer protocol.
The Interface for Metadata Access Points (IF-MAP) specified by the
Trusted Computing Group and emerging derivatives of that protocol
create a series of updates to a graph representation of related
network-related security information. The requests created by
IF-MAP clients are bundled operations of updates to a
MAP Graph, which compose a Series Transfer Pattern of bundled atomic
operations that ensure the integrity of the MAP Graph. [Henk Birkholz]
netflow/IPFIX was defined to transfer a series of data items about flows.
Information about PDU flows accounted by network interfaces of
endpoints is emitted in a series of counter bundles via the IPFIX
protocol. Only a series of these continuous Flow Records creates a
meaningful bigger picture about the current traffic in the network
topology of an administrative domain. Depending on the characteristics
measured, loss of a Flow Record can range from harmless to missing the
only vital counter measurement. [Henk Birkholz]
TO DO: Add more items.
is a problem
statement that will require the design of another scheme to transfer
Series-shaped data.ObjectivesSeries transfer applications may have rather different objectives.
The completeness of the Series transfer may be of utmost importance
(e.g., if each item represents a sale), it may be desirable but can
be jettisoned in an overload situation, or it may just be a likely
outcome with a very active client (e.g., with Atom). Note that
there is never a way to guarantee completeness unless all of the
rate and size of incoming new items, the transfer capacity
available, and the processing capabilities of the client are
controlled; however, system designs may want to give the illusion of
"reliability".
Minimizing the latency of the transfer may be important, as may be
limiting it below a defined maximum (note that these are different
objectives). The latter can be supported in a polling system by
polling at least as often as that maximum latency; this may be
considered inefficient and "push" mechanisms may be developed. Mail
environments have developed "push" services to enable minimizing the
latency. Where latency requirements go below the time that might be
needed for an end-to-end retransmission, error concealment may
provide an acceptable user experience (e.g., in RTP).
In general, minimizing latency and ensuring completeness are competing
objectives.Series transfer environments sometimes centralize information
distribution functions, leading to "broker" architectures (often
combined with the "publish/subscribe" pattern). With brokers, Series
publishers may use an entirely different interface to the brokers from
that used by the receiving clients, or the interfaces can be designed
so they are similar for all the forwarding steps.A REST Series Transfer Pattern (STP)Basic collectionsA series of items can be represented by a single collection resource:While this is adequate in many cases, it has a number of limitations:
Each retrieval fetches the entire collection
As long as the collection does not change, this can be mitigated
with ETags (Section 5.10.6 of, Section 2.3 of ).
When the collection becomes too large, the server has to prune older
items. These then no longer can be retrieved, and there is even no
way for the server to indicate that there used to be older items.
Pagination and Observing linked listsIn the Browser Web, it is usual to provide Pagination for collection
resources that can grow large (e.g., search results):Without modification, this does not work well for resources that
actually change by themselves: Once a new page needs to be added,
what previously was page 1 now becomes page 2. Obviously, the naming
of pages better remains unchanged with new pages added a the front.However, now the client has no idea what initial page to request to
get the freshest items and the head of the list. It is easy to add a
link to the freshest page:The head of the linked list can now be simply observed; the addition
of pages will then be notified to the observer.As usual in series transfer, the following considerations remain:
When can the server decide to no longer retain older items?
There may be a desire for an observer to be able to catch all
items in the series.
How does the server know who are the observers? E.g., what to
do with newly joining observers?
How does an observer signal that it has caught up (to a specific
item)?
What to do when the decision to remove items from the list cannot be
made and there is no room for new items?
The link head can also include items that have so far not been added
to pages; this can be used to fill up pages evenly without them ever
changing. Obviously, the best number of items to prenotify in this
way as well as the best time to open a new page are different for
different applications.The "cursor" patternA GET on a resource representing a Series may return a collection item
that contains the following pieces of information
An array of Series items, either as an array of media-typed objects
in a suitable representation format (e.g., CBOR, MIME) or by using
an array-like media type (e.g., SenML).
Items may be full items or limit themselves to some metadata and
a link; the client can then follow that link if it is interested
in the data (possibly basing that decision on the metadata and/or
a measure of load).
A "cursor" that can then be used as a parameter in further GET
requests (see below) in order to receive only newer items than those
received with this transfer.
A "more bit" that indicates whether such further items already exist
but could not be returned in the present response.
In , the cursor is implemented as a URI that can be used
as a link to the next page.A GET may be enhanced with additional parameters (possibly turning it
into a FETCH):
The cursor.
A "wait bit" that indicates whether a (possibly empty) reply should
be given right away or the server should wait for new items to
become available. (To avoid the equivalence of the "silly window
syndrome", the wait bit may be enhanced by a minimum number of items
and a timeout after which even a smaller number is made available.)
In effect, this requests a form of "long polling"; see
for some considerations for this in HTTP.
A server may implement a form of custody transfer by interpreting the
cursor as an acknowledgement that the client has received all data up
to the cursor. This is not necessarily acting as an unsafe request
("destructive GET"), as other clients may be active that have not yet
received all these data. To implement a full custody semantics, the
server needs to be aware of all the clients that expect a full Series
Transfer (a classical group management problem).(Explain how Observe can help. Can it?)IANA considerationsThis memo registers a number of media types: TO DO.
A media type for FETCH selectors ():
An alternative way to encode this information into the URI
of a GET should also be available.
A Series media type as alluded to in .
Security considerationsTO DOInformative ReferencesYANG Push Operations for CoMIThis document provides a problem statement, derives an initial gap analysis and illustrates a first set of solution approaches in regard to augmenting YANG data stores based on the CoAP Management Interface with YANG Push capabilities. A binary transfer mechanism for YANG Subscribed Notifications addresses both the requirements of constrained-node networks and the need for semantic interoperability via self-descriptiveness of the corresponding data in motion.Work in ProgressNotification Message Headers and BundlesThis document defines a new notification message format. Included are: o a new notification mechanism and encoding to replace the one way operation of RFC-5277 o a set of common, transport agnostic message header objects. o how to bundle multiple event records into a single notification message. o how to ensure these new capabilities are only used with capable receivers.Work in ProgressUDP based Publication Channel for Streaming TelemetryThis document describes a UDP-based publication channel for streaming telemetry use to collect data from devices. A new shim header is proposed to facilitate the distributed data collection mechanism which directly pushes data from line cards to the collector. Because of the lightweight UDP encapsulation, higher frequency and better transit performance can be achieved.Work in ProgressYANG Notification Headers and BundlesThere are useful capabilities not available with existing YANG notifications as described in Section 7.16 of [RFC7950]. These include:Work in ProgressThe Atom Syndication FormatThis document specifies Atom, an XML-based Web content and metadata syndication format. [STANDARDS-TRACK]Known Issues and Best Practices for the Use of Long Polling and Streaming in Bidirectional HTTPOn today's Internet, the Hypertext Transfer Protocol (HTTP) is often used (some would say abused) to enable asynchronous, "server- initiated" communication from a server to a client as well as communication from a client to a server. This document describes known issues and best practices related to such "bidirectional HTTP" applications, focusing on the two most common mechanisms: HTTP long polling and HTTP streaming. This document is not an Internet Standards Track specification; it is published for informational purposes.Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and RoutingThe Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document provides an overview of HTTP architecture and its associated terminology, defines the "http" and "https" Uniform Resource Identifier (URI) schemes, defines the HTTP/1.1 message syntax and parsing requirements, and describes related security concerns for implementations.Hypertext Transfer Protocol (HTTP/1.1): Conditional RequestsThe Hypertext Transfer Protocol (HTTP) is a stateless application- level protocol for distributed, collaborative, hypertext information systems. This document defines HTTP/1.1 conditional requests, including metadata header fields for indicating state changes, request header fields for making preconditions on such state, and rules for constructing the responses to a conditional request when one or more preconditions evaluate to false.The Constrained Application Protocol (CoAP)The Constrained Application Protocol (CoAP) is a specialized web transfer protocol for use with constrained nodes and constrained (e.g., low-power, lossy) networks. The nodes often have 8-bit microcontrollers with small amounts of ROM and RAM, while constrained networks such as IPv6 over Low-Power Wireless Personal Area Networks (6LoWPANs) often have high packet error rates and a typical throughput of 10s of kbit/s. The protocol is designed for machine- to-machine (M2M) applications such as smart energy and building automation.CoAP provides a request/response interaction model between application endpoints, supports built-in discovery of services and resources, and includes key concepts of the Web such as URIs and Internet media types. CoAP is designed to easily interface with HTTP for integration with the Web while meeting specialized requirements such as multicast support, very low overhead, and simplicity for constrained environments.Subscription to YANG NotificationsThis document defines a YANG data model and associated mechanisms enabling subscriber-specific subscriptions to a publisher's event streams. Applying these elements allows a subscriber to request and receive a continuous, customized feed of publisher-generated information.Subscription to YANG Notifications for Datastore UpdatesThis document describes a mechanism that allows subscriber applications to request a continuous and customized stream of updates from a YANG datastore. Providing such visibility into updates enables new capabilities based on the remote mirroring and monitoring of configuration and operational state.Dynamic Subscription to YANG Events and Datastores over RESTCONFThis document provides a RESTCONF binding to the dynamic subscription capability of both subscribed notifications and YANG-Push.HTML Living Standard -- 9.2 Server-sent eventsWHATWGn.d.AcknowledgementsThe need for a Series Transfer Pattern has been made clear by a number
of people that contribute to the IRTF Thing-to-Thing Research Group
(T2TRG), e.g. Matthias Kovatsch and Henk Birkholz (both of whom also
provided feedback on an early draft). Henk also contributed further
examples for the use of Series Transfers in protocols.Authors' AddressesUniversität Bremen TZIPostfach 330440BremenD-28359Germany+49-421-218-63921cabo@tzi.orgEricssonTorshamnsgatan 23Stockholm16483Swedenklaus.hartke@ericsson.com