Guidelines for YANG Example Validation and Coverage Analysis in IETF Documents

Abstract

This document defines guidelines for including YANG example instances in IETF documents in a way that enables automatic extraction and validation. It introduces the concept of YANG module "coverage" to measure how thoroughly example instances exercise a YANG module's data nodes. The goal is to improve the quality and usability of YANG models in IETF documents by providing authors with tools to validate their examples and assess coverage completeness.¶

1. Introduction

Although the YANG data modeling language has been in use for over a decade, there remains a lack of tooling to help YANG model authors analyze and validate their YANG modules and example instances. People defining or evaluating YANG modules have limited guidance on how to include and test example data for those modules. Example data instances are vital to illustrate how a model is intended to be used, but the guidelines for presenting and validating these examples are scarce. The updated guidelines in RFC 8407bis improve the situation, but gaps still exist in providing tools that can automatically give authors feedback on what they might be missing in their models.¶

The objective of this document is to define guidelines that enable the development of tools which, given one or more IETF documents (RFCs or drafts), can verify that the YANG model examples in those documents are correct and sufficiently comprehensive. To achieve this, we address two main aspects:¶

• Guidelines for Including YANG Examples in RFCs: A clear method for authors to include example data instances of YANG models in IETF documents in a way that tools can automatically extract and validate them. In addition, we outline how tools can perform this extraction and validation, and we propose a method to include extra YANG modules (when necessary) for validating more complex examples without affecting the main normative content.¶

• Definition of YANG Module "Coverage": A new concept to measure how much of a YANG module's data nodes are exercised by the example instances in a document. By analyzing the examples, a tool can inform authors which parts of their YANG model are covered and which are still missing from the examples. A proof-of-concept implementation for computing such coverage is also considered.¶

Scope: The proposals in this document focus on typical use cases. It is expected that proof-of-concept tools will confirm the viability of these methods for moderately complex examples as this document progresses. If such automatic validation proves infeasible in practice, the approach may need adjustment. Extremely complex example scenarios (e.g., those involving schema mount or highly dynamic state) are outside the current scope and could be addressed in future work.¶

2. Including YANG Example Instances in IETF Documents

When authors include example YANG instances in an RFC or Internet-Draft, those example instances SHOULD be formatted in a way that allows automatic extraction and validation by tools.¶

Specifically, example instances SHOULD be enclosed as code blocks using the CODE BEGINS / CODE ENDS convention (as defined in the updated guidelines, e.g., draft-ietf-netmod-rfc8407bis). They should also be given a file name that clearly identifies them as examples, analogous to how YANG modules are named (see RFC 8407bis guidelines). We propose that the file name ends with the suffix .instance.xml for XML instances or .instance.json for JSON instances. This naming convention makes it easy for tools to identify which code snippets are YANG instance examples that can be parsed and validated.¶

Each example file name SHOULD be unique within a document (and ideally unique across related documents being analyzed together). Tools may complain or error out if they encounter duplicate example file names, since that could indicate conflicting or repeated content. For example, an XML instance might be included as:¶

// Not an actual instance
<CODE BEGINS> file "interface-config.instance.xml"
<?xml version="1.0" encoding="UTF-8"?>
<data> ... </data>
<CODE ENDS>

And a JSON instance as:¶

// Not an actual instance
<CODE BEGINS> file "interface-config.instance.json"
{
  "interface": { ... }
}
<CODE ENDS>

By following this approach, a tool can scan the document for code blocks with filenames ending in .instance.xml or .instance.json, extract their contents, and validate them against the appropriate YANG modules.¶

In cases where an included code snippet is not meant to represent a valid YANG instance (for instance, it might be pseudocode, a partial fragment, or intentionally invalid to illustrate an error), the author can indicate this so that automated validators skip it. For example, an author might choose not to use the CODE BEGINS/CODE ENDS markers or might omit the special .instance.xml/.json filename suffix for such a snippet. Without the markers or filename, the tool will know not to treat that snippet as a YANG instance example to be validated. An author could also include an explicit note in the text indicating that a given snippet is not for validation. Using these conventions strikes a balance between making examples machine-checkable and allowing flexibility for authors to include non-valid illustrative snippets when necessary.¶

3. YANG Module Coverage by Examples

Providing examples in YANG model documents is not just helpful for readers, but can also improve the quality of the model itself. We introduce the concept of YANG module coverage as a way to measure how thoroughly the provided examples exercise the data model.¶

By analogy to code coverage in software testing (which checks what fraction of code is executed by tests), YANG module coverage looks at what fraction of a module's schema nodes are present in at least one of the example instances. More specifically, we can define coverage as the set of data nodes (particularly leaf nodes) from the YANG module that have a value in one or more example instances. A simple quantitative metric could be the percentage of the module's leaf nodes that appear in at least one example.¶

For instance, if a YANG module defines 100 configurable or state leaf nodes, and the examples collectively include 45 of those leaves, one could say the example coverage is 45%. This number, while not telling the whole story, gives an indication of how much of the model's functionality is demonstrated by the examples. Low coverage might suggest that important parts of the model are never shown in any example. This could mean those parts are either less relevant or simply that the document might benefit from additional examples to cover them.¶

Probably more helpful than the coverage percentage, we can also define a "non-coverage tree" - essentially a view of the data model highlighting the nodes that were not covered by any example. This is perhaps the most direct way of showing the authors which portions of their model remain unused in the examples. For example, a tool could output a tree or list of all leaf nodes not present in any example instance, indicating potential gaps in example coverage.¶

This kind of feedback can help authors identify gaps and improve their documents. If a particular feature or branch of the model has no example, the authors might choose to add one for completeness or clarity. In some cases, it might reveal that certain parts of the model are hard to exemplify or not yet implemented, which could prompt re-thinking that part of the design or at least adding an explanation for why it's not covered by any example.¶

Coverage works directly on the data nodes from the YANG module. It should be as simple as possible, therefore, we will consider that any data leaf under a list is covered if there is at least an example with an instance of the list that includes the leaf. That is, for a list with 2 different leafs, one leaf covered under one key and another under another one will still yield a 100% coverage of the list.¶

module example-device {
  namespace "http://example.com/device";
  prefix "dev";

  container device {
    leaf hostname { type string; }
    leaf location { type string; }
    leaf admin-state { type boolean; }

    list interface {
      key "name";
      leaf name { type string; }
      leaf enabled { type boolean; }
      leaf other { type boolean; }
    }
  }
}

<CODE BEGINS> file "device-config.instance.xml"

<device xmlns="http://example.com/device">
  <hostname>router1</hostname>
  <interface>
    <name>eth0</name>
    <enabled>true</enabled>
  </interface>
</device>

<CODE ENDS>

device
├── location
├── admin-state
└── interface
    └── other

4. Implementation

TODO. The progress of this document depends on the correct operation of its proposal in cases of relative complexity using an actual implementation.¶

5. Conclusion

By standardizing how YANG examples are included and validated, we make it easier for implementers and reviewers to trust that the examples in a specification are correct and that a sufficient range of scenarios is covered. The concept of YANG module coverage, while exploratory, can further encourage authors to include more comprehensive examples and help pinpoint which parts of a data model might need additional illustration. Together, these measures aim to improve the overall quality and usability of YANG models in IETF documents.¶

Ultimately, the success of these proposals will depend on tool support. As a next step, developing proof-of-concept tools or integrations (for example, enhancements to existing YANG validators like yanglint, or new scripts to extract and analyze examples) will be crucial. If moderately complex examples can be automatically extracted, validated, and measured for coverage with such tools, it will validate the approach. On the other hand, if significant obstacles arise during implementation, the recommendations in this document may need to be refined. Our hope is that these initial guidelines spark discussion and experimentation, leading to more robust support for YANG examples and better validation practices in the near future.¶

8. References