draft-ietf-opsawg-collected-data-manifest-02.txt		draft-ietf-opsawg-collected-data-manifest-03.txt
	OPSAWG B. Claise		OPSAWG B. Claise
	Internet-Draft J. Quilbeuf		Internet-Draft J. Quilbeuf
	Intended status: Standards Track Huawei		Intended status: Standards Track Huawei
	Expires: April 25, 2024 D. Lopez		Expires: September 2, 2024 D. Lopez
	I. Dominguez		I. Dominguez
	Telefonica I+D		Telefonica I+D
	T. Graf		T. Graf
	Swisscom		Swisscom
	October 23, 2023		March 1, 2024
	A Data Manifest for Contextualized Telemetry Data		A Data Manifest for Contextualized Telemetry Data
	draft-ietf-opsawg-collected-data-manifest-02		draft-ietf-opsawg-collected-data-manifest-03
	Abstract		Abstract
	Network elements use Model-driven Telemetry, and in particular YANG-		Network platforms use Model-driven Telemetry, and in particular YANG-
	Push, to continuously stream information, including both counters and		Push, to continuously stream information, including both counters and
	state information. This document documents the metadata that ensure		state information. This document documents the metadata that ensure
	that the collected data can be interpreted correctly. This document		that the collected data can be interpreted correctly. This document
	specifies the Data Manifest, composed of two YANG data models (the		specifies the Data Manifest, composed of two YANG data models (the
	Platform Manifest and the Data Collection Manifest). The Data		Platform Manifest and the Data Collection Manifest). These YANG
			modules are specified at the network (i.e. controller) level to
			provide a model that encompasses several network platforms. The Data
	Manifest must be streamed and stored along with the data, up to the		Manifest must be streamed and stored along with the data, up to the
	collection and analytics system in order to keep the collected data		collection and analytics system in order to keep the collected data
	fully exploitable by the data scientists.		fully exploitable by the data scientists.
	Status of This Memo		Status of This Memo
	This Internet-Draft is submitted in full conformance with the		This Internet-Draft is submitted in full conformance with the
	provisions of BCP 78 and BCP 79.		provisions of BCP 78 and BCP 79.
	Internet-Drafts are working documents of the Internet Engineering		Internet-Drafts are working documents of the Internet Engineering
	Task Force (IETF). Note that other groups may also distribute		Task Force (IETF). Note that other groups may also distribute
	working documents as Internet-Drafts. The list of current Internet-		working documents as Internet-Drafts. The list of current Internet-
	Drafts is at https://datatracker.ietf.org/drafts/current/.		Drafts is at https://datatracker.ietf.org/drafts/current/.
	Internet-Drafts are draft documents valid for a maximum of six months		Internet-Drafts are draft documents valid for a maximum of six months
	and may be updated, replaced, or obsoleted by other documents at any		and may be updated, replaced, or obsoleted by other documents at any
	time. It is inappropriate to use Internet-Drafts as reference		time. It is inappropriate to use Internet-Drafts as reference
	material or to cite them other than as "work in progress."		material or to cite them other than as "work in progress."
	This Internet-Draft will expire on April 25, 2024.		This Internet-Draft will expire on September 2, 2024.
	Copyright Notice		Copyright Notice
	Copyright (c) 2023 IETF Trust and the persons identified as the		Copyright (c) 2024 IETF Trust and the persons identified as the
	document authors. All rights reserved.		document authors. All rights reserved.
	This document is subject to BCP 78 and the IETF Trust's Legal		This document is subject to BCP 78 and the IETF Trust's Legal
	Provisions Relating to IETF Documents		Provisions Relating to IETF Documents
	(https://trustee.ietf.org/license-info) in effect on the date of		(https://trustee.ietf.org/license-info) in effect on the date of
	publication of this document. Please review these documents		publication of this document. Please review these documents
	carefully, as they describe your rights and restrictions with respect		carefully, as they describe your rights and restrictions with respect
	to this document. Code Components extracted from this document must		to this document. Code Components extracted from this document must
	include Simplified BSD License text as described in Section 4.e of		include Simplified BSD License text as described in Section 4.e of
	the Trust Legal Provisions and are provided without warranty as		the Trust Legal Provisions and are provided without warranty as
	described in the Simplified BSD License.		described in the Simplified BSD License.
	Table of Contents		Table of Contents
	1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2		1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
	1.1. Use Cases . . . . . . . . . . . . . . . . . . . . . . . . 4		1.1. Use Cases . . . . . . . . . . . . . . . . . . . . . . . . 4
	1.1.1. Network Analytics . . . . . . . . . . . . . . . . . . 4		1.1.1. Network Analytics . . . . . . . . . . . . . . . . . . 4
	1.1.2. New Device Onboarding . . . . . . . . . . . . . . . . 5		1.1.2. New Device Onboarding . . . . . . . . . . . . . . . . 5
	1.1.3. Data Mesh Principles in Networking . . . . . . . . . 5		1.1.3. Data Mesh Principles in Networking . . . . . . . . . 5
	2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 6		2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 6
	3. Platform Manifest . . . . . . . . . . . . . . . . . . . . . . 6		3. Platform Manifest . . . . . . . . . . . . . . . . . . . . . . 6
	3.1. Overview of the Model . . . . . . . . . . . . . . . . . . 6		3.1. Overview of the Model . . . . . . . . . . . . . . . . . . 6
	3.2. YANG module ietf-platform-manifest . . . . . . . . . . . 8		3.2. YANG module ietf-platform-manifest . . . . . . . . . . . 8
	4. Data Collection Manifest . . . . . . . . . . . . . . . . . . 14		4. Data Collection Manifest . . . . . . . . . . . . . . . . . . 14
	4.1. Overview of the Model . . . . . . . . . . . . . . . . . . 14		4.1. Overview of the Model . . . . . . . . . . . . . . . . . . 14
	4.2. YANG module ietf-data-collection-manifest . . . . . . . . 16		4.2. YANG module ietf-data-collection-manifest . . . . . . . . 16
	5. Data Manifest and the Collected Data . . . . . . . . . . . . 22		5. Data Manifest and the Collected Data . . . . . . . . . . . . 23
	5.1. Collecting the Data Manifest . . . . . . . . . . . . . . 23		5.1. Collecting the Data Manifest . . . . . . . . . . . . . . 23
	5.2. Mapping Collected Data to the Data Manifest . . . . . . . 24		5.2. Mapping Collected Data to the Data Manifest . . . . . . . 24
	5.3. Operational Considerations . . . . . . . . . . . . . . . 24		5.3. Operational Considerations . . . . . . . . . . . . . . . 24
	6. Example . . . . . . . . . . . . . . . . . . . . . . . . . . . 25		6. Example . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
	7. Security Considerations . . . . . . . . . . . . . . . . . . . 27		7. Security Considerations . . . . . . . . . . . . . . . . . . . 27
	8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 27		8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 27
	9. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 27		9. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 27
	10. Open Issues . . . . . . . . . . . . . . . . . . . . . . . . . 27		10. Open Issues . . . . . . . . . . . . . . . . . . . . . . . . . 27
	11. References . . . . . . . . . . . . . . . . . . . . . . . . . 28		11. References . . . . . . . . . . . . . . . . . . . . . . . . . 28
	11.1. Normative References . . . . . . . . . . . . . . . . . . 28		11.1. Normative References . . . . . . . . . . . . . . . . . . 28
	11.2. Informative References . . . . . . . . . . . . . . . . . 29		11.2. Informative References . . . . . . . . . . . . . . . . . 29
	Appendix A. Example of use based on MDT . . . . . . . . . . . . 30		Appendix A. Example of use based on MDT . . . . . . . . . . . . 30
	Appendix B. Changes between revisions . . . . . . . . . . . . . 32		Appendix B. Changes between revisions . . . . . . . . . . . . . 33
	Appendix C. YANG module ietf-yang-push-modif . . . . . . . . . . 34		Appendix C. YANG module ietf-yang-push-modif . . . . . . . . . . 35
	Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 50		Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 52
	Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 51		Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 52
	1. Introduction		1. Introduction
	Network elements use Model-driven Telemetry (MDT), and in particular		Network platforms use Model-driven Telemetry (MDT), and in particular
	YANG-Push [RFC8641], to continuously stream information, including		YANG-Push [RFC8641], to continuously stream information, including
	both counters and state information.		both counters and state information.
	This document specifies what needs to be kept as metadata (i.e., the		This document specifies what needs to be kept as metadata (i.e., the
	Data Manifest) to ensure that the collected data can still be		Data Manifest) to ensure that the collected data can still be
	interpreted correctly throughout the collection and network analytics		interpreted correctly throughout the collection and network analytics
	toolchain. When streaming YANG-structured data with YANG-Push		toolchain. When streaming YANG-structured data with YANG-Push
	[RFC8641], there is a semantic definition in the corresponding YANG		[RFC8641], there is a semantic definition in the corresponding YANG
	module definition. This is the semantic information for the		module definition. This is the semantic information for the
	collected objects: While this semantic is absolutely required to		collected data nodes: While this semantic is absolutely required to
	correctly decode and interpret the data, understanding the network		correctly decode and interpret the data, understanding the network
	element and collection environment contexts information is equally		platform and collection environment contexts information is equally
	important to interpret the data.		important to interpret the data.
	This document proposes the Data Manifest, which is composed of two		This document proposes the Data Manifest, which is composed of two
	YANG data models, namely, the Platform Manifest and the Data		YANG data models, namely, the Platform Manifest and the Data
	Collection Manifest, in order to keep the collected data exploitable		Collection Manifest, in order to keep the collected data exploitable
	by the data scientists.		by the data scientists.
	The Platform Manifest contains information characterizing the		The Platform Manifest contains information characterizing the
	platform streaming the telemetry information, while the the Data		platform streaming the telemetry information, while the the Data
	Collection Manifest contains the required information to characterize		Collection Manifest contains the required information to characterize
	how and when the telemetry information was metered.		how and when the telemetry information was metered.
	The two proposed YANG modules in the Data Manifest do not expose many		The two proposed YANG modules in the Data Manifest do not expose many
	new information but rather define what should be exposed by a		new information but rather define what should be exposed by a
	platform streaming telemetry. Some related YANG modules have been		platform streaming or storing telemetry data. Some related YANG
	specified to retrieve the platform capabilities:		modules have been specified to retrieve the platform capabilities:
	o The IETF YANG Library [RFC8525].		o The IETF YANG Library [RFC8525].
	o YANG Modules Describing Capabilities for Systems and Datastore		o YANG Modules Describing Capabilities for Systems and Datastore
	Update Notifications [RFC9196] for the platform capabilities		Update Notifications [RFC9196] for the platform capabilities
	regarding the production and export of telemetry data.		regarding the production and export of telemetry data.
	o [I-D.claise-netconf-metadata-for-collection], which is based on		o [I-D.claise-netconf-metadata-for-collection], which is based on
	the previous draft to define the optimal settings to stream		the previous draft to define the optimal settings to stream
	specific items (i.e., per path).		specific items (i.e., per path).
	skipping to change at page 3, line 47 ¶		skipping to change at page 4, line 4 ¶
	o [I-D.claise-netconf-metadata-for-collection], which is based on		o [I-D.claise-netconf-metadata-for-collection], which is based on
	the previous draft to define the optimal settings to stream		the previous draft to define the optimal settings to stream
	specific items (i.e., per path).		specific items (i.e., per path).
	These related YANG modules are important to discover the capabilities		These related YANG modules are important to discover the capabilities
	before applying the telemetry configuration (such as on-change).		before applying the telemetry configuration (such as on-change).
	Some of their content is part of the context for the streamed data.		Some of their content is part of the context for the streamed data.
	We first present the module for the Platform Manifest in Section 3		We first present the module for the Platform Manifest in Section 3
	and then the module for the Data Collection Manifest in Section 4.		and then the module for the Data Collection Manifest in Section 4.
	The full Data Manifest is obtained by combining these two modules.		The full Data Manifest is obtained by combining these two modules.
	We explain in Section 5 how the Data Manifest can be retrieved and		We explain in Section 5 how the Data Manifest can be retrieved and
	how collected data is mapped to the Data Manifest.		how collected data is mapped to the Data Manifest.
	1.1. Use Cases		1.1. Use Cases
	1.1.1. Network Analytics		1.1.1. Network Analytics
	Streamed information from network elements is used for network		Streamed information from network platforms is used for network
	analytics, incident detections, and in the end closed-loop		analytics, incident detections, and in the end closed-loop
	automation. This streamed data can be stored in a database		automation. This streamed data can be stored in a database
	(sometimes called a big data lake) for further analysis.		(sometimes called a big data lake) for further analysis.
	As an example, a database could store a time series representing the		As an example, a database could store a time series representing the
	evolution of a specific counter collected from a network element.		evolution of a specific counter collected from a network platform.
	When analyzing the data, the network operator/data scientist must		When analyzing the data, the network operator/data scientist must
	understand the context information for these data:		understand the context information for these data:
	o This object definition in the YANG model.		o This counter definition in the YANG model.
	o The network element specific vendor, platform, and OS.		o The network platform specific vendor, model, and OS.
	o The collection parameters.		o The collection parameters.
	Characterizing the source used for producing the data (vendor,		Characterizing the source used for producing the data (vendor,
	platform, and OS) is useful to complement the data. As an example,		platform, and OS) is useful to complement the data. As an example,
	knowing the exact data source software specification might reveal a		knowing the exact data source software specification might reveal a
	particularity in the observed data, explained by a specific bug, a		particularity in the observed data, explained by a specific bug, a
	specific bug fix, or simply a particular specific behavior. This is		specific bug fix, or simply a particular specific behavior. This is
	also necessary to ensure the reliability of the collected data. On		also necessary to ensure the reliability of the collected data. On
	top of that, in particular for YANG-Push [RFC8641], it is crucial to		top of that, in particular for YANG-Push [RFC8641], it is crucial to
	know the set of YANG modules supported by the platform, along with		know the set of YANG modules supported by the platform, along with
	their deviations. In some cases, there might even be some backwards		their deviations. In some cases, there might even be some backwards
	incompatible changes in native modules between one OS version to the		incompatible changes in native modules between one OS version to the
	next one. This information is captured by the proposed Platform		next one. This information is captured by the proposed Platform
	Manifest.		Manifest.
	From a collection parameters point of view, the data scientists		From a collection parameters point of view, the data scientists
	analyzing the collected data must know that the counter was requested		analyzing the collected data must know that the counter was requested
	from the network element as on-change or at specific cadence.		from the network platform as on-change or at specific cadence.
	Indeed, an on-change collection explains why there is a single value		Indeed, an on-change collection explains why there is a single value
	as opposed to a time series. In case of periodic collection, this		as opposed to a time series. In case of periodic collection, this
	exact cadence might not be observable in the time series. Indeed,		exact cadence might not be observable in the time series. Indeed,
	this time series might report some values as 0 or might even omit		this time series might report some values as 0 or might even omit
	some values. The reason for this behavior might be diverse: the		some values. The reason for this behavior might be diverse: the
	network element was under stress, with a too small observation		network platform was under stress, with a too small observation
	period, compared to the minimum-observed-period		period, compared to the minimum-observed-period
	[I-D.claise-netconf-metadata-for-collection]. Again, knowing the		[I-D.claise-netconf-metadata-for-collection]. Again, knowing the
	conditions under which the counter was collected and streamed (along		conditions under which the counter was collected and streamed (along
	with the platform details) help drawing the right conclusions. As an		with the platform details) help drawing the right conclusions. As an
	example, taking into account the value of 0 might lead to a wrong		example, taking into account the value of 0 might lead to a wrong
	conclusion that the counter dropped to zero. This document specifies		conclusion that the counter dropped to zero. This document specifies
	the Data Collection Manifest, which contains the required information		the Data Collection Manifest, which contains the required information
	to characterize how and when the telemetry information was metered.		to characterize how and when the telemetry information was metered.
	The goal of the current document is to define what needs to be kept		The goal of the current document is to define what needs to be kept
	skipping to change at page 9, line 6 ¶		skipping to change at page 9, line 9 ¶
	organization		organization
	"IETF OPSAWG (Network Configuration) Working Group";		"IETF OPSAWG (Network Configuration) Working Group";
	contact		contact
	"WG Web: <https://datatracker.ietf.org/wg/opsawg/>		"WG Web: <https://datatracker.ietf.org/wg/opsawg/>
	WG List: <mailto:opsawg@ietf.org>		WG List: <mailto:opsawg@ietf.org>
	Author: Benoit Claise <mailto:benoit.claise@huawei.com>		Author: Benoit Claise <mailto:benoit.claise@huawei.com>
	Author: Jean Quilbeuf <mailto:jean.quilbeuf@huawei.com>		Author: Jean Quilbeuf <mailto:jean.quilbeuf@huawei.com>
	Author: Diego R. Lopez <diego.r.lopez@telefonica.com>		Author: Diego R. Lopez <diego.r.lopez@telefonica.com>
	Author: Ignacio Dominguez		Author: Ignacio Dominguez
	<ignacio.dominguezmartinez@telefonica.com>		<ignacio.dominguezmartinez@telefonica.com>
	Author: Thomas Grapf <thomas.graf@swisscom.com>";		Author: Thomas Graf <thomas.graf@swisscom.com>";
	description		description
	"This module describes the platform information to be used as		"This module describes the platform information to be used as
	context of data collection from a given network element. The		context of data collection from a given network element. The
	contents of this model must be streamed along with the data		contents of this model must be streamed along with the data
	streamed from the network element so that the platform context		streamed from the network element so that the platform context
	of the data collection can be retrieved later.		of the data collection can be retrieved later.
	The data content of this model should not change except on		The data content of this model should not change except on
	upgrade or patching of the device.		upgrade or patching of the device.
	skipping to change at page 17, line 33 ¶		skipping to change at page 17, line 36 ¶
	organization		organization
	"IETF OPSAWG (Network Configuration) Working Group";		"IETF OPSAWG (Network Configuration) Working Group";
	contact		contact
	"WG Web: <https://datatracker.ietf.org/wg/opsawg/>		"WG Web: <https://datatracker.ietf.org/wg/opsawg/>
	WG List: <mailto:opsawg@ietf.org>		WG List: <mailto:opsawg@ietf.org>
	Author: Benoit Claise <mailto:benoit.claise@huawei.com>		Author: Benoit Claise <mailto:benoit.claise@huawei.com>
	Author: Jean Quilbeuf <mailto:jean.quilbeuf@huawei.com>		Author: Jean Quilbeuf <mailto:jean.quilbeuf@huawei.com>
	Author: Diego R. Lopez <diego.r.lopez@telefonica.com>		Author: Diego R. Lopez <diego.r.lopez@telefonica.com>
	Author: Ignacio Dominguez		Author: Ignacio Dominguez
	<ignacio.dominguezmartinez@telefonica.com>		<ignacio.dominguezmartinez@telefonica.com>
	Author: Thomas Grapf <thomas.graf@swisscom.com>";		Author: Thomas Graf <thomas.graf@swisscom.com>";
	description		description
	"This module describes the context of data collection from a		"This module describes the context of data collection from a
	given network element. The contents of this model must be		given network element. The contents of this model must be
	streamed along with the data streamed from the network		streamed along with the data streamed from the network
	element so that the context of the data collection can		element so that the context of the data collection can
	be retrieved later.		be retrieved later.
	This module must be completed with		This module must be completed with
	ietf-platform-manifest		ietf-platform-manifest
	to capture the whole context of a data collection session.		to capture the whole context of a data collection session.
	skipping to change at page 23, line 48 ¶		skipping to change at page 23, line 51 ¶
	We propose to reuse the existing telemetry system (in-band approach)		We propose to reuse the existing telemetry system (in-band approach)
	in order to lower the efforts for implementing this draft. To enable		in order to lower the efforts for implementing this draft. To enable
	a platform supporting streaming telemetry to also support the Data		a platform supporting streaming telemetry to also support the Data
	Manifest, it is sufficient that this platform supports the models		Manifest, it is sufficient that this platform supports the models
	from Section 3 and Section 4. Recall that each type of manifest has		from Section 3 and Section 4. Recall that each type of manifest has
	its own rough frequency update, i.e. at reboot for the Platform		its own rough frequency update, i.e. at reboot for the Platform
	Manifest and at new subscription or CPU load variation for the Data		Manifest and at new subscription or CPU load variation for the Data
	Collection Manifest. The Data Manifest MUST be streamed with the		Collection Manifest. The Data Manifest MUST be streamed with the
	YANG-Push on-change feature [RFC8641] (also called event-driven		YANG-Push on-change feature [RFC8641] (also called event-driven
	telemetry).		telemetry). Appendix A shows how the in-band approach would work
			while storing to a time-series database (TSDB).
	5.2. Mapping Collected Data to the Data Manifest		5.2. Mapping Collected Data to the Data Manifest
	With YANG-push, each notification sent by the device is part of a		With YANG-push, each notification sent by the device is part of a
	subscription, which is also one of the YANG keys used to retrieve the		subscription, which is also one of the YANG keys used to retrieve the
	Data Manifest, the other key being the platform ID. In order to		Data Manifest, the other key being the platform ID. In order to
	enable a posteriori retrieval of the Data Manifest associated to a		enable a posteriori retrieval of the Data Manifest associated to a
	datapoint, the collector must:		datapoint, the collector must:
	o Keep the subscription id and platform id in the metadata of the		o Keep the subscription id and platform id in the metadata of the
	skipping to change at page 24, line 29 ¶		skipping to change at page 24, line 29 ¶
	With this information, to retrieve the Data Manifest from the		With this information, to retrieve the Data Manifest from the
	datapoint, the following happens:		datapoint, the following happens:
	o The subscription id and platform id are retrieved from the		o The subscription id and platform id are retrieved from the
	datapoint metadata		datapoint metadata
	o The Data Manifest for that datapoint is obtained by using the		o The Data Manifest for that datapoint is obtained by using the
	values above as keys.		values above as keys.
	We don't focus on the timing aspect as storing both the data and		We don't focus on the timing aspect as storing both the data and
	their manifest in a time series database will allow the data		their manifest in a time series database (TSDB) will allow the data
	scientists to look for the Data Manifest corresponding to the		scientists to look for the Data Manifest corresponding to the
	timestamp of the datapoint. In that scenario, the reliability of the		timestamp of the datapoint. More precisely, given the timestamp of a
	collection of the Data Manifest is the same as the reliability of the		collected datapoint, the query to the TSDB would be to get the last
	data collection itself, since the Data Manifest is like any other		version of the Data Manifest before that timestamp. In that
	data.		scenario, the reliability of the collection of the Data Manifest is
			the same as the reliability of the data collection itself, since the
			Data Manifest is like any other data.
	5.3. Operational Considerations		5.3. Operational Considerations
	It is expected that the Data Manifest is streamed directly from the		It is expected that the Data Manifest is streamed directly from the
	network equipment, along with YANG-Push [RFC8641] data. However, if		network equipment, along with YANG-Push [RFC8641] data. However, if
	the network element streaming telemetry does not support yet the YANG		the network equipment streaming telemetry does not support yet the
	modules from the Data Manifest specified in this document, the		YANG modules from the Data Manifest specified in this document, the
	telemetry collector could populate the Data Manifest from available		telemetry collector could populate the Data Manifest from available
	information collected from the platform. However, this option		information collected from the platform. However, this option
	requires efforts on the telemetry collector side, as the information		requires efforts on the telemetry collector side, as the information
	gathered in the Data Manifest proposed in this document could be		gathered in the Data Manifest proposed in this document could be
	scattered among various standard and vendor- specific YANG modules		scattered among various standard and vendor- specific YANG modules
	[RFC8199], that depend on the platform.		[RFC8199], that depend on the platform.
	That Data Manifest should be kept and available even if the source		That Data Manifest should be kept and available even if the source
	platform is not accessible (from the collection system), or if the		platform is not accessible (from the collection system), or if the
	platform has been updated (new operating system or new		platform has been updated (new operating system or new
	skipping to change at page 26, line 48 ¶		skipping to change at page 27, line 4 ¶
	}		}
	}		}
	]		]
	}		}
	}		}
	]		]
	}		}
	}		}
	<CODE ENDS>		<CODE ENDS>
	The file above contains the Data Collection Manifest for two Xpaths		The file above contains the Data Collection Manifest for two Xpaths
	subscriptions. With the Data Collection Manifest for the first one,		subscriptions. With the Data Collection Manifest for the first one,
	with subscription id 4242, the exact semantics of the collected path,		with subscription id 4242, the exact semantics of the collected path,
	here the administrative status of the network interfaces, can be		here the administrative status of the network interfaces, can be
	obtained by looking up the module in the yang-library of the		obtained by looking up the module in the yang-library of the
	corresponding plaftorm manifest, in order to obtain the exact		corresponding Platform Manifest, in order to obtain the exact
	revision of ieft-interfaces used at collection time. Also, the "on-		revision of ieft-interfaces used at collection time. Also, the "on-
	change" container indicates that data will be sent only if there is a		change" container indicates that data will be sent only if there is a
	change, thus not receiving data indicates that the administrative		change, thus not receiving data indicates that the administrative
	status of the interface did not change.		status of the interface did not change.
	The other example of Data Collection Manifest, with subscription id		The other example of Data Collection Manifest, with subscription id
	4243, shows how a periodic subscription is reported. In that		4243, shows how a periodic subscription is reported. In that
	example, the current-period indicates that the requested period of		example, the current-period indicates that the requested period of
	10s (1000 centiseconds) could not be attained and is now of 20s, for		10s (1000 centiseconds) could not be attained and is now of 20s, for
	instance because the device is overloaded.		instance because the device is overloaded.
	skipping to change at page 28, line 7 ¶		skipping to change at page 28, line 10 ¶
	library from the platform? Should we use a YANG mount to capture		library from the platform? Should we use a YANG mount to capture
	them as well (they would not be captured with our use of the main		them as well (they would not be captured with our use of the main
	yang-library grouping). Similarly, the ietf-data-collection-		yang-library grouping). Similarly, the ietf-data-collection-
	manifest.yang includes many lines of copy-pasting from ietf-yang-		manifest.yang includes many lines of copy-pasting from ietf-yang-
	push.yang and ietf-subscribed-notifications.yang since we want to		push.yang and ietf-subscribed-notifications.yang since we want to
	include the information from these modules. Reusing groupings is		include the information from these modules. Reusing groupings is
	not suitable as some leafrefs are pointing to nodes that are not		not suitable as some leafrefs are pointing to nodes that are not
	at the same location in our network level module. Maybe we need		at the same location in our network level module. Maybe we need
	to find a solution (deviations + some kind of schema mount?) maybe		to find a solution (deviations + some kind of schema mount?) maybe
	we can live with similar modules since they have common nodes but		we can live with similar modules since they have common nodes but
	different purposes?		different purposes? Current effort in
			[I-D.jouqui-netmod-yang-full-include] might be a potential
			solution.
	o Henk: how does this interact with SBOM effort?		o Henk: how does this interact with SBOM effort?
	o What is the link with the RFC8345 NodeId and IVY?		o What is the link with the RFC8345 NodeId and IVY?
			o Handling of deletion in [I-D.kll-yang-label-tsdb].
	11. References		11. References
	11.1. Normative References		11.1. Normative References
	[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate		[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
	Requirement Levels", BCP 14, RFC 2119,		Requirement Levels", BCP 14, RFC 2119,
	DOI 10.17487/RFC2119, March 1997,		DOI 10.17487/RFC2119, March 1997,
	<https://www.rfc-editor.org/info/rfc2119>.		<https://www.rfc-editor.org/info/rfc2119>.
	[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC		[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
	skipping to change at page 29, line 22 ¶		skipping to change at page 29, line 26 ¶
	netmod-model-catalog-03>.		netmod-model-catalog-03>.
	[I-D.claise-netconf-metadata-for-collection]		[I-D.claise-netconf-metadata-for-collection]
	Claise, B., Nayyar, M., and A. Sesani, "Per-Node		Claise, B., Nayyar, M., and A. Sesani, "Per-Node
	Capabilities for Optimum Operational Data Collection",		Capabilities for Optimum Operational Data Collection",
	draft-claise-netconf-metadata-for-collection-02 (work in		draft-claise-netconf-metadata-for-collection-02 (work in
	progress), July 2021,		progress), July 2021,
	<https://datatracker.ietf.org/doc/html/draft-claise-		<https://datatracker.ietf.org/doc/html/draft-claise-
	netconf-metadata-for-collection-02>.		netconf-metadata-for-collection-02>.
			[I-D.jouqui-netmod-yang-full-include]
			Joubert, T., Quilbeuf, J., and B. Claise, "YANG Full
			Include", draft-jouqui-netmod-yang-full-include-00 (work
			in progress), November 2023,
			<https://datatracker.ietf.org/doc/html/draft-jouqui-
			netmod-yang-full-include-00>.
			[I-D.kll-yang-label-tsdb]
			Larsson, K., "Mapping YANG Data to Label-Set Time Series",
			draft-kll-yang-label-tsdb-00 (work in progress), October
			2023, <https://datatracker.ietf.org/doc/html/draft-kll-
			yang-label-tsdb-00>.
	[I-D.lopez-opsawg-yang-provenance]		[I-D.lopez-opsawg-yang-provenance]
	Lopez, D., "Applying COSE Signatures for YANG Data		Lopez, D. and A. Pastor, "Applying COSE Signatures for
	Provenance", draft-lopez-opsawg-yang-provenance-00 (work		YANG Data Provenance", draft-lopez-opsawg-yang-
	in progress), July 2023,		provenance-01 (work in progress), October 2023,
	<https://datatracker.ietf.org/doc/html/draft-lopez-opsawg-		<https://datatracker.ietf.org/doc/html/draft-lopez-opsawg-
	yang-provenance-00>.		yang-provenance-01>.
	[I-D.tgraf-netconf-notif-sequencing]		[I-D.tgraf-netconf-notif-sequencing]
	Graf, T., Quilbeuf, J., and A. Feng, "Support of Hostname		Graf, T., Quilbeuf, J., and A. Feng, "Support of Hostname
	and Sequencing in YANG Notifications", draft-tgraf-		and Sequencing in YANG Notifications", draft-tgraf-
	netconf-notif-sequencing-02 (work in progress), October		netconf-notif-sequencing-03 (work in progress), January
	2023, <https://datatracker.ietf.org/doc/html/draft-tgraf-		2024, <https://datatracker.ietf.org/doc/html/draft-tgraf-
	netconf-notif-sequencing-02>.		netconf-notif-sequencing-03>.
	[RFC8199] Bogdanovic, D., Claise, B., and C. Moberg, "YANG Module		[RFC8199] Bogdanovic, D., Claise, B., and C. Moberg, "YANG Module
	Classification", RFC 8199, DOI 10.17487/RFC8199, July		Classification", RFC 8199, DOI 10.17487/RFC8199, July
	2017, <https://www.rfc-editor.org/info/rfc8199>.		2017, <https://www.rfc-editor.org/info/rfc8199>.
	[RFC8343] Bjorklund, M., "A YANG Data Model for Interface		[RFC8343] Bjorklund, M., "A YANG Data Model for Interface
	Management", RFC 8343, DOI 10.17487/RFC8343, March 2018,		Management", RFC 8343, DOI 10.17487/RFC8343, March 2018,
	<https://www.rfc-editor.org/info/rfc8343>.		<https://www.rfc-editor.org/info/rfc8343>.
	[RFC9196] Lengyel, B., Clemm, A., and B. Claise, "YANG Modules		[RFC9196] Lengyel, B., Clemm, A., and B. Claise, "YANG Modules
	skipping to change at page 30, line 9 ¶		skipping to change at page 30, line 27 ¶
	[RFC9371] Baber, A. and P. Hoffman, "Registration Procedures for		[RFC9371] Baber, A. and P. Hoffman, "Registration Procedures for
	Private Enterprise Numbers (PENs)", RFC 9371,		Private Enterprise Numbers (PENs)", RFC 9371,
	DOI 10.17487/RFC9371, March 2023,		DOI 10.17487/RFC9371, March 2023,
	<https://www.rfc-editor.org/info/rfc9371>.		<https://www.rfc-editor.org/info/rfc9371>.
	Appendix A. Example of use based on MDT		Appendix A. Example of use based on MDT
	In this example, the goal is to collect the administrative status and		In this example, the goal is to collect the administrative status and
	number of received bytes for the interfaces of a fictional ACME		number of received bytes for the interfaces of a fictional ACME
	device, and store the result in a Influx database. The metrics are		device, and store the result in a time-series database (TSDB). The
	collected via YANG-Push, which is configured by specifying their		metrics are collected via YANG-Push, which is configured by
	XPaths and when they should be collected (periodically or on-change).		specifying their XPaths and when they should be collected
	More precisely, we want collect "ietf-		(periodically or on-change). More precisely, we want collect "ietf-
	interfaces:interfaces/interface/enabled" on every change and "ietf-		interfaces:interfaces/interface/enabled" on every change and "ietf-
	interfaces:interfaces/interface/statistics/in-octets" every 100		interfaces:interfaces/interface/statistics/in-octets" every 100
	milliseconds. The paths here are referring to the YANG module from		milliseconds. The paths here are referring to the YANG module from
	[RFC8343]. The configuration of YANG push is out of scope for this		[RFC8343]. The configuration of YANG push is out of scope for this
	document. Since they don't have the same trigger, each of the path		document. Since they don't have the same trigger, each of the path
	must be collected in its own subscription. Figure 3 presents an		must be collected in its own subscription. Figure 3 presents an
	example for such a collection.		example for such a collection.
	+------------+ +----------+		+------------+ +--------+
	| MDT | | Influx |		| MDT |--------------> | TSDB |
	| Collector |--------------> | DB |		| Collector | +--------+
	+------------+ +----------+		+------------+
	^		^
	|		|
	|		|
	+---------+		+---------+
	| Device |		| Device |
	+---------+		+---------+
	Figure 3: Example of collection from a device to Influx DB		Figure 3: Example of collection from a device to a TSDB
	In the scenario from Figure 3, the collector receives YANG-push from		In the scenario from Figure 3, the collector receives YANG-push from
	the device and stores it into InfluxDB. We first present a version		the device and stores it into a TSDB. We first present a version
	without data manifest and then how to enrich it with the data		without Data Manifest and then how to enrich it with the Data
	manifest.		Manifest.
	In InfluxDB, a datapoint is specified by giving the name of the
	measurement, zero or more key value entries named tags, one or more
	named values called fields and the timestamp for the datapoint. In
	our case a measurement could be "admin-status". The tags, whose aim
	to identify a particular instance of the measurement could be the
	name of the device and the name of the interface. The fields contain
	the values to store. InfluxDb defines a textual notation, named line
	protocol, to represent one datapoint per line. We use this line
	protocol in Figure 4 and Figure 6 to represent the way data could be
	fed to InfluxDB, omitting the timestamp for readability. See
	<https://docs.influxdata.com/influxdb/cloud/reference/syntax/line-
	protocol/> for more details.
	Without the data manifest, the YANG-push collector is likely to store		We use the notation from [I-D.kll-yang-label-tsdb] to represent how
	something similar to Figure 4 in InfluxDB. In that case, only the		the data is stored in the TSDB. Without the data manifest, the
	value is stored, without any way to know how the value was obtained.		result of the collection would be stored as showed in Figure 4. The
			"host" label indicates the devices from which the data is collected
			and the YANG keys are included as well. Here the interface "eth0" is
			enabled and received 1234 octets. In that case, the value is stored,
			without any way to know how the value was obtained.
	admin_status,device="PE1",interface="gig1" val=T		* Metric: interfaces_interface_enabled
	sent_bytes,device="PE1",interface="gig1" val=1234		* Value: True
			* Labels:
			- host: "PE1"
			- interfaces_interface_name: "eth0"
			--
			* Metric: interfaces_interface_statistics_in_octets
			* Value: 1234
			* Labels:
			- host: "PE1"
			- interfaces_interface_name: "eth0"
	Figure 4: Storing datapoints without data manifest		Figure 4: Storing datapoints without Data Manifest
	A possibility for keeping the data manifest with the data is to store		A possibility for keeping the Data Manifest with the data is to store
	it directly into InfluxDB. In that case, the collector can subscribe		it directly into the TSDB. In that case, the collector can subscribe
	to the data exported by the module presented in this draft and store		to the data exported by the module presented in this draft and store
	it inside influxDB. For the Platform Manifest, assuming the platform		it as other metrics. For the Platform Manifest, assuming the
	ID is "PE1", the collector subscribes to the path "ietf-platform-		platform ID is "PE1", the collector subscribes to the path "ietf-
	manifest:platforms/platform[id=PE1]". For the Data Collection		platform-manifest:platforms/platform[id=PE1]". For the Data
	Manifests, the collector subscribes to the path "ietf-data-		Collection Manifests, the collector subscribes to the path "ietf-
	collection-manifest:data-collections/data-collection[platform-		data-collection-manifest:data-collections/data-collection[platform-
	id="PE1"]/yang-push-subscriptions/subscription[id=X]" where X is the		id="PE1"]/yang-push-subscriptions/subscription[id=X]" where X is the
	subscription id of existing subscriptions. The data, for instance		subscription id of existing subscriptions. With the approach from
	serialized in JSON, can be stored in InfluxDB as shown in Figure 5		[I-D.kll-yang-label-tsdb], the corresponding subtrees would be split
	where "<platform-manifest>" and "<data-manifest>" represent the		into a set of datapoints, one per leaf. Figure 5 shows two examples
	contents of respectively the Platform Manifest and the Data		of storing leaves in a TSDB. The first leaf is the vendor PEN
	Collection Manifest.		number, which is part of the Platform Manifest. The second leaf is
			the Xpath filter used for subscription to the interface status.
	platform-manifest,device="PE1" val=<plaftorm-manifest>		* Metric: platforms_platform_vendor_pen
	collection-data-manifest,device="PE1",subId=4242 val=<data-manifest>		* Value: 32473
	collection-data-manifest,device="PE1",subId=4243 val=<data-manifest>		* Labels:
			- host: "PE1"
			- platforms_platform_id: "PE1"
			--
			* Metric: data_collections_data_collection_yang_push_subscriptions_
			subscription_datastore_xpath_filter
			* Value: "ietf-interfaces:interfaces/interface/enabled"
			* Labels:
			- host: "PE1"
			- data_collections_data_collection_platform_id: "PE1"
			- data_collections_data_collection_yang_push_subscriptions_
			subscription_id: 4242
	Figure 5: Storing data manifest		Figure 5: Example of storing Platform and Data Collection Manifest:
			Vendor PEN and Xpath filter.
	In our example, The link between a collected datapoint and the		In the labels, the "host" might be different from the
	corresponding Platform Manifest is done via the common "device" tag.		"platforms_platform_id" in case the collector is the one assembling
	In order to link a datapoint with the corresponding Data Collection		it, i.e. for devices that do not natively support the Data Manifest.
	Manifest, the collector can add fields to specify where the Data		In that case, the value of this label could be the hostname of the
	Collection Manifest is located for that specific datapoint. For		collector. The host value does not matter for retrieving the Data
	instance, the same datapoints as in Figure 4 could be stored as in		Manifest as the platform id is the meaningful field.
	Figure 6.
	admin_status,device="PE1",interface="gig1" val=T,subId=4242		In our example, we can retrieve the Platform Manifest associated to a
	sent_bytes,device="PE1",interface="gig1" val=1234,subId=4243		collected datapoint by looking for datapoints that have the label
			"platforms_platform_id" equal to the value of the host for that
			collected datapoint. In order to link a datapoint with the
			corresponding Data Collection Manifest, we need to add an additional
			label for the subscription id. For instance, the same datapoints as
			in Figure 4 could be stored as in Figure 6.
	Figure 6: Storing datapoints with data manifest		* Metric: interfaces_interface_enabled
			* Value: True
			* Labels:
			- host: "PE1"
			- interfaces_interface_name: "eth0"
			- data_collections_data_collection_yang_push_subscriptions_
			subscription_id: 4242
			--
			* Metric: interfaces_interface_statistics_in_octets
			* Value: 1234
			* Labels:
			- host: "PE1"
			- interfaces_interface_name: "eth0"
			- data_collections_data_collection_yang_push_subscriptions_
			subscription_id: 4243
	In our simple example, from the "admin_status" datapoint, one can		Figure 6: Storing datapoints with information to retrieve the Data
	retrieve the corresponding Platform Manifest by looking at the last		Manifest
	value for the "platform-manifest" measurement with the same value for
	the "device" tag. From the "admin-status" datapoint, one can		From the "interfaces_interface_enabled" datapoint, one can retrieve
	retrieve the corresponding Data Collection Manifest by looking at the		the corresponding Data Collection Manifest by looking for datapoints
	last value for the "data-manifest" measurement with tags "device" and		that have the label data_collections_data_collection_yang_push_subscr
	"subId" matching respectively with the tag "device" and the field		iptions_subscription_id equal to 4242.
	"subId" of the measurement.
			Various optimizations could be done, such as relying on on-change
			subscription to modify only the leaves that changed. In that way,
			the amount of data needed for updating and storing the Data Manifest
			in the TSDB would be limited.
	Appendix B. Changes between revisions		Appendix B. Changes between revisions
			v02 -> v03
			Explicit that modules are network (Controller) level
			InfluxDB example changed to TSDB example aligned with
			[I-D.kll-yang-label-tsdb]
			Minor edits i.e. network element -> platform , object -> data node
	v01 -> v02		v01 -> v02
	Updated example with latest version of the model.		Updated example with latest version of the model.
	v00 (WG adoption) - v01		v00 (WG adoption) - v01
	Solve integrity issue by delegating to		Solve integrity issue by delegating to
	[I-D.lopez-opsawg-yang-provenance].		[I-D.lopez-opsawg-yang-provenance].
	v05 -> v06		v05 -> v06
	skipping to change at page 50, line 48 ¶		skipping to change at page 52, line 32 ¶
	}		}
	uses hints;		uses hints;
	}		}
	}		}
	}		}
	<CODE ENDS>		<CODE ENDS>
	Acknowledgements		Acknowledgements
	Thanks to Mohamed Boucadair and Tianran Zhou for their reviews and		Thanks to Mohamed Boucadair, Tianran Zhou and Jan Lindblad for their
	comments.		reviews and comments.
	Authors' Addresses		Authors' Addresses
	Benoit Claise		Benoit Claise
	Huawei		Huawei
	Email: benoit.claise@huawei.com		Email: benoit.claise@huawei.com
	Jean Quilbeuf		Jean Quilbeuf
	Huawei		Huawei
End of changes. 53 change blocks.
	111 lines changed or deleted		177 lines changed or added
This html diff was produced by rfcdiff 1.45. The latest version is available from http://tools.ietf.org/tools/rfcdiff/