@@ -3,10 +3,22 @@ produce a series of recommendations concerning the key performance
characteristics of internetworking technologies, or benchmarks for
network devices, systems, and services. Taking a view of networking
divided into planes, the scope of work includes benchmarks for the
management, control, and forwarding planes.
Each recommendation will describe the class of equipment, system, or
service being addressed; discuss the performance characteristics that
management, control, and forwarding planes.
The scope of BMWG has been extended to develop methods for
virtual network functions (VNF) and their unique supporting
infrastructure (such as SDN Controllers and vSwitches).
Benchmarks for platform capacity and performance characteristics
of virtual routers, firewalls (and other security functions),
signaling control gateways, and other forms of gateways are included.
The benchmarks will foster comparisons between physical and virtual
network functions, and also cover unique features of
Network Function Virtualization systems. Also, with the emergence
of virtualized test systems, specifications for test system
calibration are also in-scope.
Each recommendation will describe the class of network function, system,
or service being addressed; discuss the performance characteristics that
are pertinent to that class; clearly identify a set of metrics that aid
in the description of those characteristics; specify the methodologies
required to collect said metrics; and lastly, present the requirements
@@ -18,8 +30,8 @@ and from those affected by the benchmarks when they are published
(networking and test equipment manufacturers). When possible, the
benchmarks and other terminologies will be developed jointly with
organizations that are willing to share their expertise. Joint review
requirements for a specific work area will be included in the detailed
description of the task, as listed below .
requirements for a specific work area will be included in the
description of the task.
To better distinguish the BMWG from other measurement initiatives in the
IETF, the scope of the BMWG is limited to the characterization of
@@ -36,81 +48,6 @@ acceptance criteria or performance requirements.
An ongoing task is to provide a forum for development and
advancement of measurements which provide insight on the
capabilities and operation of implementations of inter-networking technology.
Ideally, BMWG should communicate with the operations community
through organizations such as NANOG, RIPE, and APRICOT.
The BMWG is explicitly tasked to develop benchmarks and methodologies
for the following technologies:
BGP Control-plane Convergence Methodology (Terminology is complete):
With relevant performance characteristics identified, BMWG will prepare
a Benchmarking Methodology Document with review from the Routing Area
(e.g., the IDR working group and/or the RTG-DIR). The Benchmarking
Methodology will be Last-Called in all the groups that previously
provided input, including another round of network operator input during
the last call.
SIP Networking Devices: Develop new terminology and methods to
characterize the key performance aspects of network devices using
SIP, including the signaling plane scale and service rates while
considering load conditions on both the signaling and media planes. This
work will be harmonized with related SIP performance metric definitions
prepared by the PMOL working group.
Traffic Management: Develop the methods to characterize the capacity
of traffic management features in network devices, such as classification,
policing, shaping, and active queue management. Existing terminology
will be used where appropriate. Configured operation will be verified
as a part of the methodology. The goal is a methodology to assess the
maximum forwarding performance that a network device can sustain without
dropping or impairing packets, or compromising the accuracy of multiple
instances of traffic management functions. This is the benchmark for
comparison between devices. Another goal is to devise methods that
utilize flows with congestion-aware transport as part of the traffic
load and still produce repeatable results in the isolated test environment.
IPv6 Neighbor Discovery: Large address space in IPv6 subnets presents
several networking challenges, as described in RFC 6583. Indexes to
describe the performance of network devices, such as the number of
reachable devices on a sub-network, are useful benchmarks to the
operations community. The BMWG will develop the necessary
terminology and methodologies to measure such benchmarks.
In-Service Software Upgrade: Develop new methods and benchmarks to
characterize the upgrade of network devices while in-service,
considering both data and control plane operations and impacts.
These devices are generally expected to maintain control plane session
integrity, including routing connections. Quantification of upgrade
impact will include packet loss measurement, and other forms of recovery
behavior will be noted accordingly. The work will produce a definition
of ISSU, which will help refine the scope. Liaisons will be established
as needed.
Data Center Benchmarking: This work will define additional terms,
benchmarks, and methods applicable to data center performance evaluations.
This includes data center specific congestion scenarios, switch buffer
analysis, microburst, head of line blocking, while also using a wide mix
of traffic conditions. Some aspects from BMWG's past work are not
meaningful when testing switches that implement new IEEE specifications
in the area of data center bridging. For example, throughput as defined
in RFC 1242 cannot be measured when testing devices that implement three
new IEEE specifications: priority-based flow control (802.1Qbb);
priority groups (802.1Qaz); and congestion notification (802.1Qau).
This work will update RFC1242, RFC2544, RFC2889 (and other key RFCs),
and exchange Liaisons with relevant SDOs, especially at WG Last Call.
VNF and Related Infrastructure Benchmarking: Benchmarking Methodologies
have reliably characterized many physical devices. This work item extends
and enhances the methods to virtual network functions (VNF) and their
unique supporting infrastructure. A first deliverable from this activity
will be a document that considers the new benchmarking space to ensure
that common issues are recognized from the start, using background
materials from industry and SDOs (e.g., IETF, ETSI NFV).
Benchmarks for platform capacity and performance characteristics of
virtual routers, switches, and related components will follow, including
comparisons between physical and virtual network functions. In many cases,
the traditional benchmarks should be applicable to VNFs, but the lab
set-ups, configurations, and measurement methods will likely need to
be revised or enhanced.
capabilities and operation of implementations of inter-networking
technology. Ideally, BMWG should communicate with the operations
community through organizations such as NANOG, RIPE, and APRICOT.
wkumari revised this gist