First Switch-off: The Rules

The Measurement Factory

These rules are not final and are subject to change. If you have any suggestions on improving the rules, please contact us.

Table of Contents

1. Terminology
2. Requirements for participation
    2.1 Product prerequisites
    2.2 Preparation control
    2.3 Disclosures
3. Access to the switch-off facility
4. Benches
    4.1 TMF zone
    4.2 Participant zone
    4.3 Inter-Zone communication
    4.4 Inter-Bench network
5. Testing
    5.1 Official tests
    5.2 Test execution
    5.3 Results selection
    5.4 Equipment failures
6. Switch-off report
    6.1 Comments on the results
    6.2 Derivative works, publications, and such
7. Bailout and failures
8. Conflict resolution

1. Terminology

Vendor
An organization bringing a Product to be tested at the switch-off. To simplify the terminology all commercial, non-profit, virtual, etc. organizations are labeled as ``Vendors.''
Product under test
A Product/solution/box being tested as a single load balancer. A Product includes software, hardware, network gear, etc. required for load balancer operation. A Product may include several units. A Product may also be referred to as DUT (device under test).
Entry
A (Vendor, Product) tuple.
Participant
A Vendor associated with a particular Entry.
Test Bench
A group of machines and network gear dedicated to benchmarking an Entry, including the Product.
TMF
The Measurement Factory; TMF is the company responsible for Web Polygraph development and switch-off organization.

2. Requirements for participation

2.1 Product prerequisites

A Product (as defined in the ``Terminology'' section) must satisfy all of the following criteria to be considered for the switch-off.

2.2 Preparation control

A hard copy of an auto-generated Polygraph report should be presented to TMF before the start of the first official test for the corresponding Entry. TMF reserves the right to make public the fact that the report was (or was not) submitted. The contents of the report will not be made public. The report should be generated with the latest 2.6.x version of Polygraph and correspond to the same (to the extent possible) Product as will be tested at the switch-off. The report must be based on a successful performance test.

If such a report is not presented, the Entry will receive the lowest handling priority during the switch-off, including troubleshooting and technical support or advice.

An Entry's first official performance test will use the same, or lower, request rate from the submitted report.

2.3 Disclosures

Each Participant must disclose the following:

This documentation must be made available prior to the start of the official test sequence. TMF will not start official tests unless the Participant provides all necessary details.

All prices must be US prices specified in US dollars. All prices must be based on unlimited usage licenses and must carry no discriminatory discounts or usage restrictions.

There will be no private rooms for test benches. Participants will be able to see each other's equipment.

During the switch-off, TMF will take photographs of the equipment as tested. Photographs from other sources will not be accepted. The photographs taken at the switch-off may be included in the official report to document Entry configuration and appearance.

3. Access to the switch-off facility

Working hours are Monday through Friday, 9am till 7pm with an hour lunch break.

4. Benches

A Bench is logically divided into a TMF zone and a Participant zone. Zones are connected using 100BaseT Ethernet cables. Refer to the figure below:

4.1 TMF zone

TMF provides a number of machines to operate as clients and servers. Client machines have similar hardware and software configuration (identical to the extent possible). Server machines have similar hardware and software configuration (identical to the extent possible).

To manage the tests, TMF provides monitoring stations, one per Bench. The configuration of monitoring stations does not affect the tests.

Workload parameters (e.g., peak request rate) determine the number of client and server machines in a Bench. Some Benches may require more clients and servers than other Benches.

TMF is responsible for setup, operation, and maintenance of the client and server machines. The exact hardware and software specs will be included in TMF's switch-off report.

4.2 Participant zone

The Participant zone includes the load balancer under test, cables, and any networking gear (switch, router, hub) used to connect the client and server machines to the load balancer.

All equipment in the Participant zone must meet the ``Product prerequisites'' described previously.

L2 aggregation devices may and, in some cases, must be used to aggregate traffic to/from Polygraph clients and servers. No more than four of the Product's ports may be used to connect to Polygraph clients. Thus, if your bench uses more than four client PCs, you must use a L2 aggregation device between clients and your Product. The list price of client-side L2 aggregation devices must be disclosed, but does not affect the total Entry price.

L2 aggregation devices must be ``L2 only'' or must be configured only for L2 forwarding.

You may also use Server-side L2 aggregation devices, but these do contribute to the total list price of your Entry.

All aggregation devices must be disclosed to TMF as early as possible and before the start of the switch-off. TMF reserves the right to grant or revoke permission to use an aggregation device at ``zero cost''. Since TMF's decision may be somewhat arbitrary, it is in the Participant's interests to double check with TMF early in the process, to avoid unexpected Entry price increase.

With the exception of client-side L2 aggregation devices (if any), all equipment in the Participant zone, including networking hardware, is included in the price of the Participant's configuration. Substitutions are not allowed. For example, a participant brings a 24-port switch and uses 8 ports, the price cannot be scaled down to an 8 port switch of the same manufacturer. Removing ``blades'' or ``cards'' from network gear is allowed if the gear can be purchased without those components.

The Participant is responsible for setup and maintenance of all the equipment in the Participant zone.

The Participant should specify any special accommodations (extra cooling, extra power outlets, power other than 110V AC, etc.) as early as possible.

4.3 Inter-Zone communication

Participant zone must allocate at least one 10/100BaseT Ethernet port for the monitoring station.

Participant zone setup must provide IP level connectivity between the monitoring station and all client and server machines. For example, the monitoring station must be able to ``ping'' or ``ssh'' to all clients and servers.

Participant may request that their cables are used for networking clients and servers. Otherwise, TMF provides cables.

In the event of performance, compatibility, or other problems between TMF and Participant zones, TMF and the Participant must work together to isolate and eliminate the problem.

4.4 Inter-Bench network

In order to facilitate centralized management of all client and server machines, TMF may utilize a routed network that connects all Benches. This network is to be physically dismantled during performance tests. All benches are physically separated, and no connections between benches are allowed (except for temporary network connections described above).

5. Testing

Tests are run independently and concurrently on every Bench. There may be fewer benches than Entries. Test schedule for Participants is determined by TMF.

The rest of the section describes the test execution procedure for one Participant.

5.1 Official tests

Official tests are:

  1. MSL test
  2. Layer 4 performance tests
    1. no-balancing test
    2. SrvLB-L4 strict test
    3. SrvLB-L4 lenient test
  3. Layer 7 performance tests
    1. no-balancing test
    2. SrvLB-L7 strict test
    3. SrvLB-L7 lenient test

All other tests are unofficial.

In the L4 workload, the device under test is expected to distribute load among N identical servers. Each server contains all the content and can serve any request. In the L7 test, the content is partitioned among the servers. The device under test is expected to use URL suffix matching to distribute load.

The L4 and L7 workloads are further split into ``Strict'' and ``Lenient'' sub-configurations. The difference between the Strict and Lenient configurations are optional server-side changes only. The Strict configuration is meant to simulate a customer with typical but inflexible requirements. This customer will not be convinced to change its server configuration to accommodate a particular load-balancing solution. The Lenient configuration, on the other hand, represents a customer who is willing to make certain server-side changes to accommodate the vendor's optimal load-balancing solution. In other words, the Strict configuration is identical for all participants, and no server-side customizations are allowed. For the Lenient configuration, we are willing to make a number of server-side customizations.

For Strict configurations, Polygraph servers must have routes to either (a) the Polygraph client ``router interfaces'' or (b) a single default gateway IP address belonging to the Product.

For Lenient configurations, you may disable persistent HTTP connections on Polygraph servers. You may also create an IP alias (the VIP address) on each Polygraph server, bound to the loopback interface. If you can think of other server-side customizations, please let us know.

For each workload, the Strict test is mandatory, but the Lenient test is optional.

The L4 test is mandatory for those products that support it. It is difficult for TMF to know for sure whether or not a particular product can support L4 traffic redirection. If the participant tells us that their product does not support L4 redirection, or can not complete the test, our report will say:

Product X does not support L4 traffic redirection.
Because L4 configurations are a significant piece of the load-balancing market, we feel that this is a strong incentive for L4-capable products to actually take the test.

5.2 Test execution

All tests must start and finish within the time allocated for the switch-off. Participants running tests for a single layer (L4 or L7) receive 16 hours of ``busy'' time. Participants running tests for both layers (L4 and L7) receive 24 hours of ``busy'' time. ``Busy'' time includes the time when official tests are running (or can be running as far as TMF is concerned). ``Busy'' time also includes 15 minutes to prepare each test. If the participant shows up late, or is not ready for some reason, time is subtracted from their 16 hours until they are ready for testing.

TMF treats requests to repeat a test with a lower priority than requests to execute the first test. Rerunning unexpectedly failed tests gets higher priority that rerunning other tests. TMF may refuse to repeat the test when 16 hours run out and/or when further trials are unlikely to change the outcome of the tests.

Test execution rules also include the following provisions.

5.3 Results selection

Reasonable efforts will be made by TMF to share run-time performance with Participants.

A participant may repeat any test and choose the best results among all non-failed tests executed. Participants must inform TMF of their choice(s) within 96 hours of the end of the switch-off.

TMF archives all the logs from the official tests. On Participant request, TMF releases these results to the Participant after the switch-off is over (or earlier, time permitting).

5.4 Equipment failures

In the event of equipment failure, the failed parts can be replaced, but the interrupted test (if any) is invalidated. The fact of the failure is not reported unless the failure leaves a participant without valid results.

6. Switch-off report

TMF prepares an official switch-off report. TMF makes reasonable efforts to discuss the content of the report with Vendors before making the report public. TMF makes all switch-off results publicly available on the Web. TMF must keep the switch-off archive available for at least one year.

Switch-off participants agree not to make public (or publicly discuss) the results of the switch-off prior to public release of the official report by TMF.

TMF may release draft copies of the report to members of the press in order to facilitate timely publication of the results.

6.1 Comments on the results

A Participant may request inclusion of a ``Participant Comments'' section into the report. The comments must not contain any references to external documents. The comments should only discuss matters relevant to the switch-off. Vendors are strongly discouraged from including speculations and accusations in their comments.

The comments are limited to 300 words per Entry and 750 words per Vendor, whichever is less. All comments must be submitted in plain ASCII text format.

Once included in the report, the comments cannot be modified.

TMF reserves the right to not include the Participant Comments.

TMF may add their own comments, up to three months after the publication of the report.

The Comments become a part of the official result distribution. These Rules do not require any part of the results to be included in the derived works.

6.2 Derivative works, publications, and such

Anyone can publish or otherwise make public any derivate work based on the switch-off results (i.e., logs and the official report), provided that the ``Terms of use'' clause is satisfied.

Anyone can get a free copy of a Participant result from the Official Site, provided they agree to the ``Terms of use''.

Terms of use
  1. The Measurement Factory holds the copyright to the results and the official report.
  2. Any derived work, distribution, publication, presentation, etc. of the results must include the following explicit reference in its entirety:
    Original raw data and independent analysis of the first switch-off results are at http://www.measurement-factory.com/results/
  3. All modifications of the results and the official report are prohibited. Only verbatim copies may be distributed.

7. Bailout and failures

After the start of the first official test, a participant must not bailout or prevent their results from being published.

A Participant may request that a test is considered ``failed''. If all tests for the Entry fail, the Entry is considered ``failed''. TMF may declare an Entry ``failed'' if the Entry cannot produce reasonable results despite TMF and Participants efforts such as decreasing request rate by 50% of the registered peak load.

Failed Entries (if any) will be counted in the report preamble and listed in the executive summary table with all measurements replaced with ``n/a''.

A special section at the end of the report will contain details necessary for a reader to understand why no results are available for each failed Entry (and, possibly, links to re-test results; see below). Vendors of failed Entries may submit ``vendor comments'' sections.

Failed Entries will be given an option to be re-tested at TMF lab within two weeks after the end of the switch-off. The results of those tests will be linked from the official switch-off report, but will not be included into the switch-off report. The Vendor of failed Entries is expected to pay for any extra equipment required for the re-tests.

8. Conflict resolution

During the switch-off, situations requiring Vendor input may arise. Presence of all Vendors during the entire competition is strongly encouraged. Absent Vendors are assumed to side with TMF in case of run-time disputes or conflicts. Participants agree to honor any addendum developed during the switch-off, regardless of participants presence at the decision making meeting.

A reasonable effort on the part of a TMF will be made to engage all present Vendors and to find a compromise solution.

In the event of any form of conflict, including interpretation of these Rules, TMF will be the final arbiter and judge.