First TMF Switch-Off: Preliminary Call for Participation

The Measurement Factory

Measurement Factory is organizing an industry-wide competition of L4/7 load balancing products. Below you will find preliminary details about the event. We encourage all vendors with relevant products to contact us as soon as possible and participate in the organizational meeting. No formal commitments are required at this time.

Table of Contents

1. Nature of the event
2. Scope of the tests
3. Testing methodology
4. Publication of the results
5. Organizational meeting
6. Location
7. Important dates
8. Contact information

1. Nature of the event

The switch-off is an all-under-one-roof competition of load balancing products. All participating products will be tested under identical rules, using the same software and workload, and at about the same time. Participants are responsible for configuration and tuning of their products. Measurement Factory provides the test harness, executes the tests, and analyses the results. The results, prices, and all configuration details will be made public shortly after the tests.

The rules of the competition and workload details are developed with the cooperation of the user community and test participants.

Participation fee is required to compete.

2. Scope of the tests

We propose that the first round of tests is limited to Layer 4/7 server load balancing solutions. We do not plan to test proxy load balancers. L7 products are encouraged to participate, but the workload will not have special features that benefit L7 products only. These limitations will be discussed and can be changed at the organizational meeting.

All commercial vendors, non-profit organizations, and any other entities with a L4+ load balancing product may participate. No artificial participation limits are planned at this time.

Switch-off participants may choose to tighten or change the scope of the tests by, for example, eliminating beta-quality products or introducing restrictions on supported network interfaces. These issues will be discussed at the organizational meeting.

3. Testing methodology

Web Polygraph benchmark will be used for the tests. Polygraph test suite includes simulated HTTP clients and servers. A participant will specify peak request rate they want to support (as a workload parameter). A typical Polygraph setup will include thousands of simulated clients and a few simulated servers, all with unique IP addresses. Using those simulated agents, Polygraph will collect performance measurements and verify that a product can sustain the advertised request rate.

A single test is expected to last for about two hours or less. Polygraph client and servers will use real HTTP/TCP/IP transactions with appropriate features (e.g., HTTP persistent connections), headers and content. Simulated Web objects will be of realistic sizes (about 11KByte mean and 4KByte median). Exact workload specifications and execution rules will be available in advance.

Polygraph PCs will have 100baseTX interfaces. Participants will be able to use L2 switches as traffic aggregation devices if necessary. The details of traffic aggregation rules will have to be determined with participant input.

Reported measurements will include throughput (HTTP requests per second), bandwidth (bytes per second), mean response time of an HTTP transaction, load distribution abilities, and errors. The official report will also include pricing, availability, and configuration information.

4. Publication of the results

An official public report will be available at Measurement Factory Web site. At least one major trade magazine is expected to publish the results of the tests. Participants will be able to create their derivative works from the official report.

5. Organizational meeting

Measurement factory would like to host a face-to-face meeting with potential switch-off participants. During the meeting, test scope and methodology, participation fees, preparation tips, and logistics details will be discussed, and the final shape of the event will be determined. Attendance is not mandatory but strongly encouraged. Nobody can represent participant's interests better than the participant itself.

Registration is required to attend the meeting. Please send an e-mail to <team at measurement-factory.com>. The attendance is limited to one-two persons per organization.

The meeting will be held in Denver, Colorado unless most of the participants prefer a different location.

6. Location

Measurement Factory plans to host the switch-off near Denver, Colorado. Proposals for alternative locations will be considered. The requirements are lots of power, space, and appropriate cooling for 150+ PCs.

7. Important dates

Here is the proposed schedule. Deadlines may be changed at the organizational meeting.

Date
(2001)
Time
(MDT)
Event
May 17* 09:30
17:30
Organizational meeting
May 28 15:00 Workload and code freeze
June 15 15:00 Switch-off registration
July 16 09:00 Switch-off starts
July 20 18:00 Switch-off ends

*The date has been changed! The old date, May 7th, conflicted with N+I event. We hope that no further changes are required, but please let us know about new conflicts ASAP!

8. Contact information

Questions? E-mail <info at measurement-factory.com>.