David Swenson's blog

We do a lot of interoperability and performance testing, and this often involves schlepping a bunch of equipment to the partner or OEM site. Here in Silicon Valley, the facility may be just a few miles away so we drive, and that means packing up a bunch of moving cartons with servers, WAN emulators, and of course the two DMS systems. Today we were packing up after completing some tests and one of their QA guys who was helping commented on how light the DMS was, especially compared with all the servers that we needed to generate the +40Gbps flows.

“RAM,” I told him. “Our algorithm is really efficient.”

Huh? What does a little math formula have to do with the weight of our network deduplication switch?

Here’s the backstory...

Big Data implementations such as Hadoop are becoming more common, and as they do, organizations are discovering that Big Data drives Big Traffic. Ashish Shah has written a great piece on this subject, available here:  Wikibon.

Over the next five years, machine-to-machine traffic between data centers (over Data Center Interconnects, or DCIs) is expected to increase faster than traffic within data centers, forcing organizations to respond by implementing multiple 10 Gbps WAN links. In most cases, however, simply scaling up the WAN infrastructure is a weak long-term strategy. Instead, keeping pace with DCI traffic requires a new class of WAN optimization technologies that can scale to 10Gbps speeds while introducing minimum latency to the end-to-end workflow.

Two common ways of gauging DMS performance are the reduction rate, which looks at the data handled by the DMS, and throughput, which looks at the WAN link on which the data is carried.  There is a third way to measure DMS performance, however, and that is to look at the impact the DMS has on endpoint performance.  We call this the compound effect, and although it is perhaps the most telling measure of performance, it is often overlooked. The compound effect is simply the product of the reduction rate and the throughput increase.  

Let’s say after deploying the DMS you consistently observe reduction rates of 5x. Not bad! End of story, right?  No.