Menu
Microsoft beats data-sorting record with new approach

Microsoft beats data-sorting record with new approach

A new approach, called Flat Datacenter Storage, pushes computational sorting to each data server

Besting a record set by Yahoo in 2009, the research arm of Microsoft have deployed a new technique for quickly sorting large amounts of data, called Flat Datacenter Storage (FDS).

The researchers will discuss their work at an Association for Computing Machinery conference dedicated to databases this week in Scottsdale, Arizona. They are also implanting their data-sorting techniques in Microsoft's Bing search engine, where it could boost response times to user queries.

"Improving big-data performance has a wide range of implications across a huge number of businesses," said Microsoft Research project leader Jeremy Elson, in an online entry describing the work. "Almost any big-data problem now becomes more efficient, which, in many cases, will be the difference between the work being economically feasible or not."

In tests conducted under the MinuteSort benchmark, the system set up by Elson and his colleagues was able to sort 1,401Gb of data in a minute, which beat Yahoo's previous record of 500GB in the same time. Microsoft also boasted of sorting the data using fewer resources: The system used 1,033 disks in 250 machines while Yahoo required 5,624 disks across 1,406 machines to complete their operation.

FDS starts with a similar approach as Google's MapReduce -- as it is implemented in Apache Hadoop -- by moving the computational sorting to each individual data server. Unlike Hadoop, however, every server trades information with all the other server in the sorting cluster. The researchers used an additional Microsoft networking technology, called full bisection bandwidth networks, to boost the bandwidth, allowing each computer to both send a receive send up to 2GB per second.

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's e-mail address is Joab_Jackson@idg.com

Follow Us

Join the ARN newsletter!

Error: Please check your email address.

Upcoming

Slideshows

IN PICTURES: Nutanix's .NEXT channel event in Sydney (+20 photos)

IN PICTURES: Nutanix's .NEXT channel event in Sydney (+20 photos)

Nutanix recently held its customer and channel event, .NEXT, in Sydney. The event, held at the Sheraton on the Park saw attendance from more than 150 channel and technology partners and customers. It was the first in a series of events Nutanix is holding in A/NZ in August and September, the objective of which is to brief partners and customers on “what’s next” in the design and management of datacentre technology.

IN PICTURES: Nutanix's .NEXT channel event in Sydney (+20 photos)
IN PICTURES: EDGE 2015 sponsor debrief (+23 photos)

IN PICTURES: EDGE 2015 sponsor debrief (+23 photos)

Some of the sponsors of ARN's inaugural EDGE 2015 event got together at the ARN office for a debrief of the event. Over some drinks and cheese, these attendees got an update on some key statistics that arose from the EDGE event and discussed potential topics and improvements that can be made at next year's event.

IN PICTURES: EDGE 2015 sponsor debrief (+23 photos)
IN PICTURES: ARN Distributor Roundtable, Sydney, 26.08.15 (+26 photos)

IN PICTURES: ARN Distributor Roundtable, Sydney, 26.08.15 (+26 photos)

ARN hosted a distributor roundtable at Cafe Del Mar in Sydney, at which attendees and their partners discussed the changing role of the traditional IT distributor. They spoke about the challenges of digital disruption, the blurring lines of the channel in the age of digital transformation, and examined the ever-evolving business models. This roundtable was sponsored by Distribution Central, Exclusive Networks, Rhipe, and Hemisphere Technologies. Photos by ARN Editorial Director, Mike Gee.

IN PICTURES: ARN Distributor Roundtable, Sydney, 26.08.15 (+26 photos)

iasset.com is a channel management ecosystem that automates all major aspects of the entire sales, marketing and service process, including data tracking, integrated learning, knowledge management and product lifecycle management.

Show Comments