Featured Project
Join Red
The RedTie Revolution!

The Red Shift Theory

By Richard Martin, InformationWeek
Aug. 18, 2007
URL: http://www.informationweek.com/story/showArticle.jhtml?articleID=201800873


Greg Papadopoulos, CTO Sun Microsystems Red-shifting companies will experience explosive growth, predicts Sun's Papadopoulos (Photo by Gabriela Hasbun.)

At the same time, companies faced with the rising costs of powering, cooling, and maintaining racks of servers in conventional data centers are stretching their IT resources beyond capacity--and looking for alternatives. Papadopoulos uses an energy-utility metaphor to capture this shift: "Why build your own generator in your back yard when you can plug into the energy grid?"

 Red Shift Redefined

• Red shift refers to companies experiencing exponential growth in demand for raw computing power

• Red-shift companies tend to be Web 2.0-focused like YouTube and MySpace, or big financial, energy, and pharmaceutical companies

• Those companies, Sun CTO Greg Papadopoulos says, will experience similarly high levels of growth in users, revenue, etc., while blue-shift companies will grow relative to GDP

• Along with the cost of powering and cooling in-house data centers, the red shift is driving a surge in utility computing and software as a serviceDatabase calls have gone from millions to billions, says Salesforce's Fisher Red-shifting companies will experience explosive growth, predicts Sun's Papadopoulos



An example of a red-shift company is Twitter, an instant-update social networking service that lets users post brief messages to the Web from cell phones or PCs. The number of Twitter users, now at more than 50,000, posting 30,000 telegraphic updates a day, is doubling every two to three weeks, according to Obvious Corp., the startup that launched the service in July 2006.

Hoping to grow into that neighborhood is GigaVox Media, a company that provides an Internet-based production and distribution platform for podcasters and video bloggers. "Like all startups, we envision ourselves as the next YouTube," founder Doug Kaye says with a laugh. What does that mean in terms of processing requirements? "I don't know," admits Kaye. "But we have to have that level of scalability."

To ensure that GigaVox does, Kaye has chosen Amazon Web Services, an on-tap application, storage, and computing infrastructure. That's one way red-shift companies are adapting--some, like Amazon and Google, are building supersized IT infrastructures, while others, like GigaVox, are tapping into them.

Red shift presents more complex quandaries for large companies. Papadopoulos began to understand the phenomenon, he says, in conversations over the last 18 months. "I would hear this conflict directly from customers, who would first say, 'I've got this really poor utilization--only 10% or 15% of our data center capability is really being used,'" he recalls. "And then I'd hear, 'I'm running out of data center space and power. I don't know how I'm going to continue to manage the growth,' and so on. On the one hand, the infrastructure is being poorly utilized, yet you're bursting at the seams."

The aha! moment came when Papadopoulos realized that there are really two different application sets driving computing demand: one consisting mostly of newer Web-facing applications driving exponential growth in both user demand and computing requirements; the other comprising back-end systems that are growing at more historical rates. "All this is really about which side of Moore's Law you're on," Papadopoulos says. "If your applications are growing faster than Moore's Law, you've got a fundamental set of issues about scale and power. If they're growing slower than Moore's Law, you've got all kinds of opportunities around consolidation."

It's important to emphasize that these rates of growth have seldom been seen in the computing world, even in the early days of the Internet boom. In August, for example, a little-known startup called Yoomba, which offers a peer-to-peer application that lets users place voice-over-IP calls or send instant messages to e-mail addresses automatically, announced that it had signed up 500,000 people in less than a month.

And it's not just user numbers that have IT infrastructures at Web 2.0 companies bursting at the seams. Research firm ComScore says Americans watched more than 8.3 billion video streams online in the month of May--nearly a video per day for every man, woman, and child in the United States. The largest provider, of course, was YouTube, which served up some 1.7 billion videos during that month--a crushing burden for any storage and networking system, even one as expansive as that of Google, YouTube's owner.

Mainstream companies also are feeling the processing pinch. Arizona Federal Credit Union, a midsize financial institution that services 244,000 consumers and small-business employees, is growing about 25% a year. In itself, that doesn't outpace Moore's Law. (In its original form, Moore's Law said that the number of transistors on a chip would double every two years, which is an annual growth rate of about 42%.) Expanding faster, though, is Arizona Federal's credit card division, a data-processing-intensive business that's growing at "an exponential rate" in terms of cards outstanding and transactions per card, says CIO James Phillips. Putting a strategy in place for that kind of growth has been Phillips' primary task since he joined the company in 2005.

Virtualization software let Phillips reduce the number of servers in Arizona Federal's data center to 61 from 87 over the last year, but that sort of consolidation won't keep up with the requirements he foresees in the next few years. "We will continue to expand our data center, but as it reaches maximum capacity we've got to make some long-range decisions," Phillips says. Over time, that could mean moving up to 30% of Arizona Federal's IT infrastructure to hosted services accessed over the Internet, he says.

What if you're much, much bigger? What if you're Visa, one of the world's largest transaction processors?



Red Shift