Evidence Soup

According to McKinsey, the evidence says that cloud computing might have a cloudy future. Others disagree.

A McKinsey report, Clearing the Air on Cloud Computing, finds that "While it has great potential, many of the claims being made about cloud computing have lead some to the point of 'irrational exuberance' and unrealistic expectations. The purpose of this report is to focus the nascent cloud industry and its consumers on setting realistic expectations...." I've taken a look at their evidence, and so have many others. While McKinsey does provide valid evidence, they've apparently overlooked key parts of the equation. The report (pdf) is labeled as a "confidential discussion document", but it's available on the Uptime Institute web site. Shown here are numbers for per-unit costs for amazon's Elastic Compute Cloud (EC2) service.

¹ Cost for comparable configuration (75% of EC2 Large Standard Windows)
² Typical CPU/month cost for 3GHz dual-core Xeon Windows-based servers
³ Estimated based on 10% labor savings (from page 24)

Under a heading Current cloud computing services are generally not cost effective for larger enterprises, the report concludes that "Most EC2 options are more costly than TCO [total cost of ownership] for a typical data center" (page 23).

 

McKinsey shows "the thin green line" = TCA for typical data center around $45/month for CPU equivalent, where "Total Cost of Assets for 'typical' data center: 10% utilization, $20M/MW for facility, $.1kW-hour, $14K/Server (2 CPU, 4 core)."

A Wall Street Journal piece, Cloud Computing Could Cost More, McKinsey Says (behind paywall) says that "Using the model of Amazon.com Inc.'s (AMZN) cloud service, McKinsey said an actual customer, whose identity it didn't disclose, more than doubled the cost of technology by using a cloud service, compared with a typical corporate data center.... Some companies offering cloud services challenged McKinsey's findings.... IBM, which is pushing cloud services, said in an emailed statement that the report failed to consider that most companies would only use cloud services for a small portion of their technology needs." But that's not much of an argument -- as pointed out by Joe Weinman on GigaOM, "Unfortunately, that sounds a little bit like the argument that burning hundred-dollar bills is fine as long as you do it for only a portion of your portfolio. But there’s another reasons the McKinsey report missed the mark: Unit cost doesn’t matter — total cost is what counts."

There's other evidence to consider. Weinman continues: "I’ve argued the same thing before: It isn’t clear that alleged economies of scale exist for cloud service providers, since they use essentially the same building blocks as enterprises. The conclusion McKinsey draws from this fact is this that while cloud computing and services may make sense for smaller companies, enterprises should hold off until the financial disparity is resolved. But wait a minute.  If McKinsey had studied, say, automotive transportation, the conclusion they might have come to would be that, since enterprises and “cloud service providers” — rental car companies — use the same infrastructure components — cars — there really aren’t any economies of scale and the service provider premium then makes rental cars financially unattractive.... So wouldn’t the analogous conclusion then be that enterprises should be wary of using rental cars, until that industry comes up with a better value proposition? ...In other words, sure, service providers typically cost more for infrastructure on a unit-time basis.  So?  That’s not the relevant comparison, total cost is."

Innovation and flexibility are also important. As discussed on TechCrunchIT, Rajen Sheth, Google’s Senior Product Manager of Google Apps, responded in the blog post What we talk about when we talk about cloud computing: "While the cost advantages of cloud computing can be great, there's another advantage that in many ways is more important: the rapid pace of innovation. IT systems are typically slow to evolve. In the virtualization model, businesses still need to run packaged software and endure the associated burden. They only receive major feature enhancements every 2-3 years, and in the meantime they have to endure the monthly patch cycle and painful system-wide upgrades. In our model, we can deliver innovation quickly without IT admins needing to manage upgrades themselves. For example, with Google Apps, we delivered more than 60 new features over the last year with only optional admin intervention. The era of delayed gratification is over – the Internet allows innovations to be delivered as a constant flow that incorporates user needs, offers faster cycles for IT, and enables integration with systems that were not previously possible. This makes major upgrades a thing of the past, and gives the customer greater and greater value for their money."

ROI calculator looks at the evidence about health quality improvements.

To reform the U.S. health care system, evidence-based medicine can't just be about clinical findings -- it also needs to be about evidence of cost-effectiveness: Making hard choices to improve quality and get costs under control. The Robert Wood Johnson Foundation has funded an ROI Forecasting Calculator, developed by the Center for Health Care Strategies and intended to help Medicaid agencies & health plans assess quality initiatives.

What's this? I'm using Tiny Soapbox to summarize key points about the ROI calculator. This is an approach I use in my work, so it's easier for people to quickly assess ideas, findings, and insights. As shown  below, each Tiny Soapbox describes the connection between a specific action or occurrence and an associated outcome or result: Supporting evidence and analysis are transparent, and links to detailed information are provided.

 

To advocate adoption of the ROI calculator, CHCS provides a policy briefing: Maximizing Quality and Value in Medicaid: Using Return on Investment Forecasting to Support Effective Policymaking (10-page pdf here), funded by the Commonwealth Fund -- it says that "Return on investment (ROI) analysis, which has been used in the private sector for decades to inform the allocation of limited resources, is a useful technique for supporting cost-effective policy decisions.  Effective allocation of limited Medicaid resources is a policy imperative, and the use of ROI forecasting can ensure responsible stewardship of taxpayer dollars...."

Continuing the briefing: "There are a number of specific policy decisions in Medicaid that are ripe for ROI analysis. This policy brief describes how Medicaid stakeholders can use return on investment analyses to support value-based purchasing decisions. It outlines a number of specific opportunities where Medicaid stakeholders can apply ROI forecasting, including:

1. Assessing the relative value of various policy options to efficiently allocate limited resources;
2. Supporting evidence-based policymaking through the identification of proven interventions and analysis of program data; and
3. Informing financing reform by identifying opportunities to realign incentives to promote high quality, cost-effective service delivery."

Regarding the ROI calculator, CHCS says "Medicaid presents tremendous opportunities to demonstrate the business case for improving health care quality. Of the more than 60 million people served under Medicaid, adults with disabilities and chronic conditions represent only one quarter of the population, while consuming nearly 70 percent of Medicaid’s resources. Investing in programs that more effectively manage the care of these high-risk, high-cost beneficiaries can improve health outcomes, reduce unnecessary utilization, and control Medicaid expenditures."

To help Medicaid stakeholders demonstrate a return on investment (ROI) from quality-improvement projects, CHCS developed the ROI Calculator for Quality Initiatives (chcsroi.org). Besides ROI and NPV, the calculator supports scenario testing and sensitivity analysis. A User's Guide to the ROI Forecasting Calculator: Estimating ROI for Medicaid Quality Improvement Programs is available (24-page pdf here). Here's the information needed to use the calculator:

Intervention	Clinical focus Intervention strategies Timeframe/duration of initiative
Target Population	Targeted population subgroups Disease prevalence among target groups Expected enrollment rate Risk stratification
Utilization	Average 12 month baseline costs for intervention group members Trend (expected growth in claims costs) Anticipated utilization changes resulting from initiative
Program Costs	Estimated costs of launching and operating the initiative
Discount Rate	Organizational cost of capital

They saved the best for last. In the section titled Communicating ROI Analyses, CHCS says "When sharing ROI forecasts with colleagues, managers, government officials, or other stakeholders, it is important to keep a number of considerations in mind.... One of the key benefits to using the ROI Calculator is the transparency it creates for sharing analyses with broader audiences. Accordingly, the ROI Calculator can be used to foster dialogue with multiple stakeholders who may be interested in the prospective financial impact of a given quality initiative. When communicating ROI analyses conducted with the ROI Calculator or elsewhere, users are encouraged to provide maximum transparency including:

• Description of the methodology used to calculate costs and savings;
• Justification for key forecast assumptions;
• Identification of sources of uncertainty in the estimates; and
• Indication of how such uncertainty may have been accounted for through scenario testing or sensitivity analyses.

"...For example, conversations could concentrate on expected participation rates in the intervention, or around the anticipated effects of the intervention on emergency department use – instead of on skepticism around methodology and concerns of 'funny math.'" Transparent evidence for Medicaid decision-making? I hope it happens, and soon.