Daily Archives: October 14, 2007

HPTS 2007 – time to reinvent everything?

It’s been a busy week between HPTS and the most recent OSGi enterprise expert group meeting. This entry is about HPTS. I’ll post another one about the OSGi meeting.

It was pretty clear from the presentations and discussions at last week’s HPTS Workshop that database and middleware software vendors are failing to meet the requirements of Web businesses, both large and small.


My room in the Julia Morgan designed Scripps building at Asilomar Conference Center

During the tribute to Jim Gray Tuesday evening Pat Helland showed a great video interview with Jim in which he mentions the need to scale things out, not up.

Pat Helland introduces the tribute to Jim Gray

The “scale up” thinking has a limit – there is always such a thing as the biggest box you can buy. But “scale out” does not have any limits – you can always add more computers to a cluster. The presentation from Google in particular was a really interesting explanation for some of the techniques they are using to achieve a massive scale out. Pat’s summary of it is something like “people are willing to trade consistency for latency.” And because current products are designed around the “scale up” concept, they don’t tend to work very well in the modern “scale out” environments.

One disappointment was that the scheduled presentations from Amazon.com were pulled at the last minute. But the other presentations from Yahoo, Betfair, PayPal, Second Life, Salesforce.com, Rightnow, Workday, Google, and eBay all made it pretty clear that Web based businesses are developing custom solutions because off the shelf software currently does not cut it for them.

Aaron Brashears of Second Life

Mike Stonebreaker‘s presentation contained perhaps the most direct challenge to the status quo, arguing that industry changes during the past 30 years have invalidated the relational database and SQL. This provoked some lively discussion since among the attendees were several people who helped define and create the relational database and build DB2 and Oracle. One of his arguments is that the relational database just isn’t fast enough for modern requirements.

Some great discussions broke out between the representatives of the Web companies and the folks working for established software companies. For example, one of the Web guys commented that they did not want to have to buy the whole set (i.e. complete product) when all they really needed was a paring knife, yet the only choice they are offered is to buy the whole set. (And here OSGi has some potential to change that dynamic – but more about that in the next post.)

“Bloatware” is definitely part of the issue. Ironically so are many of the things a lot of us have worked to define, create, and promote during the past two or three decades around guarantees of atomicity and isolation. None of the Web companies use distributed two-phase commit, for example. They only use it to deque an element and update a database in the same local transaction. So much for all that work on WS-TX! 😉

Anyway, the idea seems much more to be how to replicate data across multiple computers, so that if one of them crashes another one can take its place, to write programs assuming that failures will occur early and often (since they are going to occur, why not assume they will occur often?), and to allow the replacement of disks and computers as needed without taking the application down.

It will be very interesting to observe over the next few years the extent to which the ideas and techniques in the custom built solutions become more widely adopted and incorporated into commercial products. One of the inevitable questions, as raised during the discussions, is how broad the market is for such things as Google’s file system and big table, or Amazon’s S3 and Dynamo.

In the old days we always talked about moving mainframe based systems to distributed environments, but maybe our mistake was in trying to replicate the features and functions of the mainframe. These new “scale out” architectures may finally accomplish it.

Some of the way too many photos 😉

View from outside the dining hall

Me and Ed Lassettre

One of the boardwalks through the dunes between the conference center and tbe beach

View of Monterey Bay from one of the boardwalks

Down at the beach, view from the end of the dunes

Down at the ocean, the tide coming in around the seaweed

Up on the path someone walks his dog

Through the rocks, the clouds and the surf

The path beside the beach, looking north

Surf and mist, looking southward

The Julia Morgan designed chapel, where the workshop is held