Monthly Archives: October 2007

HPTS 2007 – time to reinvent everything?

It’s been a busy week between HPTS and the most recent OSGi enterprise expert group meeting. This entry is about HPTS. I’ll post another one about the OSGi meeting.

It was pretty clear from the presentations and discussions at last week’s HPTS Workshop that database and middleware software vendors are failing to meet the requirements of Web businesses, both large and small.


My room in the Julia Morgan designed Scripps building at Asilomar Conference Center

During the tribute to Jim Gray Tuesday evening Pat Helland showed a great video interview with Jim in which he mentions the need to scale things out, not up.

Pat Helland introduces the tribute to Jim Gray

The “scale up” thinking has a limit – there is always such a thing as the biggest box you can buy. But “scale out” does not have any limits – you can always add more computers to a cluster. The presentation from Google in particular was a really interesting explanation for some of the techniques they are using to achieve a massive scale out. Pat’s summary of it is something like “people are willing to trade consistency for latency.” And because current products are designed around the “scale up” concept, they don’t tend to work very well in the modern “scale out” environments.

One disappointment was that the scheduled presentations from were pulled at the last minute. But the other presentations from Yahoo, Betfair, PayPal, Second Life,, Rightnow, Workday, Google, and eBay all made it pretty clear that Web based businesses are developing custom solutions because off the shelf software currently does not cut it for them.

Aaron Brashears of Second Life

Mike Stonebreaker‘s presentation contained perhaps the most direct challenge to the status quo, arguing that industry changes during the past 30 years have invalidated the relational database and SQL. This provoked some lively discussion since among the attendees were several people who helped define and create the relational database and build DB2 and Oracle. One of his arguments is that the relational database just isn’t fast enough for modern requirements.

Some great discussions broke out between the representatives of the Web companies and the folks working for established software companies. For example, one of the Web guys commented that they did not want to have to buy the whole set (i.e. complete product) when all they really needed was a paring knife, yet the only choice they are offered is to buy the whole set. (And here OSGi has some potential to change that dynamic – but more about that in the next post.)

“Bloatware” is definitely part of the issue. Ironically so are many of the things a lot of us have worked to define, create, and promote during the past two or three decades around guarantees of atomicity and isolation. None of the Web companies use distributed two-phase commit, for example. They only use it to deque an element and update a database in the same local transaction. So much for all that work on WS-TX! 😉

Anyway, the idea seems much more to be how to replicate data across multiple computers, so that if one of them crashes another one can take its place, to write programs assuming that failures will occur early and often (since they are going to occur, why not assume they will occur often?), and to allow the replacement of disks and computers as needed without taking the application down.

It will be very interesting to observe over the next few years the extent to which the ideas and techniques in the custom built solutions become more widely adopted and incorporated into commercial products. One of the inevitable questions, as raised during the discussions, is how broad the market is for such things as Google’s file system and big table, or Amazon’s S3 and Dynamo.

In the old days we always talked about moving mainframe based systems to distributed environments, but maybe our mistake was in trying to replicate the features and functions of the mainframe. These new “scale out” architectures may finally accomplish it.

Some of the way too many photos 😉

View from outside the dining hall

Me and Ed Lassettre

One of the boardwalks through the dunes between the conference center and tbe beach

View of Monterey Bay from one of the boardwalks

Down at the beach, view from the end of the dunes

Down at the ocean, the tide coming in around the seaweed

Up on the path someone walks his dog

Through the rocks, the clouds and the surf

The path beside the beach, looking north

Surf and mist, looking southward

The Julia Morgan designed chapel, where the workshop is held

Assembly Required

Some of the greatest advantages of SOA derive from adopting interface contracts. (Of course the benefit of interface contracts isn’t limited to SOA, but they are definitely applied to good advantage in an SOA environment).

By the way the definition of SOA that I use is that it’s a style of design, not a technology. The design is implemented using one or more technologies.

One of the crucial aspects of any SOA design is the contract between the service requester and the sevice provider, which is typically expressed (at least in part) using an interface definition language such as IDL and/or WSDL.

Among the many benefits of designing contracts and defining interfaces is the ability to divide up the work. Once the contract is in place (at a minimum this would be the definition of the data to be shared, often expressed using XML) separate teams can go off and work in parallel to develop the service provider and requester.

This approach can scale effectively as the library of reusable services grows, dividing up the work across multiple development teams, perhaps in different geographies. And this approach also creates the possibility that the work can be divided across companies, too. And that it could more easily be outsourced and/or offshored.

But with all the great potential for dividing up the labor involved in IT development comes an equally great responsibility to ensure that things go as planned when the divided up work gets assembled. Someone has to ensure that the interface definitions are interpreted and implemented consistently, and preferably before the whole application is deployed.

The folks at Business Management recently interviewed me about this. We have found, through sometimes painful experience, that it is best to validate individual interfaces as they are created rather than wait until final assembly. Many of our customers are starting to realize they need to change their approach to application assembly when using an SOA based approach.