Second Edition of TP Book out Today

It’s hard to believe, but the second edition of Principles of Transaction Processing is finally available. The simple table of contents is here, and the entire front section, including the dedication, contents, introduction and details of what’s changed, is here. The first chapter is available here as a free sample.

Photo of an early press run copy

Photo of an early press run copy

It definitely turned out to be a lot more work than we expected when we created the proposal for the second edition almost four years ago.  And of course we originally expected to finish the project much sooner, about two years sooner.

But the benefit of the delay is that we were able to include more new products and technologies, such as EJB3, JPA, Spring,  the .NET Framework’s WCF and system.transactions API, Spring, SOA, AJAX, REST/HTTP, and ADO.NET even though it meant a lot more writing and rewriting.

The first edition was basically organized around the three-tier TP application architecture widely adopted at the time, using TP monitor products for examples of the implementations of the principles and concepts covered in the book. Then as now, we make sure what we describe is based on practical, real-world techniques, although we do mention a few topics more of academic interest.

The value of this book is that it explains how the world’s largest TP applications work – how they use techniques such as caching, remote communications (synchronous as well as asynchronous), replication, partitioning, persistence, queuing, database recovery, ACID transactions, long running transactions, performance and scalability techniques, locking, threading, queuing, business process management, and state management to process up to tens of thousands of transactions per second with high levels of reliability and availability. We explain the techniques in detail and show how they are programmed.

These techniques are used in airline reservation systems, stock trading systems, large Web sites, and in operational computing supporting virtually every sector of the economy. We primarily use Java EE-compliant application servers and Microsoft’s .NET Framework for product and code examples, but we also cover popular persistence abstraction mechanisms, Web services and REST/HTTP based SOA, Spring,  integration with legacy TP monitors (the ones still in use), and popular TP standards.

We also took the opportunity to look forward and include a few words about the potential impact on TP applications of current trends toward cloud computing, solid state memory, streams and event processing, and the changing design assumptions in the software systems used to power large Web sites.

Personally this has been a great project to work on, despite its challenges, complexities, and pressures. It could not have been done without the really exceptional assistance from 35 reviewers who so generously contributed their expertise on a wide variety of topics. And it has been really great to work with Phil again.

Finally, the book is dedicated to Jim Gray, who was so helpful to us in the first edition, reviewed the second edition proposal, and still generally serves as an inspiration to all of us who work in the field of transaction processing.

Advertisements

11 responses to “Second Edition of TP Book out Today

  1. Congrats Eric! Great to see the inclusion of Spring, EJB3 and REST. I’m interesting in reading those sections.

    Any chance you could do a follow-up posting on your thoughts on TP and REST ?

    Thanks.

  2. Hi Ciaran,

    Thanks very much! Yes, it was lucky in a way that it took so long to finish the draft since we could include that newer stuff…

    I can do more on the TP and REST, sure. It seems sort of like “back to the future” to me inasmuch as the REST approach requires the application to do more, and TP infrastructure to do less, basically.

    The short summary is that in the REST approach, state is never shared between client and server, so locks are never held waiting for a response. But this means the application has to check for the possibility someone else updated the state during the exchange. More soon.

    Eric

  3. Eric

    Looks like an interesting books – I’m going to give it a read and give you my feedback.

    “We also took the opportunity to look forward and include a few words about the potential impact on TP applications of current trends toward cloud computing, solid state memory, streams and event processing, and the changing design assumptions in the software systems used to power large Web sites”

    That is an area of specific interest to me. Have you covered some of the new trends such as:

    1. No to SQL? Anti-database movement gains steam – My Take
    2. Lessons from Pat Helland: Life Beyond Distributed Transactions

    Anyway If your planning a third edition I’ll be more then happy to contribute.

  4. Nati,

    We do recognize and understand the trends away from SQL and 2PC, especially for large web site systems. However when we investigated them, we discovered that everyone currently creates a lot of custom code, and we could not identify common patterns among the applications using some of the techniques you are referring to.

    The book is primarily focused on current practice, meaning we need to be able to include examples of how developers would create programs using widely adopted product APIs to include these concepts and principles into TP applications, and our conclusion is that such products do not yet exist.

    To be clear, some products are beginning to emerge, but the trends are not yet mature enough to clearly predict which of them will achieve widespread adoption – such as SQL, Java EE, or .NET, to name a few.

    The industry is in transition, and it’s too early to focus the book on the new trends, regardless of how interesting they are – and things like Stonebreaker’s paper, and Helland’s papers, and the Dynamo paper are very interesting indeed.

    One reviewer asked us to please invent a new type of transaction coordination for loosely-coupled TP application architectures. As much as we’d like to, proposing a new design is one thing, having it adopted and used in large TP deployments is another.

    We are as interested as anyone in what’s going on with the new database designs, compensation-driven transaction patterns, relaxed consistency, and in-memory replica updating and synchronizing. And these things are all included, yes, but the main focus of the book was and remains firmly on current practice.

    I’m looking forward to your feedback on the book!

    Thanks,

    Eric

  5. ps I should add that the book covers the fundamentals of database and transaction processing – whether a database uses SQL or not, the internal mechanisms and structures tend to be similar – how journaling works, how locking works, transactions, recovery, replication, synchronization, etc. All of these things are independent of SQL.

  6. The book is primarily focused on current practice, meaning we need to be able to include examples of how developers would create programs using widely adopted product APIs to include these concepts and principles into TP applications, and our conclusion is that such products do not yet exist

    I believe that if your referring only the internet companies as a reference then your probably right. Based on my own personal experience majority of the application in the financial world (Order managemetn) are not using the traditional 2pc model and there are products that are well established in that space already and are used to run those financial applications in production in the past few years. A good example is obviously the company that i work for (GigaSpaces) but IBM (ExtremeScale) and Oracle (Oracle Coherence) have a fairly good solution on that space as well.

    The pattern that i often use to describe this model is named Space Based Architecture which can be summarized as follows:

    # Rule #1 – Build your data model from a set of independent data items (Tuples)
    # Rule #2 – Partition your different data items based on your data affinity (Customer-id, account-id,..)
    # Rule #3 – Execute transaction as a set of individual steps scoped within a single partition.
    # Rule #4 – Minimize the amount of moving parts involved under the context of a transaction

    You can read full details here

    Cheers
    Nati S.

    • Nati,

      Yes, we are well aware of the in-memory replication products starting to emerge from what I would call the “scale out” space, which I would also characterize as pioneered by the large web site infrastructures. And I know personally about the adoption of some of these new products, including yours, in the financial services industry to reduce trading latency etc.

      Coherence, GigaSpaces, and ExtremeScale are examples of products that are starting to emerge, but we could not really say that any one of them is either a complete TP environment, or widely enough adopted, for us to use as the basis of examples that illustrate the principles and concepts in the book.

      The focus of the book is on explaining how it all works. We have an entire chapter on replication, which covers the topic you reference in your comment.

      Eric

  7. Eric

    I understand your point of view even though i may have different perspective wrt to the maturity of the new transactions processing models , (XTP) as they are referred to by Gartner.

    I’ll have a deeper look at your book and let you know if i have any further comments.

    Good luck anyway!
    Nati S.

    • ps the point with regard to adoption that we were looking at was whether all of the products solved the problem the same way – which I don’t believe they do. In other words, we can’t yet be sure which one will win.

      • It depends how would you define “the same”.
        All products have similar but different API to solve the same problems.

        All products uses the same principles that i outlined above to handle transactions processing in a scale-out environment.

        So from a pattern perspective the way all products deals with XTP is pretty much the same. We all gone through various competitive situations and were forced to converge on those terms simply since most customers want the same thing when it comes to transaction processing and most customers want to at a position where they would be able to compare all various solutions.

  8. Nati,

    I think the question is where to draw the line. I am personally very interested in the new models, and think they have a lot of potential.

    Yes, please let us know if you think we haven’t covered the material correctly. It’s always good to hear another point of view!

    Thanks,

    Eric

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s