Software Productivity

The biggest problem for business and industry today is software productivity. It is just too expensive and difficult to develop and maintain the necessary IT applications and systems. And yet everyone depends upon computers to operate their businesses.
View image
Companies have literally no idea what their IT spend should be relative to other corporate expenses, and have difficulty understanding the relationship of IT spend to strategic corporate initiatives. IT projects sustain failure rates that would never be tolerated in any other functional department. Nobody understands the relationship of the high cost of enterprise IT to its benefits.
The solution to the problem is easily stated but very hard to achieve. The IT industry needs meaningful enterprise software standards.
View image
We have seen the “Web of documents” (aka the World Wide Web) succeed beyond anyone’s wildest imagination.
View image
But the Web of data and services for enterprise software systems remains brittle, inflexible, and a mixed bag. Companies still have to deal with the classic spaghetti problem.
View image
While HTML and HTTP have provided consistent standardization for human to computer interaction, and today you can find or buy almost anything on the Web, standards for program to program communication are not yet widely and consistently implemented. Companies still can’t connect their departments or assimilate aquisition applications easily enough or for a reasonable enough price.
The reason IT cost remain so high is labor. Desipte all its advances, software remains a craft industry, relying upon the skills of individuals to bridge the gap between technology and business.
View image
Scientific and industrial methods need to be applied to IT software. It should be possible to take a given set of inputs and match them predictably to a desired set of outputs. It should be possible to develop a set of reusable, interchangable software components and assemble them rapidly into finished applications.
View image
The key to all of this — the mass assembly of software, the introduction of scientific methods and predictable projects, the ability to easily understand and reasonably predict the value of IT investments — is the standardization of application endpoints, or interfaces.
Historically, many attempts have been made to standardize software: COBOL, C, C++, Java, CORBA, MIA, SPIRIT, J2EE, DCE, DCOM… All of these efforts tried to unify the computing world using a single technology. Web services are different. WSDL and SOAP, for example, were written to be communication protocol independent. Meaning these standards acknowledge and embrace technolgoy diversity.
Also, the industry itself appears finally ready for standardization. Software is not changing as rapidly as it was a decade ago and it is starting to commoditize. Companies today have invested in enough IT assets and instead need a better way to use them — a way to improve on what they have. They don’t need more foundational software such as application servers, EAI brokers, database management systems, or packaged applications. Instead they need an architecture (SOA), the standards (WSDL), and the tools (a WSDL based ESB for example) to rapidly expose services and predictably and reliably assemble them into composites.
I gave a version of this presentation at the W3C Advisory Committee meeting in Montreal last December. The occasion was a panel discussion about the future direction of the W3C. I was basically proposing that the W3C shift its investments toward Web services and help solve the problem of enterprise software standardization. Fixing the problem would be the biggest impact the W3C could have on business and industry, and therefore on the world’s economy. Several representatives from other companies agreed with me. I don’t know what, if anything, will come of it but I know they are considering the idea.
View image
In the early days of Web services I used to think that this would be the basis of a unifying architecture across the enterprise firewall, unifying the Web of documents with the Web of data and services (to use the W3C terms).
Today I’m not sure it really makes sense to think about one Web like this. Maybe we need to define a different Web for the exernal world, and another for the internal world of enterprise software. Then we could talk (for example) about connecting REST and WS-* rather than debating which is the best. Maybe this would make it easier to finally achieve the goal.
Anyway, as I told Tim Berners-Lee after the panel discussion, I am an optimist, and I am hopeful we can do something to fix this problem.

5 responses to “Software Productivity

  1. Not sure if this is about programmer productivity –in the application space– or about the lack of really good tools with which application designers can build applications.
    Writing lots of code seems to me to be something most of us should no longer need to do. We should concentrate on the business logic, making use of the service-oriented nature and way of thinking most people devising new applications actually think.
    With the advent of “managed code” systems such as Java and .NET, and their inherent ability to both optimize and change things “on the fly”, it seems time that we should stop having to write application code and start concentrating on getting business functions designed and generated.
    An additional factor is the use of existing, 3GL code here. There is a lack of tools –IDEs, SCCS, build utilities, test and deployment, management– for environments where 3GL code is being used together with new code, be it .NET, Java, Ruby or whatever.
    Last but not least we are spending far too much time trying to get databases to fit the service-oriented model and often end up doing the opposite, i.e., trying to make the service model fit the database.

  2. John – Great observations. It is true that the industry still focuses too much on tools for a specific environment. The promise of Web services, which is not yet fully realized, as you basically point out, is that these standards have the potential to provide a uniform XML layer across all types of software systems – procedure oriented, database management systems, middleware, scripting languages, object oriented languages etc. Once that is done the problem should move to the next level – i.e. the vendor value add should build on that layer and provide tools for it so that we can interact with the XML layer rather than dealing with all the various underlying types of languages and systems.
    And today we can find implementations of Web services for each of these types of environments. You can use XML with anything. But it is difficult finding consistency among these, or tools that work across multiple environments. Standardization should be the foundation for fixing these problems. Let’s hope so anyway.
    Actually I believe the economic costs of not adopting standards are prohibitive. I don’t think it’s a matter of optimism but a practical reality. It is just a matter of time. The question is how much ineffeciency and unnecessary cost can business, industry, and government afford until then.

  3. There is not really a big benefit to communications standards, without a standardization all the way back to the programming language and associated tools. Why? Because then we have multiple independent, language dependent tools that will have proprietary solutions, or at least implementation specific “features” that create incompatibilities.
    Also, reducing the choice of communications format to just XML based content, still leaves the type of transport as a barrier. It won’t matter if a device does XML if it needs USB, and you only have ethernet. So, web services are not a panacea. It won’t solve all your problems.
    In terms of software lifecycle, you still have to be able to run that application somewhere. So, portability of software is a predominate issue. If the software application is written for one platform/OS, and the next generation of devices uses a different platform/OS, how will a communications standard improve the life of the software, and manage your investment in that application.
    We still have a very high cost of software development related to choosing a particular programming platform/language and associated toolset. The productivity of the programming environment is still an overwhelming cost issue.
    Yes, Java’s platform neutrality and the focus on neutrality in the APIs’ design is an important part of future software systems. The notion of a Service Provider Interface allows different vendors and users to augment the system with their choice of implementation. This is a very enabling technology.
    Lots of different developments are pending in the software industry. Many users are waiting for vendors to solve their problems using standards created by another group of individuals, some with commercial interest in those standards.
    What the software development community needs to do, is to vote with their mouths and their feet to make it clear to vendors where they feel the most value is added.

  4. Absolutely. The productivity and cost issues hit the end users the hardest, and that’s where the pressure for standardization will inevitably come from.
    Web services may be subject to over-hype and under implementation, but the importance of standardization at the interface and interoperability level promises the biggest benefit.
    It is also true that if all computers were to run the same operating system, and all developers use the same programming language, we would also achieve the benefits of standardization without sacrificing operating system or language features.
    But the computer industry isn’t homegenous and it’s unlikely it ever will be, at least not in the foreseeable future. So we need some glue and standards for fastening disparate pieces of software so they can work together.
    A key question is where to draw the line between what needs standardization and what doesn’t. In automobile manufacturing standards are necessary for assembling interchangable parts. But standards aren’t necessary for determining the size of the steering wheel, placement of pedals and levers etc.
    It’s important to divide the problem between standards required for diverse parts of technology to work together in an industrial sense, and conventions that support usability of applications for less technically trained people. In one case it is necessary for people to adapt themselves to the technology – i.e. to get trained on how to fit things together, and in the other it’s necessary for the technology to adapt to how people do things.
    The point is that it isn’t necessary to standardize everything, just those key areas that facilitate productivity. And yes, it needs to be comprehensively adopted.

  5. I’m sorry but due to the daily effort of cleaning up spam comments and pings I’m going to close comments and pings for this entry.
    Usually I close them after a month.
    If anyone wants to post a comment after that please email me and I can open it up again.