SQL is Dead

Last week while talking with Frank Martinez during the BEA show (and this is really one of the great things about my job, that I get a chance to meet and talk with interesting people like Frank), I started on a version of my mini-rant about SQL and GUIs being labor-draining anachronisms. Frank basically agreed, and that’s about all the encouragement I need to try to write it all down here ;-).
The point is that it just takes too much time and effort to design, develop, and adminster a database and associated GUI. Who can really afford to take on all the data modeling, normalization, and structuring work necessary to define a sensible database and usable GUI?
The benefits seem very small given the huge amount of effort. After all, most database applications just extract the variable data items from the static fields, figure out how to store them in the database, and then retrieve them so they can “rehydrate” them back into the GUIs from which they were extracted.
This all started when storage space was tight in the early days of business computing, when every byte counted and we (all) did things like store “82” instead of “1982” and measured program load sectors in 16K chunks (for example). Processing power was non-existent on the desktop, and even the servers were constrained. Data and its presentation needed to be managed carefully to maximize hardware and software capacity.
Today storage space is essentially free, and there’s an abundance of cheap and available processing power from the desktop to the mid tier to the back end. Why not just store the data along with its formatting information in an XML file and just render it when it needs to be displayed or interacted with? Just store everything — variable and static — in the XML documents and XML Forms and eliminate all that unnecessary database and GUI design and development labor.
Of course, SQL is really no more dead than CORBA, Java, or the mainframe — at least not yet.
But there is no longer any good reason to assume that structured data is the right solution for every data management problem. A large number of applications manipulate data intended for use by one person at a time. A database and GUI application is really overkill for that.
On the other hand, applications like airline reservations and buying concert tickets will probably always need structured data, since they rely upon the ability to accurately manage shared access to a single data item instance (i.e. the airplane or concert hall seat) so that it is sold only once and on a first come first served basis.
Applications like purchase order approval, medical records distribution, expense report filing, engineering and repair drawing distribution, safety inspections, and so on, really do not derive any benefit from the significant effort required to structure their data.
These are also the applications that are increasingly important to “mobilize,” and structured data is not the right approach, not just because of the excessive labor involved, but also because the database replication and synchronization problem is basically unsolvable over a WAN.
This isn’t to say that structured data isn’t important, but it is more important for field workers to have the ability to orchestrate the distribution of unstructured, personal content to the right individual at the right time, and SQL just can’t do the job.


5 responses to “SQL is Dead

  1. Provocative suggestion, Eric – if aired in the right quarters (i.e. not Iona’s Web site) it could result in unparalleled numbers of puce faces and outraged rebuttals. In most organisations the data management hierarchy has a standing similar to a priesthood, and the relational model is their dogma.
    In view of the vast increases in processing speed, storage capacity and network bandwidth, it makes sense for us all to look for new optimisations. But I am not sure that storing whole GUI pages is the right idea. Surely it is a useful design principle to uncouple the raw data from any particular display format? If we have so much computing power, what is the objection to repeatedly storing and retrieving the data, “relational garage” style?
    If your suggestion were adopted (if I understand it right), developers’ work would actually be made harder. Every time they wanted to tweak a form, they would have to change the database layout too.

  2. i agree with you about storage being free, but i think the main reasons for sql surviving (aside from inertia) are:
    1) search time is not free. trying o search through a million records does require organized data (unless you have a server farm like google behind it building the meta-data and indexes for you);
    2) portability. by having normalized data and all that, it is possible to reuse the little chunks.
    however, storing say the presentation and entry through xml is only useful if you do not have normaized data. you really need a layer on-top of the most efficient way to store the data. I think tools like hibernate really help to abstract the database away even further. sleepy cat’s berkely db now supports native xml sotrage too.
    basically what i am saying is we need a conceptual layer (the xml layer) that can be mapped by software (And maybe hand-tuned) to the most efficient storage mechanism, so if you are using a relation db it will be autonomalised, but on a filesystem it will directly written out.

  3. Tom & Aman,
    My apologies for taking so long to respond. I’ve been trying to finish up a book and was on vacation for a week.
    I think the title is a kind of exaggeration. I don’t really think SQL is dead. Nothing is really dead in software. Not CICS, not IMS, not VMS, MPE, or Tandem NonStop. Not COBOL, not PL/I, not C or C++. Not even Java. But people often say that when something new comes along.
    In this case, I think the industry needs to think long and hard about the idea of storing everything in SQL. It’s not always practical to store XML in SQL format. It’s not always a good idea to store a file as a blob in a database.
    I think we need to be clear that for certain types of user interactions, especially those involving data intended for a use by a single person at a time, SQL is not the right format. The data does not need to be artifically or unnecessarily structured simply because it needs to be persisted.
    For lots of data it makes sense to expend the energy and design time on normalization, indexing, buffering, and page layouts. But for lots of other data it makes more sense to avoid all that time, trouble, and expense when all that’s really needed is a way to get the data to the user, present it, capture inputs, and send it along to the next user who needs to see it.
    Database not needed in that case. XML is enough. Therefore SQL is dead šŸ˜‰

  4. Thanks for your reply, Eric. I see where you are going, but – although I Am Not A Database Expert – I think you may be over-simplifying the issue. First, this may be nit-picking, but SQL is not a storage format. It is a declarative language for telling a reasonably intelligent DBMS what you want it to do for you. How the DBMS does that is mostly up to it. From this if follows, secondly, that efficient RDBMS do not store data on disk in anything remotely resembling SQL, or for that matter tables. The funny thing is that SQL is a (relatively) invariant interface between two systems (the database and the application) that have much more freedom to change. But surely the reason why SQL is invariant is because everyone wants to have a single standard that they can rely on! It’s not ideal, of course, but that is an inevitable consequence of trying to be all things to all people.

  5. Tom,
    Thanks for the comment.
    You are right about the over simplification, but otherwise it’s kind of a long story šŸ˜‰
    First, SQL is two things – a storage format and a programming language to interact with the storage format. And the storage format, the data definition language part, is mapped to physical storage. During database creation, the administrator defines the database size and extensibility parameters. The database software then goes out and maps disk sectors to allocate the requested storage. The DDL defines the layout of the tables for that storage, including the fields, rows, columns, and indexes that establish relationships. The magic of relational database is that they store the pointers to the tables separately from the data.
    Once a database is created, database programmers embed SQL in their code, or call stored procedures to manipulate the tables. The structure of the tables needs to match the requirements of the applications for storing and retrieving data, and there’s usually a lot of back and forth over this since it’s hard to get a database design right. For example, it’s hard to know the best way to store the information in a table for optimum retrieval later. Should you use a hash index or a B-tree?
    The practice of structuring data dates way back to the early days of computing of course, and SQL databases are just kind of the most recently popular way to do it. I always hate to bring up how long ago I started in this business, but some of the first programs I wrote had to put data in the right structure and order for access via sequential tape.
    The point is that we have been living with limitations of technology, and structuring our data accordingly forever in this business. Isn’t it about time to look at all the CPU, disk, memory, and network bandwitdh we have and say “enough is enough”! I mean, all that time and effort over structuring data to make the most effecient use of system resources, or to get the software systems to perform as needed given their limitations – come on, let’s move on!
    The typical IT project involves gathering requirements for the application, desiging a form or GUI for interacting with the user, designing the database in which to store the data and from which to retrieve it, and writing the programs to join the two.
    I don’t know about you, but it doesn’t seem to make a whole lot of sense to me anymore to look at all applications this way. All you’re doing is taking a bunch of dynamic fields out of another bunch of static fields, organizing the dynamic fields into rows to insert in one or more SQL tables, and then later on dragging the fields back out of the database and repopulating the form or GUI with the same information. That seems like an awful lot of time and effort to spend on design, structuring, and programming when you could just be creating an XML document instead.