At this point, we had established how this metadata could be the condensation formed from the vapor of our thoughts, especially when it came to the overall design of the model and functionality within the architecture. By molding both the tables and the files through a series of iterations, we were routinely applying a system that seemed to work methodically and successfully. At this point of the design, we were optimistic enough to start discussions about another important aspect of our distributed architecture: communication. Namely, how were the various processes (which were executing on heterogeneous systems) supposed to communicate with one another? Almost a decade ago (when these design sessions occurred), web services were not as popular as today, and so they were not given much consideration. (Due to the volume of data that was expected to pass between machines, web services were not favored as an option anyway.) Since there were no cheap options available for use and since we were limited by a miniscule budget, we decided to create our own mechanism for communication. And how would we do that? To that general question, we again gathered around the runes of our metadata and peered closely in order to decipher the needed answer to our inquiry.

Since XML had already been used as a markup language both to showcase the metadata and to help specify some of the project’s needed functionality, a few of us proposed that we nominate it as a candidate for a homegrown protocol. At the time, XML and JSON (along with other contemporary standards like REST) had not yet become normal options for communication, but we pushed for such an adoption. Some senior managers had concerns, but for the most part, they were very intrigued by the idea. We just needed a thorough example to make the final argument. In order to make the argument rock solid, we would have to incorporate all of the other changes made to our metadata within the past few weeks. For example, we had added other important points to our security metadata known as ‘fields’; now users could lock the specific field of a specific product, so that no other user could alter it. After a day or two of consideration and playing with various options, we created the first iteration of our communication protocol:


<?xml version="1.0" encoding="windows-1252"?>
<Message>
  <Action>
    <ActionName>SaveProduct</ActionName>
    <Parameter>
      <Type>SourceId</Type>
      <Value>123</Value>
    </Parameter>
  </Action>
  <Product>
    <MainAttributes>
      <Ean>&lt;![CDATA[9780123456789]]></Ean>
      <Title validate=”check_max_length”><![CDATA[This Is an Example Title]]></Title>
      <PriceValue><![CDATA[1.99]]></PriceValue>
      <PriceType><![CDATA[R]]></PriceType>
      <PriceCurrency><![CDATA[USD]]></PriceCurrency>
    </MainAttributes>
    <Field>
      <LockStatus>Locked</LockStatus>
      <Name>fieldPrice</Name>
    </Field>
  </Product>
</Message>


Of course, we knew the final implementation of our communication layer would need to incorporate other complexities (like an intelligent XML parser and a homegrown TCP/IP networking layer), but those components were part of the actual software implementation. Here, we simply wanted to demonstrate how we could utilize the metadata as a basis for our communication protocol, and we had done so. In the months and years to follow, we would be pleasantly surprised to learn that this choice would pay in dividends. In the seven years since the delivery date of our product data system, this protocol and its supporting libraries have been flexible enough to never require any code alteration. By relying on the metadata, any required changes have always been made to the numerous tables and files of our metadata, but the code has never needed such. In truth, we would learn how this pleasant surprise would become applicable to many other subsystems based on our metadata-driven design. Would you believe me if I said that the changes needed for our product data system (and its distributed architecture) have increased by 50% each year, but the actual code changes to our system have dropped by 20% each year? Even though MDD does require some level of commitment, there are quite a few rewards which can be returned from such an investment.

So, as mentioned previously, we had just gone through our first iteration of inserting and then altering the metadata in our tables, and along with forming an initial resolution for our permissions requirement, we had also achieved our first milestone with this new method of design. Now, you might be saying, “But that only exemplifies a data model. Big deal…what about designing the actual functionality?” Of course, we need to design that as well, and as you will see, there’s no reason that we can’t apply the same method here. However, in designing the flow of the functionality, we will need to design the execution blocks (i.e., a high-level view of the functions).

After achieving some success with the permissions issue and subsequent others, the various stakeholders reconvened to address the very subject of the functionality in our architecture. In this particular case, we needed to review established business rules that would validate product data before saving/persisting it, and these rules were cryptically embedded into some COBOL on our legacy systems. After I had spent days deciphering most of it (and hating every moment), we sat down to scrutinize and then approve of my translation sans a Rosetta stone. Initially, I was going to present these business rules as just simple random pseudocode, but since I’m a big fan of code reuse, why not somehow build upon our previous meetings and incorporate the metadata from those newly created tables? So, using XML, I created a markup language of my own that utilized our attributes and fields, still white-hot from the forge:


<!-- In this case, AlphaInd is an attribute -->
<if description="Alpha Case Product">
   <criteria>
     <eval>(AlphaInd) == ('Y')</eval>
   </criteria>

In this example, for any product flagged as an ‘Alpha’ product, we would ensure that certain fields had all of their respective attributes populated. In addition, it would look at the attribute ‘Title’ (of the field ‘fieldTitle’) and ensure that its length wasn’t longer than the maximum allowed. On success, it would return the text from ‘success_message’, and in the case of failure, it would return the text from ‘failure_message’.

I had taken the incremental step of creating metadata in files that referenced our metadata in tables, and even though it existed in something other than SQL columns, it was yet another milestone for our new design method. Before the meeting, I was somewhat apprehensive about the reactions of some of our stakeholders, but quickly, it became clear that I had been mistaken about my concerns. Nearly all attendees immediately grasped the rules being presented, since it was simply an additional layer to the language established with the metadata in previous sessions. Amid the enthusiasm, some people began to ponder other possibilities, like placing the business rules in a set of tables…but, after I explained the difficulty with such an implementation, that idea was gently placed to the side. (That’s not unexpected, since excited people holding a hammer will see a nail everywhere.) Even though other suggestions were made and discarded, I was enjoying the palpable enthusiasm in the room. We had found a cadence with this method, and we were making progress. I and a few others considered again how we could develop an architecture around this metadata, with properties in database tables and flow in markup files. Even though each ponderance about the possibility brought a fearful jolt, it was slowly appearing to be less crazy of an idea.

Now that I’ve spent some time describing the various benefits of using metadata-driven design, I think that it’s probably time to finally expound on the process via a working example. After all, it doesn’t seem like a concrete method until it has been put into practice. So, as I had described in a previous post, my group here at Barnes & Noble needed a new product data system and all of its inherent constituents (databases, applications, etc.). However, before we started the project, we needed to define its scope and its functionality, and in this case, domain-driven design assisted in the creation of a manifest for our requirements. However, since this system had never been created with such a focus on foresight and since we were all new to architecture design, we didn’t even know how to start this phase of the design process. It became frustrating when we tried to simply converse about a particular point of the project, and documentation with images and text didn’t seem to help the situation much.

For example, all of the stakeholders agreed that the system should know how to handle each data point via a number of dimensions: retrieving/persisting, validating, applying permissions, auditing, error handling, etc. In order to discuss how we would create a design that accounted for all of these specifications, we needed to put something on the whiteboard that we could all scrutinize together. Since the managers and directors were not as familiar with programming, the use of UML, pseudocode, and prototyped classes did not resonate with them. However, to some degree, it did seem that everyone was familiar with databases and SQL. So, a few of us decided to draw some hypothetical tables on the whiteboard and populate them with data, which would give shape to the items listed in the manifest. Suddenly, the room became alive as the whiteboard became a sounding board, and people began molding this table data that described our requirements. After that initial meeting, we created the tables on the whiteboard in a separate database, and everyone obtained access to them. We then went through multiple iterations of people altering the schema’s tables, conversing through emails about those changes, and then having another rendezvous to talk about them on the whiteboard.

Soon, this simple iteration began to produce results. Initially, the information about the data points’ dimensions (to which we began referring as metadata) was organized into appropriate metadata tables:

In one case, discussions quickly came to a consensus that in some cases, permissions should not be applicable to only one data point. For example, a product’s price was actually a set of data points: its numerical value, its type (retail, cost, etc.), its currency, etc. In that case, an user authorized to change price data should be able to change all of those data points, and the system should be intelligent enough to group the permissions for such a set of related data points. So, in an effort to create our own version of an access control list, we then took the metadata about permissions and altered it to reflect this new enhancement to our system:

It’s somewhat difficult to emphasize its psychological impact, but this emergent productivity from creating a few simple tables excited us as we successfully began to build an architecture through small increments. Through only a few simple iterations of this metadata, we had quickly understood our permissions problem and drafted a decent approach to resolving the issue. At this point, we were only thinking of it as a design tool, not as a potential foundation layer for our architecture. However, as we repeatedly reshaped this metadata to better fit our desired solution, someone eventually pondered with the question “I wonder if we could build something that just ran off of these tables?” To which, a few of us nodded in cautious, pensive optimism…but that’s a story for another day.