During one of the discussions about the design for our upcoming system, several stakeholders (in the form of our system’s consumers) would sometimes visit and listen to our upcoming plan. One of these fellows was in charge of a system which specialized in extracting/importing massive data streams, so that this data could be quickly loaded into a set of cache servers (which I believe used an assortment of software like memcached); these cache servers were fast read-only repositories used by several departments. As he listened to some of my talking points, he became interested in the metadata being discussed, and we talked briefly about how he might utilize the same metadata when he exported our product data from our database. He wanted to utilize the modules in the planned engine which would dynamically build SQL statements based upon the metadata, but since his solution was exclusively in C# and our implementation was going to utilize something more low-level (like C++ ‘98), code reusability seemed to be a pipe dream. So, initially, we started to talk about how he would need to write his own version of our project’s SQL layer, which was responsible for the generation of various complex statements:

During those discussions, an epiphany suddenly struck me, and I chastised myself for not thinking of something earlier. Our new planned server, which would be a generic engine fed by metadata, was designed to handle REST-style requests that were populated with metadata; the requests would then dynamically build a SQL statement for the pull of the data which had been indicated by the request’s metadata. (At the time, REST-style requests were nascent, and it took a few discussions before everyone became comfortable with the idea of sessionless requests.) So, why not then simply create a type of request which could just return the SQL statement itself? I mentioned it to him, and his face failed to conceal his interest at such a possibility:

Consequently, when the server was finally completed, the supervisor of cache loading was easily able to obtain his complex SQL statements from our server, and his code never needed to know the exact implementation of our database’s schema. Through the common language that was our metadata, we had used our abstract solution again in order to satisfy another process in the data pipeline. In this way, the strength of our metadata proved that in addition to running our platform, it could become a way to integrate our platform and others. For almost a decade now, it has served this particular stakeholder without any issues, and for the same amount of time, it has served various other stakeholders in similar ways. Such is the might and glory of metadata! (Okay, I’ll admit that I got a little carried away there at the end. Occasionally, I need to toot my own horn.)

Before we had even started the project phase regarding domain-driven design, I was already dreading the thought of collaborating with our management. Historically, there had always been a division of ideas between our managers and our developers, and consequently, poor communication had always been a byproduct of those alternate lines of thought. In some cases, the methods between them were different, but in many cases, the problems could be traced simply to a lack of understanding between the two parties (especially in regard to terminology). Unfortunately, since people often don’t delve deep enough into complex conundrums, our developers and managers hadn’t yet deconstructed our failures in order to recognize that communication was even an issue. In the end, it seemed like it would take a miracle for everyone to sit at the same table and to collectively overcome our inability to collaborate effectively. As it turns out, that miracle would eventually come to us in the form of metadata-driven design.

At one point, we were discussing the need for some of our product data to have complex relationships. Some of our data points needed to be within a particular domain if a related data point happened to be within a different domain. For example, a product could have a subcategory code of “ZZ”, as long as its subject code fell within a certain domain and its category code fell within another distinct domain. In the past, much of this domain validation had been embedded within the code, and for some people, the obvious improvement was to place this metadata inside an organized table. Something akin to:

Which allowed an implementation where the subcategory code “ZZ” could be associated with all of its possible permutations:

“subcategory code”, “ZZ”, “subject code”, “SUBJ1”, “category code”, “CAT1”
“subcategory code”, “ZZ”, “subject code”, “SUBJ1”, “category code”, “CAT2”
“subcategory code”, “ZZ”, “subject code”, “SUBJ2”, “category code”, “CAT1”
“subcategory code”, “ZZ”, “subject code”, “SUBJ2”, “category code”, “CAT2”

Plus, there was an extra set of columns for a possible third relationship someday. It was a more robust solution than the previous one, and it was a decent step towards creating metadata. What more could anyone ask for?

Well, a few of us could indeed inquire further. Curious, we asked the data experts in our group the more important question: how many other relationships resembled this one? As it turns out, there were actually many other examples of this type of relationship within the data, and we needed to account for them. As we huddled over the data schema, a new set of tables begin to emerge from our discussions:

Where the three bottom tables could hold an infinite amount of relationships, and a database view could help condense the tables’ data for the managers. At this point, you might be saying, “So what? You normalized some data. What’s the big deal?” Well, there are several important points here. One, by focusing on the metadata before the actual design of the project, the entire team began to have a better understanding of what this application was actually meant to do. In effect, the metadata had become a transcribed whiteboard of our minds’ vision for the project. Two, we simply started here, but this one incursion led to a whole campaign of building an abstract form of the entire application, especially as we began to employ recursive abstraction (which I will explain later). Most importantly, though, by having these types of detailed discussions, we unintentionally developed a new language and a new grammar about our new system. Much like a domain-specific language for computers, we created a domain-specific vernacular (i.e., DSV) that was employed by our stakeholders. (Vernacular being a good word to use since it’s usually found within conversations about building architecture. Why not use it within the context of software architecture as well?) Our managers and developers actually began to speak this common vernacular, leading to a much quicker comprehension of others’ points and to faster iterations of our design. In that way, we learned how the metadata could construct a language, and the language could help design the foundation for our architecture.

Even though I haven’t yet addressed a specific guideline when it comes to practicing MDD (i.e., metadata-driven design), I’ll eventually write more about such idiosyncrasies. For the moment, though, I can assure you that there’s nothing revolutionary about it. After all, as Newton popularized the sentiment, we often find ourselves standing on the shoulders of giants. I find myself in a similar such position, since most of the requirements for MDD originate from domain-driven design. In order to surmise the appropriate metadata for an application, you must outline a domain model for your application and speak about it in terms of a domain-specific language. In fact, metadata is often a direct byproduct of data-driven design. The key difference between DDD and MDD (besides the former being the parent of the latter) is that metadata-driven design is ideal for projects where the team has the luxury of time, resources, and expertise to practice further abstraction, in order to build a solution which anticipates a domain model that will evolve with time. My manager and my colleagues found ourselves with such a rare opportunity, and we embraced the moment. After reading Domain-Driven Design by Eric Evans and putting those methods into practice, we came to such a model for our initial solution:

However, as I mentioned in my last post, we were not quite happy with that arrangement, since it left the configuration as simply data retained in flat files. On its first iteration, this metadata was supposed to be par for the course, with the usual mentions of databases, server URLs, usernames, and passwords. On subsequent iterations, it became more refined as it started to include tables, data points, and the names of stored procedures with business rules. However, as we started to converse and consequently argue about it, some aspects began to be points of contention. For the managers, the idea of trawling through a dozen (or dozens) of configuration files began to fill them with a sense of dread, emphasizing that it would eventually become a terrible mess (which, indeed, sounded likely). Several developers including myself hated our current path of making the metadata into a complete lexicon of our application’s domain, since it would inflate the size of our configuration files and since it would lead to eventual code bloat. Instead, we focused on the idea of the metadata not being static values based in files and instead being dynamically addressed in a set of database schemas and tables. In the end, after we all delivered and received many acerbic harangues, our final design became an unique combination which would satisfy both developers and managers alike:

In this case, by having certain experts on hand and by abstracting the functionality in our DDD solution, we had escalated the metadata from being simply a set of values for a specific domain model to being the fuel for a powerful, more-generic engine. By doing so, the developers became more enthusiastic about the project, since it would be more than a simple CRUD application and would be a true development challenge. For managers, it would become a system that they could understand, manage, and control through simple SQL statements. It would be an ideal solution, if we could build it together. Would we achieve that through collaboration…or would our ending be more reminiscent of The Lord of the Flies? And just how could we build such a thing? Stay tuned for next week’s episode in order to find out.