8 Dimensions for CMS Technical Evaluation

There are many places you can go to find information on the racking and stacking of CMS systems. Everyone has their preference and there is no shortage of experts who will evaluate your situation and conclude their favorite is the best for the job.  Over the years I have developed a serious disdain for a common behavior among CMS buyers.  All too often selections are made solely on the recommendations of the hired guns or by procurement weasels without genuine ownership of the technology. The end result is almost always a battered account manager, a frustrated implementation team and an exasperated user community running to the internet looking for a panacea they can buy without IT approval. The built in plausible deniability may save the career of a mediocre manager or two but the interests of the users are better served by taking the time to understand technological dimensions of content management.

Here I’ll identify eight categories for a technical evaluation of content management software.  I like to think of these topics as dimensions rather than simply groupings of requirements. The needs of a given customer will expand and contract requirements within the dimensions and change the weight of one over the other. These are however the areas that I have traditionally used to determine best fit for my situation.  The list is in alphabetical order so as not to imply one dimension’s importance over another even though certain preferences will be obvious.

Dimension One – API & Tooling

One could argue that these are two dimensions rather than one and be successful. In my experience however I have found that consideration of one without the other provides an incomplete picture of the technical agility of a platform and the productivity you might expect to see from your development team.  One would think an open, mature and well-documented API might be common to all commercial products in the space but nothing could be farther from the truth. Different vendors take decidedly different approaches to what API’s they expose and documentation quality is all over the map.

The level of customization required for a given initiative however may lessen the need for concern in this arena. Likewise, good tools can often make up for a poorly documented API by hiding the complexity. One should not assume that one product is better than another though simply because of the number of API’s available.  Excessive and complex api’s may serve agility in implementation but support may suffer having to know and account for redundant or deprecated technical pathways.  Understand what the gap is between the native behavior of a product and your requirements and evaluate the api and tooling to ensure that it is a good fit with your ability to support it once the consultants leave.

Dimension Two – Componentization

There is a school of thought that assumes you would want to horizontally scale everything, therefore every discreet capability should be a separate module. This sounds really good in an engineering meeting and adds to the number of sku’s on a price list but when it comes to deployment and support, over componetizing a product will kill a project. I once saw a brilliant demo of a banking process. After it was over I asked how many individual product installs it took to put it together and the number was in the high teens.  There is definitely a point of diminishing return when it comes to breaking a package into deployable units.  Naturally the larger the problem – the more it makes sense to break it down. The problem is that buyers are often unaware of how many pieces the puzzle is actually sliced into. The other side of the issue is just as problematic when one realizes that there is a choke point in a data flow long after the license agreement is signed. You have to understand the component architecture of the products and ensure that the level is appropriate to both the problem at hand and your ability to support it.

Dimension Three – Content Storage

All content is not created equal. The same is true for requirements and the software you select to meet them.  Most content management system offer a variety of  techniques to store the actual files. That is after all what you are doing – managing content. But how a product reconciles content management to storage management has significant implications in performance, scalability, cost and data integrity.  From BLOBS to optical to NAS to SAN , the options for combination and restrictions are endless but make sure that the mechanics for I/O are appropriate for your business problem. There are some products that simply can not handle extremely high volume ingestion and retrieval. Extremely large files and rich media types (video, audio, etc) also factor into making an appropriate selection. There are certainly products that attempt the one stop shop approach but there are installations and requirements that may lead you to conclude  specialization for a given requirement is preferred. Beware though of pseudo CMS systems that do not address this as a concern and don’t distinguish between storage vs content management. Simply writing to a file system maybe appropriate but can quickly become compliance and performance nightmares if you don’t understand the true nature of the content being stored.

Dimension Four – Database Management and Integration

All content management systems, without exception, deal with two classes of data. Unstructured and structured. You can make the argument that XML repositories bring the two together but in the final analysis – there is a point at which you stop breaking the data down to fields, attributes or properties and store large segments of contiguous data that require an application to interpret. The handling of the metadata – that information that surrounds and describes the content itself is most often handled in traditional RDBMS patterns and is unremarkable. The differentiators are in how open and extensible this model is and how many techniques are available for integration directly to the data subsystem. In general it is a bad idea to get overly concerned with the management of the data as that should be left the application but there are cases where the design and implementation can make or break a system. The easiest example of this potential is when the database gets exceptionally large and partitioning becomes necessary.  This tends to be a problem that products don’t deal with until forced to by an account manager that sells their product into a customer that exceeds their current install size by a couple of orders of magnitude. For many implementations this is never a factor because the repository will remain small and manageable. Still, this dimension must be a high priority when designing enormous systems.

Dimension Five – Transport Layer

HTTP is great. We all love it. I am using now as a type. It is however not particularly efficient for moving exceptionally large data files. Technologies to support streaming audio and video are everywhere but surprisingly few content management systems efficiently deal with large contiguous files efficiently. CMS’s that grew up never having to deal with multi-gigabyte content transfer issues on a global scale will immediately dismiss this as not being an issue, hoping that everyone’s comfort with the internet will lull them into a false sense of security. You have to understand the dynamics and metrics around the content that you will be managing and understand how the infrastructure of the CMS will support it.  There again, if all you are doing is posting tiny text based wiki entries -HTTP is all you will ever need.

Dimension Six – Scalability In General

I recently evaluated a product and was shocked to discover they had no deployment model that supported horizontal scalability. They only supported an active/passive model.  What surprised me most was that it had not affected their sales. The product was for a very specific set of users and so long as the application server was properly configured performance was not an issue.  While I had a lot of scalability alarms going off in my head I had to admit that it was a “right-sized” architecture for the problem they were trying to solve. Having worked for one of the largest retailers in the world – I had experienced first hand how scale can crush an otherwise promising product. It is overly simpleminded and really expensive to force galactic scalability requirements on every product and solution.  Don’t assume you need infinite scalability in both the horizontal and vertical dimensions. Understand the “right-size” for both your hardware and software architecture. While related to componentization dimension, they are really different topics. Evaluation of scalability represents an intersection of requirements from other areas and I have found it useful to think of the needs of a project as a whole before considering how it is manifested in the underlying categories. (database, transport,storage,etc)

Dimension Seven – Security

I never wanted to learn terms like cross-site scripting and SQL injection but there is no escaping understanding at least the basics of application security when evaluating a software product. It is not enough that the product has a sophisticated security model with the ability to integrate with directory servers and create multi-dimensional optimistic and pessimistic access control. The best laid plans of the data architect can be foiled in an instant by a lazy coder. There are clearly two aspects to product security then. Application implementation and coding practices. Within your business you must take responsibility for understanding the risks and the costs with securing the data that your system manages and how the product you select will mitigate those risks. Like the right-size argument noted above, pragmatism must be applied. Everybody does not work for the CIA. Not all content is important. Sometimes it is  just pictures of toilets on the internet. Be reasonable and beware the career building of security engineers that take the simple minded approach and over-engineer every project and redesign every product you deploy.

Dimension Eight – User Experience

Clearly this list is not in order of importance. Many CMS’s failures are blamed on this dimension not being sufficiently considered.  I have always held that this is never below third in order of importance.  It is never the most important.  User elation may help ROI but you will never cost justify a project based on it.  If it was all about user experience  we would all just be watching Hulu at work all day. Producing and managing the content has to be first. No user will be happy if the system doesn’t work. Granted – bad user experience will kill adoption which one could argue is the same thing but implementation of a CMS will and should change the way people work. User experience cannot be measured by how little a given user’s routine changes. There is never change without resistence. The products you select should mititgate the resistence, accelerate the adoption and serve the business needs that justified the purchase in the first place.