SaaS suppliers are no black box, evaluation is key

An aspect of cloud computing that is often overlooked is supplier evaluation and management. We spend time considering the pros and cons of new delivery models, but just like any other area of IT, the vendors and service providers you choose to work with have a huge bearing on the outcome of any investment or contract commitment.

This is particularly relevant in the area of Software as a Service (SaaS), the flavour of cloud computing that from an enterprise perspective is typically aimed at delivering business application functionality as a hosted service. Offerings here range from full blown ERP and CRM, through comms and collaboration solutions, to more tactical services aimed at workgroups or even individual users. Some would characterise online backup, security and other systems level capability as SaaS too, but for the purposes of our discussion, let’s stick to the business functionality layer.

The first point to make is that it is still relevant to consider pretty much all of the criteria we would normally apply when selecting a piece of traditional on-premise application software. The solution must meet immediate needs in terms of business functionality and allow configuration around policies and processes as necessary. It will also ideally be flexible and expandable enough to deal with evolving requirements over time.

Stating such things may seem obvious, but all too often the seductive immediacy of SaaS, with many providers promoting a ’subscribe and go’ approach to adoption, leads to short-term thinking and short-cutting of the important needs assessment and gap analysis process. If anything, the requirement to ensure a good functional fit with room to grow is even more important with SaaS.

While most organisations nowadays try to minimise the degree to which they customise core parts of application packages, at least it’s an option with on-premise solutions should it be necessary. With SaaS, the software installation is not yours to tamper with, beyond standard configuration facilities, so there is a limit to how much you can work around supplier imposed constraints.

This brings us onto another area of common misconception. In the context of larger scale enterprise computing environments, the chances are that SaaS solutions are likely to have to integrate with other systems and technology.

The technical architecture therefore really does matter, at least insofar as it manifests itself in terms of application programming interfaces (APIs), support for industry standards, and interoperability with existing management tools, security systems, desktop software, mobile devices, and anything else that is important to you.

Thinking of SaaS offerings as black boxes is generally a mistake, especially when we consider that so much cost, risk and general frustration and inefficiency in IT is as a result of fragmented and disjointed systems.

Turning to providers, we clearly need to consider the basics. It’s important to consider how stable the organisation is financially, how well it is tuned in to different customer requirements in terms of industry and size, and how capable it is of providing the kind of support needed in the places it’s required.

On this last point, there is a specific consideration with SaaS around the support process. Some providers are comfortable with and capable of supporting end users directly, while others require the customer’s IT department to deal with front line interaction.

There is no absolute right or wrong model here, but mismatches between requirements, expectations, capability and processes can cause significant issues if they arise.

It doesn’t help, for example, if the provider is offering end-user support but works on the basis of strict demarcation of responsibility, resulting in users being handed off too frequently to other parties because the problem is not proven to be associated with the service.

Neither is it helpful if in-house support staff are taking calls about the service, but don’t have the visibility or tooling to troubleshoot problems effectively. In today’s IT service delivery environment, the need for end-to-end service management, and therefore end-to-end visibility, diagnostics and troubleshooting capability, is becoming ever more critical to meet business expectations.

Another practical area to consider is the supplier’s policy on software release and implementation. Some providers trickle out a constant stream of updates in the spirit of continuous improvement. Others have one or two major releases per year, with very little happening in between.

There is then the question of whether you will have a choice of if or when you take a new release on board, with some providers allowing a degree of customer control, and others forcing new updates onto all subscribers unilaterally. Again, there is no right or wrong, but again, mismatches can lead to issues.

The areas we have discussed are just some examples of what matters when it comes to SaaS supplier selection. We could extend the discussion to include pricing models, contract terms and a number of other areas. The bottom line though is that due diligence is just as important with SaaS as it always has been with any key IT related decisions.


Dale is a co-founder of Freeform Dynamics, and today runs the company. As part of this, he oversees the organisation’s industry coverage and research agenda, which tracks technology trends and developments, along with IT-related buying behaviour among mainstream enterprises, SMBs and public sector organisations.