Should you opt for an on-premise CI solution or a hosted CI solution? It’s one of those never-ending questions, a lot like “should I use tabs or spaces?”
I don’t know that I nor anyone else is ever going to end this debate about continuous integration decisively. But what I can do, what I’m going to do in this post, is to compare and contrast some of the pros and cons of hosted versus on-premise CI solutions. That way, you will be in a better position to make an informed decision as to what’s best for your organization.
I may be subjective in my analysis. It’s more than likely that I could be. But, as much as possible, I hope to provide objective arguments, ones backed by research and feedback from a number of experienced colleagues.
If nothing else, I hope to stimulate an educational and informative discussion. Let me know in the comments if you agree or disagree or have more to add to the discussion.
On-Premise CI: The Pros
I’m going to start off with, for no particular reason, on-premise CI and conclude with hosted CI. We’ll begin with the positive features of an on-premise CI solution.
Control it yourself
When you have your CI solution on-premise, you have complete control of it. If it’s properly secured, no one can get involved or interfere with it. You don’t have any concerns about vendor outages, maintenance, or any other form of downtime. You can build the infrastructure for it as and when you need.
These are a series of excellent advantages. If it were hosted or provided via a SaaS, then you’d have to find the most appropriate package that that vendor offered, even if it didn’t quite suit your needs.
This approach needn’t be a bad thing. Perhaps your software or organization doesn’t need anything that one or more vendor packages don’t already provide. But if your software is significant or rather specialized, then on-premise will likely be the better choice.
Customize just for your needs
Is there a vendor who offers what you need? Is there a package tailorable to your unique situation? Do any of the default offerings fully cover your software needs both now and well into the future?
If they don’t, you may find yourself playing musical chairs with vendors as the complexity and sophistication of your software grows. As with moving database vendors, moving CI providers may not be an inexpensive endeavor.
Like most things, you need time to get acquainted with a set of tools and to build a set of workflows around them. To have to change them intermittently because they aren’t able to grow with you may end up being a rather frustrating, not to mention expensive, undertaking.
But there are other issues that can result from hosted solutions, even hosted open-source solutions. Let’s take Jenkins, for example. While it may come with a near endless array of plugins that allow for all manner of functionality above and beyond the base, does the host provide it?
You may find yourself in the position where you can only use the core functionality, plus a limited set of plugins. Or you may only be able to configure the tool within a predefined set of boundaries.
Given that, hosting your CI solution, such as with GitLab or Bamboo may be the right solution.
Because when you host it yourself, you have the flexibility to choose any number of plugins and make all the configurations that you need. You’re unconstrained by anyone else’s preconceived ideas of what you may or may not need.
More convenient for small teams
If you have a small team, do you have the capacity for the overhead that a hosted solution may incur? I don’t mean to suggest that a hosted solution is necessarily complicated. But an on-site solution may be simpler. What’s more, an on-site solution may be easier for a small team to master.
By having your CI solution on-site, it may make for a more collegiate atmosphere as well; as everyone can be involved in the entirety of the software creation process. Everyone can be a part of designing, developing, deploying, and maintaining the organization’s software.
This point, perhaps, is a little bit abstract, even a little bit of a stretch. Is there a difference between on-site and hosted in this case? If the solution is set up correctly, then there should be very little required of any developer to maintain it post-setup. The CI solution should require no input from anyone, except to review it from time to time.
But, at least during the setup phase and during periodic review and maintenance, an on-premise solution may be simpler and more manageable for a small, tight-knit team.
Latency, security, bandwidth, and disk space
Where the last point was, perhaps, a bit vague, this one is very tangible. When a solution is off-site latency, security, bandwidth, and disk space must all factor into the equation.
To be fair, unless your applications support stock exchanges or other systems that absolutely must minimize any and all delays, even down to the microsecond, it’s unlikely that a bit of latency or bandwidth will be of much concern.
However, they’re not of no concern. After all, if you’re using a hosted solution, then you’re going to have to share the resources of the vendor along with all the other users.
Yes, vendors will all attempt to ensure that, regardless of the number of users that they have, each and every one has the same experience. But you’re still going to have to share.
What’s more, if the recent Amazon S3 outage is anything to go by with so many vendors depending on S3 for storage hosting requirements, if it goes down, then so does the vendor.
Is that something that you can cope with? Can you wait for Amazon to correct the problem and for the vendor to come back online? Do you have turnaround requirements that mean that this kind of delay is unacceptable?
Now let’s consider security. Depending on how much of a security mindset your developers have, this is likely a greater or lesser issue. But either way, when you use a hosted solution to deploy your code, then you’re storing a copy of your code with that vendor.
As a result, the vendor’s staff can potentially read it and copy it. What’s more, depending on where the code is ultimately stored, there may be more eyes on it than you think. Is that something you’re comfortable with?
On-Premise CI: The Cons
But we can’t forget that on-premise CI solutions have their own shortcomings. Let’s explore a few areas where it may not be giving you everything you need.
Setup, configuration, and maintenance
If you choose to have an on-premise solution, that means that you have to take on all the work involved in managing that solution — from downloading, installing, and configuring to patching and updating the configuration as your software’s needs change.
Depending on the solution you choose, this may not be much of a concern. However, if you pick a tool that requires too much input, then you may find that your developers are spending more time — perhaps too much time — maintaining the deployment solution than they are building the software that makes you money.
Then, there’s the question of suitably experienced team members. Consider these four questions:
- Do you have them?
- Do you have a plan in place for when they eventually leave?
- Is the solution you’ve chosen one where you can readily find more suitably experienced people?
- Is the solution one where it’s nigh impossible to find someone?
The security is your concern
As the security breaches over the last 12-36 months and the revelations from such whistleblowers as Chelsey Manning and Edward Snowden have shown, no one is safe. Anyone who has a server connected to the internet can potentially be hacked.
A CI server is likely to be much less of an appealing target than is a running application. But that’s not to say that it would hold no interest. Consider these two questions:
- What would it cost your company if it wasn’t able to deploy a new feature or a bug fix?
- What would that disruption cost in lost revenue or reputation?
If your CI solution is on-premise, you have to factor in its security. Do you have staff who are sufficiently experienced to secure it? Will you be able to find new staff when existing ones leave?
Next, what information does the CI server contain that could be of interest to a potential hacker? And, are you sure that it’s not inadvertently leaking any information?
How much does it cost?
Now let’s look at overall cost. While a hosted solution’s cost is very easy to measure — you just look at the amount you’re being charged every month or year — how much does an on-premise solution cost?
That’s easy, you say. It’s open source, and I have a few servers to spare. So it costs me nothing. Wrong. There’s always a cost. Let’s consider a few of them:
- How much are you paying in wages to set up, configure, and maintain it?
- How much are you paying for electricity for the servers that host it?
- How much time are staff investing in educating themselves about the solution?
Perhaps, after an intensive analysis, you’re happy with the price. But have you checked? If you haven’t, you might be surprised when you do.
Hosted CI: The Pros
Let’s move on to the alternative to on-premise CI: a hosted CI solution. Again, we’ll let it put its best foot forward and discuss the benefits of having someone else manage the CI.
Simple cost analysis
As I mentioned earlier, a hosted CI solution’s cost, broadly speaking, is very easy to measure in contrast with on-premise solutions. Take Codeship’s basic pricing, which you can see below,
In no uncertain terms, you can see how much you pay and what you get for that price. As of this writing, $49 per/month gets you unlimited builds, one concurrent build, and two parallel test pipelines.
This holds true for Codeship’s pro pricing also. Either pick an option from the dropdown or move the slider to choose the options and price that you’re happy with. After you sign up, there’s never a question about what you pay for what you get.
All the aspects that go into providing you with that service are nothing you need to be worried about. You don’t need to factor them in and you don’t need to consider staffing needs. You find the package that works best for you and the onus is then on the vendor to consistently provide that.
Everything is managed for you
Whether you’re a small or large organization, this may be of particular importance — perhaps particularly so for small organizations. With a hosted solution, you don’t need your development staff (or any other staff) to be familiar with building and managing CI solutions. It’s great if they are, because then you can make more informed decisions. But they don’t need to be as a matter of course, other than knowing what’s required to configure the hosted solution so that it builds and deploys your software correctly.
Okay, backtracking a little, there will always be some knowledge required. My point is that the level of knowledge and experience required with a hosted solution should be significantly lower than with an on-site solution.
The vendor should provide an interface that makes it easy to build a CI pipeline. They should also handle any maintenance, outages, upgrades, and improvements in the solution.
Hosted CI: The Cons
In the first section where we discussed the benefits of on-premise CI, we got hints of why a hosted CI solution may not be right for you. Let’s go a bit more in-depth with the pitfalls of hosted CI.
Potential for data breaches
As I said earlier, when you host your code with an external third party, can you be sure that you know who has access to it? Do you know — for certain — what the access policies are and if they’re being enforced?
Far too many cases have come to light in recent years, some years after the initial fact, of companies not being either suitably prepared or appropriately accountable, such as Yahoo!, for the security of the data entrusted to them by the users and customers.
Given that, while I’m not saying don’t trust an external party, I am saying that you need to do your background research and investigate such considerations as:
- How do they respond to customer issues?
- How do they handle outages?
- What options do they have in place for staying in touch with their customers?
- How they conduct themselves?
It’s your data. You cannot abdicate responsibility by outsourcing your CI needs to a third party.
Will the provider always be around
Let’s say that you’ve signed on to a provider, one with rave reviews from a legion of customers, one even backed by VC funding. Does any of that guarantee that they will always be around? Could some external change force them out of business or make it impossible for them to stay in business?
Consider the case of Lavabit, the email service brought to fame in part by Edward Snowden, the service that (for a time) was shut down due to changes in the US legislative landscape.
According to Lavabit founder, Ladar Levison, Lavabit offered an encrypted email service, one that made it possible for people to communicate without fear of said communication being read by anyone other than the intended parties.
However, because of laws enacted by the US Federal Government, he was required to install both surveillance equipment and surrender the company’s encryption keys. He ended up deciding to terminate the service rather than betray the confidence of his users.
This, perhaps in the case of continuous integration servers, is a bit of a stretch. But perhaps it’s not either. What I’m saying is that even though things may look great now, there’s nothing to guarantee that it will always be that way.
Ability to move providers
Finally, what does it take to move providers? How much effort and cost do you need to expect in order to move should the need arise. It may not seem to be the case now, but there can be any number of reasons why you may need to. Here are just a few:
- Your provider goes out of business
- They’re bought by a competitor and one of the following happens:
- You disagree with new business practices or dislike the buyer’s history
- They’re shutdown
- The feature set is progressively reduced
- They start to focus on a segment of the market, such as a language or tool set, that is contrary to your needs
Any of these could be enough to warrant finding another provider and needing to move. If that time comes, can you? Is there a clear path to retrieving your data and getting set up again?
In Conclusion
While not conclusive about which is the better choice, these have been several reasons both for and against hosted and on-premise CI solutions. It’s almost impossible to provide a conclusive assessment as to which is better; every organization’s circumstances and needs are different to any others.
However, these considerations should help you make a more informed decision, as they cover some of the key considerations any organization is likely to face. Whichever path you choose, please ensure that you’ve done due diligence and are as prepared as possible for both the short, medium and long term.
Do you disagree with anything I’ve said? Share your feedback in the comments.
| Reference: | The Pros and Cons of Hosted Versus On-Premise CI from our WCG partner Matthew Setter at the Codeship Blog blog. |
