Elastic Cloud — is going direct a viable alternative to self hosting or AWS?

Elasticsearch has become the defacto choice for technology organizations looking to build a product or capability that need a search engine in their technology stack. Whether it be a cutting edge AI powered analytics product, an internal customer database or some other application, technologists are choosing it for it’s blistering performance, ease of use and availability of skills for the technology.

Traditionally, there have been two approaches to adopting Elasticsearch:

1. Tack it onto your Amazon Web Services spend and get them to manage it for you
2. Deploy and manage it yourself (whether on your own servers or on a cloud platform)

There are a few challenges with these approaches:

No clear winner

It’s obvious why this is still a debate as there are clear pro’s and con’s to each approach. Both have some pretty significant advantages and some pretty significant disadvantages.

Over the last few years, there has been a wave of CTOs choosing to go direct to the makers of Elasticsearch and adopting their Elasticsearch cloud offering. Going direct for anything in life has some obvious advantages, but essentially, it captures most of the benefits of each of the two traditional approaches:

1. Convenience — Elastic Cloud as a cloud product is by definition much more convient than hosting inhouse. Arguably it’s not quite as convenient as AWS if you crave above all things a single point of billing and support for your infrastructure, but if Elastic instances have become a substantial line item on your AWS bill, now might be the time to reconsider this position.
2. Configurability — Because Elastic Cloud is a platform that sims to do one thing really well, there has been no need to strip out a bunch of features to make it fit into a wider “one size fits all” the technologies in our “one stop shop”. Elastic cloud is highly configurable and this gives a huge edge in terms of performance and potentially allows each instance to stretch much further (no pun intended), leading to substantially lower costs.
3. Software Updates — One of the advantages of buying a Google phone rather than a Samsung phone is that google make the operating system and so you get the updates within days of release rather than months later or never. Exactly the same is true of electing to use Elastic Cloud. Your engineering teams benefit from getting access to all the latest technologies as soon as they are available, which can have a substantial impact on speed of innovation. Conversely, there is no requirement to have systems or dev ops teams be continually testing and managing updates.
4. Features — With Elastic Cloud, you get the fully featured Elastic Seatch experience. Whilst Amazon make the statement “Amazon Elasticsearch Service supports many of the commonly used Elasticsearch APIs”, with Elastic Cloud, nothing is stripped out or watered down to make it fit into a generic platform; All of the API endpoints are supported meaning that there will be no surprises mid project when the team encounter an unexpected blocker because they expected a feature to be there that had been stripped out.
   4b. X-tools — is an advanced set of tools and utilities for managing Elasticsearch. It’s basically a must have if you’re serious about Elasticsearch. Unsurprisingly this is not included with AWS or DIY / self host.
5. Support — It should come as no surprise to find out that the clear winner for support experience is by going direct to Elastic. They have access to bring deep expertise in some minute aspects of the technology with a minimum of friction and fuss. This can provide a huge advantage as you invest more heavily in optimising Elastic or pushing it to do more when forums or third parties just don’t have the deep inside knowledge.
6. Hosting costs — Whilst Elastic Cloud is arguably paying someone else to do something that you could do yourself and therefore could be more expensive it, however, exists in a sweetspot. The economies of scale afforded by someone doing nothing other than hosting combined with the expertise and the configurability (allowing for each instance to go far further) mean that Elastic cloud is a surprisingly affordable option.
7. Management costs — Elastic Cloud has the lowest overall management costs — it’s a cloud solution, giving it obvious advantages over the DIY approach, but with added flexibility meaning that developers and administrators spend less time working around the limitations of the reduced featureset in AWS.

Final thoughts

If you’re considering adopting Elasticsearch or substantially invested in one of the legacy approaches, it’s definitely worth evaluating Elastic Cloud alongside the other options.

• Deconstruct the TCO arguments — There will likely be enthusiasts pulling you towards DIY / self manage and at the other end of the spectrum, people politically invested in unifying AWS, so it’s worth trying to do an honest evaluation by trying to untangle all of the arguments made up in the overall TCO calculation.
• Audit the technical benefits — If you’re using Elastic on AWS, research the features that are on newer versions, have been consciously removed, or are available in Xtools and consult your tech leads to understand where these could be useful. Conversely if you’re self hosting, perform an audit on the overhead and mean time to new version.
• Evaluate — If you’re not yet ready to make the leap with both feet: start small. Pick a new project (ideally one that is likely to put it through it’s paces) and use Elastic Cloud for that one project. If it goes well, this should be a good way in to negotiate a good deal having demonstrated a serious commitment.

I hope this has been a useful comparison. If you have any thoughts or think I’m wildly off the mark, I’d be delighted to hear them in the comments below.