How do you manage your ElasticSearch clusters?
At my company, we've built our ES management system on top of AWS EC2, OpsWorks and SSM.
Security patching is automatic, we use SSM Maintenance Windows for that and it automatically runs every day. We can also easily change the size of the instances automatically. As our ES usage grows we can automatically replace all the instances with bigger instances without disruptions. We can also change the AMIs on the fly like that.
We supplement it with a CLI tool that people can use to trigger some actions people often need like replacing a single node, change the allocation settings etc.
How automated it is for you? Have you automated node replacements, security patching etc? What tools do you use?
use elastic cloud.
I'm using elastic cloud which automates almost everything while having an affordable pricing.
Maybe I'm lax but for a few months I just started one up on AWS and it's been running fine.
I only run a single node. If that's down I need to bring it manually back up most of the time all the time.
My software works in a way however which automatically mitigates the downstream effects of ES being unavailable, and keeps on collecting metadata, which is better than nothing. I realize I'm balancing on a thread here, but it's a nice exercise to make software resilient :`)
Are those CLI tools you have written in house? Can you tell a bit about the process of swapping nodes?
Those CLI tools are in-house, yeah.
Node swapping is not too complex but it works differently depending on the type of the node (master/client/data), e.g. for data nodes, the most important steps are to add a new node of desired size and settings first, then remove any shards from the old data node using the cluster.routing.allocation.exclude._name ES setting. Then once there is no data on the old data node and the cluster is green, you can kill the old data node and continue replacing other nodes.