Original context #

I originally set up a portfolio website back in 2016. At the time I was still chasing for Linux Sysadmin and as part of my studies/preparation I had been setting up LAMP/LEMP stacks, on bare metal, in VM’s, even on AWS (probably with LightSail, I can’t remember). I had some familiarity with Wordpress as a result and didn’t really look beyond it for a website solution. I wanted my own domain (so I didn’t look at WordPress.com) and I wasn’t specifically targeting cloud, so I asked around and went with an old school web hosting provider. It was actually quite a lot of work setting up the site once I got beyond the initial defaults but I liked the result well enough and I didn’t need to redo anything very often so it didn’t matter so much. As you can see by the posting dates, I don’t post that often but certainly there was always maintenance to do…

This post is also published on Medium It’s kind of a standing joke in the industry - do some cool thing with AWS to implement an infrastructure feature, and if it works well, Amazon will come along a few months later with some matching in-house feature. Sometimes that feature might be a relatively simple thing, maybe something that was obviously missing, sometimes you might have had a whole project that was essentially deprecated, and sometimes it’s a feature of a larger piece of work that means you have to adjust or re-evaluate your approach. What do you do when this happens? This article covers an example of each - one that happened with a company I was working with; one that happened with a third-party project we made use of, and one that happened with my own project.

This article has also been published on Medium

There has been much written about the right way to handle alarms and alerts for Sysadmins, Ops and Reliability Engineers. I take the approach that you can learn as much from looking at how not to do it. Here are some examples. I’m sure readers can think of many more. This is one small part of a big field and doesn’t begin to cover all the other areas that system monitoring and feedback etc. touch on. Neither am I attempting to cover the greater field of Ops, systems administration, Site Reliability Engineering etc in any detail. Lastly, I am talking pretty much about the politics of alerts rather than the technical aspects. The below are principally about out-of-hours ‘on-call’ type alerts but the principles are general:

Deploying ssh Bastion as a stateless service on AWS with Docker and Terraform #

I also have a presentation and live demonstration on the below, so far given at DevSecOps - London Gathering June 13 2018. This article has also been published on Medium The mantras of software as a service, stateless, cattle vs. pets, etc, are often and loudly repeated, but in many environments you often don’t have to look too far before you find some big fat pet box sprawling somewhere. Maybe it is the in-house file server, maybe something else, but if your infrastructure is in the cloud then it is most likely going to be your Bastion server (or ‘jump box’). Here I look at the problem, look at a couple of options and present a solution that I implemented providing Bastion ssh as a stateless service on AWS - the code is available on GitHub and also published on the Terraform Module Registry. Whilst the principles are applicable universally, this specific solution employs a Terraform plan to deploy to AWS. If you are not using AWS then you might find concentrating on the cloud-config user data stuff more useful as the rest would need to be ported, e.g. for DigitalOcean etc. If you’re using GCP then to be honest you probably don’t need this at all.

**25 Minute reading time  **(but article is composed of short, numbered sections!)

1 - why is this article different to every other blog post on the ELK stack? #

There’s a lot of articles on ElasticStack/ELK components out there, I found a LOT that were extremely basic, essentially school project level reiterations of official elastic.co documentation and also a few that were very high level, essentially assuming that you already know everything and are wanting to ’talk shop’. I really struggled to find anything that covered a full use case in any detail without hand-waving over the fiddly bits. Whilst I do give some basic info here, I don’t intend to re-iterate official documentation and I cover a full use case including some of the blind alleys and pitfalls. This article concentrates on the technical challenges and solutions. It is not intended to be an introduction or comprehensive guide to the Elastic Stack. Bear in mind that I was starting more or less from scratch without previous production experience with AWS, negligible previous exposure to Docker and no prior experience with Terraform or with Elastic Stack.

One of the big differences between being a good hobbyist with Linux and working commercially with it is dealing with proprietary software. You can use and configure all sorts of systems indefinitely on your own account and never come up against dealing with awkward proprietary software that is supposedly officially supported for your platform that you really need to make work. Recently I had this experience. Not only was I able to get it working but I was able to extend it beyond the manufacturer’s original provision to make it more user friendly and less work to administer.

Configuration management tools are a big deal these days.  Just as with Puppet, Chef and Salt, a lot of material written about Ansible presumes either a lab environment or one focussed on servers, or both. Virtualisation is also considered ‘a given’- Docker and Vagrant are both popular- and the guides expect that you are implementing on green-field estate. I thought it would be useful to write about real-world experience and use in the opposite scenario: desktop more than server, physical rather than virtual machines and a ‘mature’ environment. Oh, and learning as we go along.

Documentation. Not something to get excited about is it? Who wants to read a manual? Or a policy document? A reference guide even? How about if we could make it easier? More accessible? More relevant? I think we can, using ebooks, and in this paper I will expand on this. In case it isn’t obvious, this paper is intended for people with a technical interest. I give a description of the real world challenges and then 2 real-life examples illustrating where ebook solutions answer them. The majority of the software referred to is free and/or open source. None of it is meant to constitute a commercial endorsement.