Things to Consider for a SysAdmin???
Hi.
Getting ready to post to fiverr/freelancer/etc for a couple of part-time sysAdmin people.
If you have any thoughts on skills I should be looking for,, hit me up or add to my list.
If you're interested in talking, let me know as well!
:Dev Process
create the app, requires team members to interact with the codebase
interface with backup/version control
identify version control/backup tools
handle user access for system
identify/handle linu systems admin functions
system security - ssh/log files/dos attacks/etc..
dns functions
apache/web functions
iptables
DoS/system/internet attacks
log management/security
cloudflare/new relic
handle/manage/setup database processes
duplicate/redundancy to spin up systems in event of crashing
alert/alarm systems in event of system/apps having issues
in the event of centos/selinux -- setting up/managing selinux attributes
-docs/wiki/github as required
-define required tools
identify required services/remove/disable other services
system monitoring functions
-work with the app devs to nail version control/backup for the web/mobile apps
for the overall project
:Test Process
system integrity checks
-work with the app devs to nail the testing for the web/mobile apps for the overall project
-how to test the system/backup processes
-how to test the application processes
-test database processes
-automate testing
-help identify/develop app testing -- unit/system/etc
-everything from the above Dev Process
-anything else??
:Production Process
-develop/automate scripts/processes to move from the testing -> production process
-update/rollback testing -> production process as required
-SECUIRITY -- system/application/databases
-work with the app devs to nail the web/mobile apps for the overall project
The role of the sys admin will be critical to building the tools/process
for the secuirty of the web/mobile app for the project. We're looking for
people who've actually build dev/testing/production environments within
a cloud/linux environment.
The sys admin will be expected to transfer knowledge, as well as to help
implement the required tools/processes to the devs/team members. It'll be
the role of the sys admin(s) to work to identify what has to be done in order
of importance during the process.
thanks
-bruce
I'm a little confused as to how much of the list you expect these persons to do. If you want someone to do all of that well, you're probably looking at a $200k/year salary, if they have the time to implement/maintain everything. If you restrict it to just the DevOps/SysAdmin tasks, you can probably find someone for $100k/year.
I'd suggest looking for a platform that does CI/CD completely for you. If you want to run your own process you're going to spend a lot more money. I'd also suggest using cloud managed components as much as possible for the same reason.
I'm also concerned that you expect one person to just add security like that. Most security issues happen on the application side.
Anyway, you really need to breakdown which concerns in the list would be the person's responsibility if you're going to get any responses from people who actually know what they're doing.
Hey.
Thanks for the reply.
Everything you posted is on point. The list was/is intended to be a starting point to get one's head around what's needed. There might also be other issues that I hadn't even thought of (or forgot about. Some issues are more critical given where a project is in it's journey.
At the same time, some functions don't require a complete sysAdmin resource but need to be setup and then periodically checked.
As to the cost of a sysAdmin, if the only way a business can have a legit sysAmdin/devOps process is to have a 100-200K then most businesses are pretty well f**ked. I reject that. I also don't expect to find anyone looking for a side gig to do any of this for a few hundred.
I would also say that anyone thinking of doing anything of complexity on the net where you deal with customer data/funds/etc.. to had better damn well think about a good number of the issues I've raised or you're essentially building something with the proverbial a** hanging out in the wind!
Just my $0.02 worth.
peace
Let me clarify a bit of what I wrote above. DevOps/SysAdmin is what I do for a living, and I really just want to help you avoid pitfalls and save time in the long run.
If you want to do things as affordably as possible, when you're still at a small scale (say less than $1M/year spend on hosting), the best thing you can do is to outsource as many processes as possible. For hosting, Infrastructure-as-a-Service providers, like GCE or AWS. For CI/CD, places like GitLab or CircleCI. There are many reasons.
The first is automation. Automation takes a lot of time to set up. At my place of employment, we decided to do our own Kubernetes cluster in-house. Over the past year and a half, getting the management of the cluster automated and necessary patches upstream has taken about a full time position of a very senior person (expensive!). We figure in a few months we'll have more time to focus on full automation of our CI/CD story. If you outsource these things, you can save so much time, and so much money.
The second is maintenance. With cloud managed components, so much of that is outsourced. Updating operating systems and organizing reboots, managing logs, growing disks, etc., are no longer a concern. I'm looking forward to the day when I run no more virtual machines, because it's just so much work, even when mostly automated! Let the cloud provider do it at 1% of the cost!
The third is support. If you hire someone part time, or even just one person full time, you're not going to have 24/7 support. People get sick and need vacations and will be unavailable. That outage you're experiencing might take a couple days to get looked at, etc. With outsourced components, you can get support faster. You can also hire other people who are familiar with them, whereas it's much harder to find help for something home-brewed.
If you hire an old school system administrator, you're going to get a system that requires a lot of maintenance over time.
What you probably want to hire is a DevOps/CI-CD/Cloud specialist who can set up that infrastructure you want, and to train your team on how it works. This kind of person should know:
If configured correctly, once set up, the above is mostly maintenance-free, will scale with load, and handle crashes, etc. This is reasonable to hire someone full time for a bit, then have do work on an as-needed basis if the rest of the team can't pick it up on their own. As soon as you throw a virtual machine in you're adding maintenance, so I'd try to avoid them as much as possible.
Things that are not on the above list (that seemed to be included in the original post):
I'll give you my thoughts on those three now.
Application development is really a separate things from system administration. It's unusual to find people who are great at both (so genuinely full stack people are very expensive; I've yet to meet a single person who is great at system administration and CSS). A good system administrator will know some coding and should be able to compile and make patches to applications, but almost certainly not going to be an expert. A good application developer will understand the basics of an operating system, but thankfully in the era of containers, this is less of a concern. But in this era of containers, a decent developer will know containers (or alternatively Functions-as-a-Service).
QA tends to handle the more complex test cases. Developers should probably be writing unit tests for their functions as part of their development process. A good QA person will be competent at writing automated testing, and should have experience with continuous integration pipelines.
Data modeling and query tuning (or data architecting) is almost entirely outside the purview of a sysadmin or devops person. I happen to be our DBA, but it's really its own role. Any decent developer should understand the basics of SQL (Even our customer support people know it for running queries in AWS Athena). And I've taught many people how to write performant queries, which is easy enough if they understand how a b+tree works. All that relies on a reasonable data model, which is not something I'd expect a sysadmin or devops person to know. An experienced developer or DBA can do data architecting, but there's not much database tuning that can be done if the data architecting is bad. Database work is something to hire someone on an as-needed basis. Data modelling done wrong can be exceedingly hard and expensive to fix.
You may notice I don't mention things like selinux, nginx, iptables, etc. These disappear if you're not running VMs and doing cloud-native deployments.
Some key things that everyone on the team show know:
And regarding security: it's a shared responsibility. Everything should be protected by 2FA, no exceptions, not for anyone.
The developers should have a process in place for keeping every library they add to their code updated. Many of the recent major vulnerabilities have been in libraries, and compromises are often a result of not updating out of date libraries in a timely fashion (within a week or two; a month is too long). Using a universal package manager (see the Wiki link above) can help detect vulnerable code.
I would also hire a security specialist now and then to audit the code and processes.
I hope this helps!