I seriously feel like I wrote this somewhere recently, so please forgive me if I’m repeating myself, but:
In the early days of my hosting of BB, I didn’t have a “server” per se, so I used my own P233MMX desktop, which I installed RH Linux on, hardened, and put in that “Datacenter of Front Street” that Cory references in his story: 151 Front Street West.
The thing was, the hosting partner we were hosting with didn’t have rackspace for tower servers, so we brought - I kid you not - an Ikea IVAR shelving unit into his cage to rest the server on. I can’t even imagine seeing that nowadays, but it stayed that way for quite a long time (at least a year), until I finally upgraded to some used Dell rackmount server from a friend instead.
A good sysadmin, IMHO, is reasonable and listens to the needs of users, but stands firm and fights for the core values of their infrastructure. some of the most vociferous and venomous battles I can remember fighting were around things like root access and password-based SSH auth.
The last group I worked with had servers not in the data center (which is why I was brought into the group so I could do the sysadmin work for them) and they really didn’t believe that security would take their boxes off the network for not updating or whatever other reason. I had to tell them multiple times OH YES THEY WILL. I knew they did cause I got volunteered at 11pm to go into the data center and power a box down via the power button as I didn’t have access/permissions. Then pull the network cables from it.
RIP. I suppose this was in the days before Debian - no easy autoupdating - and before kpatch became a thing? It’s so stupidly easy to implement these days. Or you could do what I do, have a Pi pinging its address and send an email alert if server fails to come up after reboot.
Well I am a microsoft guy by my job description. Where I worked was a RedHat house for linux and oh building those boxes were a PITA compared to the windows boxen and I got laid off before the next round of linux updates which would have gone from 6.4 to 7.2,3,something and it was gonna a be nuke and start over update. I really didn’t like building the linux servers.
It was a virus event and a box that the other windows team supported the box in question. Network had already shut the switch port down. I got a call from my manager with the security team on as well, they said this is prop tag, label, rack location… go kill it, so I got my clothes on and drove to the data center and killed it. I imagine someone got on the console with the fix when we got it later.
That depends a lot on your application. In the (simple) case of BB, I could autoupdate the servers, because they’re redundant so it doesn’t matter if they come back up or not.
But in a serious installation if I autoupdate the application servers and some aspect of the update bricks the application, I’m probably out of a job. you need to QA infrastructure changes and do a staged rollout, which means little is automated.
This skillset, BTW, is one of the hardest things to learn unless someone hands you the keys to a really big network (or a network where downtime literally costs $$$ per second). I’ve hired and trained a LOT of sysadmins, and getting them to change their thinking from “small” to “big” was always the hardest part.
One of the reasons I love maintaining the BB infrastructure is that even though I get to do enterprise-y things like redundant shared storage and config management, it’s a tiny, simple setup and redundant enough that if stuff breaks, no one notices and I can fix it at my leisure. That’s a serious departure from the day-job version where something being down usually means we’re a single subsystem away from losing thousands of dollars a minute…
Thankfully for the world, I’m no longer involved in IT - I pursued, briefly, a career in serving food instead of bits. Less stressful they said. It’ll be fun they said…
The IT guys I worked with would set some computers and end-users as the expendable cannon fodder advance testers: they would get auto-updates, and if no issues cropped up, the rest of the end users would then receive that week’s patches. Not sure how many they’re managing at the moment; when I left their employ, they had just upwards of 1500 end users across 23 client companies.
Getting redundancy set up is IMHO, the single best thing you can do. For home servers - and I suspect most readers of this topic have one - setting up a RAID 5/50 is key. I intend to set up a home server this summer for diff-based backups of desktops and general media storage.
Most IT PFYs are self taught and learned their craft by cobbling together solutions on shitty handdowns. I am one such. One of my guiding principles is the 90% functionality rule: if I can get something functioning correctly 90% of the time, I am satisfied. Bailing water, that’s what I do. Designing actual solutions and networks and stuff? Above my paygrade. Maybe one day I’ll have the time and funds to set up a proper test lab and take time to learn all proper and such.
I imagine that you’ve had to write some patches for the software bb 's built on yourself and develop some scripts to make sysadminning a little easier. Have you a github?
I do not. My current day job owns all my work and it’s stupidly hard to get stuff “vetted” for public consumption.
As for BB itself, I can thankfully say other folks (including @beschizza) are responsible for banging on themes and plugins and the like. My role is the infrastructure and middleware setups (and security, and capacity planning, and so on!), but most of it is custom stuff shoehorning bits together, not the sort of thing that would be useful for others.
I might build an inotify-based sync tool for BB 3.0, though, because no such tool appears to exist that meets my needs, and going the other way an implementing some sort of clustered file system seems like an awfully big hammer for this particular nail.
I’m editorial-end only these days; design ‘n’ wordpressery is Scott’s bailiwick now. (A good thing given the complexity of adtech, it could become all-consuming for a week at a time).
You are welcome to use mine. It is pretty straightforward, has a simpler and easier to use interface than the cron-based ones, and since it’s written in C it’s pretty portable among distros and revs. I have compiled it with no source code changes on RHEL3 through RHEL7, so even systemd didn’t phase it…
Basically you use your local inotify constants to define path triggers (like, if someone does CLOSE_WRITE in this folder) and hits on the trigger call a user-written scriplet written in your language of choice (I use perl and bash mostly, but it doesn’t care if you want Java, PHP, or even Brainf***).
The code’s tight and short enough that you can audit it yourself, even if you are not a C-lord.
The last openly published version is at http://typinganimal.net/code/gidget/ , and I don’t think there have been any in-house changes but I will check my internal git repo history just in case. You build it with make, like a archetypical unix program.
If you are familiar with Centos/Red Hat, which it sounds like you are, you’ll find the init and logrotate scripts self-explanatory. I have not yet built a systemd unit file to replace the initscript because it runs fine under Red Hat’s service init architecture as is.
Feel free to PM or email me if you have questions!
