Monospace Mentor

Coders have to grow

2026-09-17T00:00:00+01:00

This video was at the top of the almost endless collection of software engineering videos in my RSS feed reader today:

Dave published his advice to grow out of the “coder” role last year, long before LLMs conquered the world, but it only gained in significance since. Translating specifications into code is becoming the domain of coding agents. It’s the definition of those specs and the design of the code’s architecture where software developers now find their work cut out for them.

Coders will have to grow to stay relevant. Which is precisely what I want to help you with in The Server Room.

Self-hosting is getting so easy

2026-08-31T15:27:51+01:00

Last week, the progress sync for the ebooks I read in KOReader on my various devices broke. No useful error message, just “Check your network connection”. The connection was fine. I assumed that the KOReader sync server was down, and decided to wait for a few days. It’s free software and a free sync service, no complaints. Last night, it was a few days later, and the sync server was still unreachable.

It was bedtime, but I didn’t want to keep reading on my Kindle Oasis without being able to pick up at the point I left off on my phone the next day. I went back to my computer, looked up my options for a self-hosted sync server, and not even 15 minutes later, I logged into my own KOSync instance.

Why did it go so quickly? Because of two things: One, the project is providing a container image and good documentation of the setup. Two, I already had a VPS running Podman containers and Nginx Proxy Manager. That investment is now paying dividends. All I had to do was create two small Quadlet files (one for the application and one for its data volume), assign a new subdomain to the server, and add its entry to NginxPM. Done.

Running your own services is getting easier and easier. It doesn’t require my decades of web operations experience any more. Which is awesome.

AI can be a great help in doing the wrong thing well

2026-08-29T00:00:00+01:00

I would not have started my Greenhouse side project last month if I weren’t able to delegate most of the coding to an LLM while I focus on product vision and engineering guidance. This week, I got to experience what can happen when that guidance is missing.

After shipping the Twitch integration for Greenhouse, my coding agent suggested a follow-up item to make it more resilient. As an old operations engineer, resilience is something I can always get behind. The agent’s stated goal was:

Replace the ephemeral EventSub ID on a Host connection with one durable subscription ownership record that persists through credential cleanup. It establishes one Workspace/broadcaster owner, a safe cleanup tombstone, independent EventSub operation fencing, and migration/backfill from deployed #53 [the previously shipped ticket] connections.

I don’t know about you, but I did not understand a single word. I knew nothing about the OAuth and EventSub mechanisms involved, so I trusted that the LLM understood them. I approved the follow-up ticket, which finally led to a day of wasted time and effort.

The first implementation round took hours and ended with an isolated agent review concluding that the result had too many flaws to salvage. Wow. I initiated a second attempt, but instructed the agent to break the task down into smaller chunks that it could implement in a more focused way. I did not yet know that this was a big mistake. A few hours later, the apparent success seemed to vindicate my intervention. There was a lot of new code with accompanying test cases, and it appeared to account for every situation Twitch and the application could conceivably encounter. But its complexity was far greater than a simple interactive website should need. As an example, there even was a code path for recovering from the case that Twitch issued a token, but the application couldn’t store it successfully. Overengineered much?!

I requested a critical review of whether the everyday usefulness of the new code warranted the added maintainability burden. The answer amounted to “Heck no”.

When I asked where it had gone wrong, the LLM said that its own use of distributed-systems language had sent it towards protocol optimisations far beyond what Greenhouse needed. It knew that Greenhouse was not safety-critical, but did not let that constrain the design.

Ultimately, I failed to recognise after the first failed attempt that it was building a castle where I needed a house. I could not read the blueprint well enough to stop it.

The lesson I learned from this is that it’s not enough that I review the results when the agent is done. Before a single line of code is generated, I need to understand what the agent is setting out to do. Even if it means admitting that I have no clue what that is. Especially then.

As Peter F. Drucker once said:

There is nothing so useless as doing efficiently that which should not be done at all.

AI took my job as a coder and made me a better software engineer

2026-06-29T09:10:00+01:00

I’ve been doing agent-supported coding for a few weeks now, and it definitely has changed the way I work. It’s not worse, just different. Which is remarkable, given that many programmers are having a disturbingly different experience: LLMs generating bad code, introducing bugs, even destroying data. I also read accounts from people lamenting skill atrophy, loss of job satisfaction, and a feeling of disconnection from their own codebase. It sounds like for these developers, the LLM became the mythical Monkey’s Paw that gives you what you want, and then takes away something dear to your heart.

I don’t want to find myself in this dark place eventually. I also don’t want to find myself stuck when LLM prices skyrocket, or the whole AI bubble bursts. The reason why I don’t think I will end up stranded is my attitude towards coding, or rather, to software engineering.

Passive and active guidance

Working with an agent is when good coding standards pay off. I got to enjoy that satisfaction before. It was when we introduced linting, unit and integration tests to our infra code a few years into our operation. We quickly realized how our confidence in the code we deployed was growing, while the amount of nightly emergency fixes was going down. Now, tests and linting rules are the guardrails that prevent agents from introducing regressions and becoming too creative with their new code.

Even with these guardrails in place, I don’t rely on the agent to find its way alone. My agent sessions look like pair programming, with the agent as the driver and me as the observer. I see other developers switch their attention to other things while the agent does its thing. I follow every step it takes. Not just because I don’t fully trust it. I also keep asking questions about its approach, and intervene to help it find the best solution. Every task results in a merge request that we review thoroughly before we integrate the changes into our codebase. (I just found a neat way to do these reviews in the terminal, about which I’ll write soon.)

Many years of experience have taught me that this kind of investment into “boring” engineering practices pays dividends. I don’t see why that should change with LLMs.

From programming to engineering

I don’t care how software is made, I care about what it does in the end. Loris Cro put it well in his article “My Software North Star:

Software should be useful to the end user
Software should be correct
Software should be maintainable and efficient

But how do you achieve that?

Before I started to use coding agents regularly, my many responsibilities at work left me little time to practice what I’d call “engineering discipline”. My process was: do a bit of thinking in advance, start coding, review, deploy. It did the job, but there wasn’t much “engineering” to it, and it always felt a bit shallow.

The software engineering methods I read about in the 80s — SA/SD, Jackson SD, SADT — are long dead, and deservedly so. But their intent didn’t die. Today’s methods have different names, and they tend to be leaner: discovery instead of requirements analysis, TDD instead of formal verification, ADRs instead of binders full of design docs. The problem is, they’re only “light” relative to 1985, but still not at all light for a team of two. Learning these engineering methods, let alone practising them regularly, always felt like a luxury we couldn’t afford. AI agents changed these economics.

Get it in writing

Like everyone, I started out by asking the LLM to implement small new features. (I found it hilarious how much better it was at Rails coding than me.) Soon, I picked up other people’s practice of ending prompts with the question of what else it needed to know. This alone eliminated a lot of backtracking and do-overs, and evolved naturally into having the agent generate a comprehensive plan first, and implement that plan in a second phase.

Plans require decisions, during both their inception and their implementation. Realizing the advantage of documenting our technological direction, I started dabbling with Architecture Decision Records a few years ago. ADRs are documents that preserve the thoughts behind decisions with significance for the shape of a software. Usually, those are also the kind of decisions that are difficult to revert later. In 16 years at the same company, I learned to appreciate the value of organizational memory. If only writing those documents wouldn’t take so much effort, especially if you’re not writing in your native language! I don’t think I ever wrote more than a handful. Thanks to a coding partner that takes great care of that, I now have a dedicated repository just for ADRs.

Once I got more proficient at guiding that coding partner, my next step was to break out of the feature-building box. I started involving the agent even before launching new projects. An extensive back-and-forth that I call “crucible sessions” results in clear documentation of a project’s motivations, goals, and most importantly, its scope. Staying focused has always been a struggle for our team. There’s always the temptation of tackling random issues we discover during project work. It’s easy to rationalize the distraction as “a good opportunity”. Starting with a clear definition of a feature and its scope helps us tremendously with shipping it without detours. In the process of refining this workflow, I learned that this kind of document is called a PRD. The agent saves them on Linear as project documents.

You might notice a pattern: all these improvements preserve knowledge and create transparency. This pattern now repeats like a fractal on all levels of my work. I’m writing less code, but I’m creating more value.

There’s more than just code

Like Mark Betz, “I don’t miss writing code”. I don’t feel disenfranchised by the agent that does it for me.

It’s probably an identity thing. My job has never been to deliver code, or to provision servers, or to type words into a blog editor. My job is to keep our business on a healthy trajectory, technologically and otherwise. I’m enjoying being able to do that job more effectively than before.

Did it turn me into an AI Booster? Hell no. I struggle to reconcile the many quandaries around AI usage with my family’s addiction to food and shelter. My message is not “Use AI, it’s great!” It’s: “When you have good software engineering practices, everything works better.” With or without LLMs.

I don’t believe this new era is “Game Over” for developers. It’s a forcing function to get clear about what we’re building, and to communicate well how we’re doing it. If that resonates with your situation, The Server Room is where we can figure it out together.

The motion that changed everything

2026-06-21T16:42:20+01:00

I switched to Vim a long ago after using a series of GUI editors like TextMate and Sublime Text. The urge for novelty has me still try out GUI editors sporadically, the most recent one being Zed. But I always come back quickly. I’m doing all my engineering work from the terminal, and by now my tooling is more or less complete and highly optimized for efficiency.

What brought me to Vim was the thought, “It’s installed on all my Unix and Linux servers, I should have at least some basic proficiency.” What made me stay was its “command grammar”; first and foremost, Vim motions.

I have to admit that even after decades, I only rarely use the number-based motions. Instead of 4dd, I mark lines for deletion in visual mode. And although I have relative line numbers switched on, I still end up holding k like an animal.

What makes Vim great for me are the motions that are based on text. For example, quickly jumping to the next colon with f:. Or selecting the current word with * and immediately being able to jump to its next occurrence with n.

Another neat feature isn’t exactly part of Vim motions, but text objects are closely related. It’ll never stop to make me smile when I can edit the contents of a pair of quotes or parentheses as easy as ci) or da".

BTW, if you’re looking for a comprehensive Vim tutorial, try the freely available chapter “Text Editing Fundamentals”, part of my community course “Basic Linux System Administration”!

This post was inspired by and written for “Hyde’s Vim Carnival”.

Fixing Flatpak Obsidian CLI

2026-05-08T21:18:11+01:00

I just tried to use the new CLI tool for Obsidian for the first time. To set it up, enabling a toggle switch in the “General” section of the settings should suffice. But if Obsidian is installed on Linux as a FlatPak, all you get when you execute the command is:

$ obsidian
The CLI is unable to find Obsidian. Please make sure Obsidian is running and try again.

The problem is that FlatPak isolates each application for safety reasons. In consequence, the Obsidian socket gets set up, but not at the location where the CLI expects it. This socket is a special file through which the CLI tool and the Obsidian application communicate. That’s why the CLI suspects that Obsidian might not even be running, even if it is.

The fix is simple: We put a symbolic link where the CLI expects the socket, and let the link point to where the socket actually is:

ln -sf /run/user/$(id -u)/.flatpak/md.obsidian.Obsidian/xdg-run/.obsidian-cli.sock /run/user/$(id -u)/.obsidian-cli.sock

With this fix, the CLI is happy:

$ obsidian version
1.12.7 (installer 1.12.7)

Now I can explore my Obsidian vaults from the shell!

By the way, you can read my DevOps notes that I manage in Obsidian right here on my website!

And if you’d like to learn more about symbolic links and the Linux filesystem in general, check out my course “Basic Linux System Administration”.

Going back to the roots for our community chat

2026-04-26T16:18:43+01:00

When Discord announced the introduction of age verification, my first thought was, “That’s what I’ve been preparing for.” I logged into one of my machines and set up an IRC server.

Like many open-source communities, The Server Room has been using Discord as a chat platform for years. But I always felt like I was building on a brittle foundation. I knew that it was only a matter of time until I had to find a more future-proof solution for our real-time conversations.

When Discord announced the introduction of age verification, potentially requiring official identification and photographic evidence, that time had come. Many of our community members are working in (or adjacent to) cybersecurity. There was no way they’d be happy to provide deeply personal information to an opaque business (or to its even more questionable business partners) just to have a chat.

Even though Discord backed off their initial plan, I’m happy that I made the change.

The Problem With Building on Someone Else’s Land

The barrier to entry to Discord is low, for server operators, and even more so for community members. In comparison to other chat platforms, its less ephemeral “forum” channels that retain messages for later reading seemed ideal for a community platform that is all about sharing and preserving knowledge. But there’s a catch. (There’s always a catch.)

Discord is a business, businesses need to make money, and it was clear from the start that it won’t offer its service for free without any downsides forever. Discord looks like the ideal candidate for the Enshittification effect - a sudden push for monetization, harming its users once it reaches a certain level of popularity and funds starts running out.

The TSR community is at the heart of my whole operation. Not only is my teaching work funded by membership fees, it’s based on connections between humans. Members don’t just study my course materials in isolation, they help each other grow. I can’t build this community on a product that can take a nosedive at any time. Joan Westenberg wrote about this kind of dependency in “The creator economy trap”. I’ve seen communities switch platforms for all kinds of reasons, and every time, they lost members in the process. For a fledgling community like mine, this could be an extinction-level event. Owning your community platform not only impacts cost and control, it also is the prerequisite for continuity.

For that reason, I set up a forum using the popular open-source software Discourse (the similarity in names is so confusing) in January 2024. Other than Discord, the forum gave me confidence that the useful conversations in my community would be preserved and easy to retrieve at any time. Within a short time, the forum became the centre of our community’s activities. With this part in the bag, real-time chat needed a reliable solution, too.

Why IRC, Though?

While Discord dominates the market of chat solutions, there are alternatives. When you look for open-source solutions, you’ll find Matrix, Mattermost, Zulip, and Discord clones like Stoat. And even among proprietary group chat services, there are well-liked options such as Signal. Why on earth did I choose the oldest chat platform of them all?

One reason is certainly that I’m even older than IRC, and I used it daily in the early years of me having access to the internet. The experience of being able to chat with people in distant countries was magical, and it informed my understanding of the internet as a “global village”.

While I did receive a gentle mocking as “the old sysadmin” when I mentioned IRC as a chat alternative, there was little pushback. People rather seemed intrigued that IRC might still have its place in internet-based communication.

As a first experiment, I created a channel on Libera.Chat. Additionally, I set up a bridge between it and our main Discord channel, et voila: people were able to chat from their IRC client applications. This only took me an hour, and most of that went into the initial bridge configuration.

Another reason for my choice was that setting up an IRC server is easy. With a modern implementation like Ergo, you can get started within an hour. You don’t have to deploy a web application backed by multiple database services, there’s no need for a complex Docker stack.

Then, IRC clients are available for every popular desktop, tablet and phone OS. In addition to the GUI apps, there are (of course!) well-maintained terminal-based ones as well. I’m using weechat, for example.

Finally, IRC is still actively developed. The latest protocol revision IRCv3 brings quality-of-life improvements such as being able to retrieve chat history when connecting. With earlier IRC versions, you’ll either miss what happened in chat while you were offline, or you’ll have to set up a client application (also known as a “bouncer”) that keeps a continuous server connection. Ergo has chat history enabled by default, and modern clients such as Halloy take advantage of it.

And as a bonus, I just found out that the IRCv3 project has a Code of Conduct, which is a good sign that they’re a reliable part of the online community.

All this, together with lively IRC communities like Libera.Chat, proves that IRC is still going strong after four decades. What’s not to like?

The technical details

Getting the IRC server started on one of my virtual private servers didn’t take longer than it took setting up our Discord server back in the day. Launching a Discord server takes a few mouse clicks. Installing Ergo from the Fedora package repository took a few keystrokes.

I admit that I took a detour setting up TLS that cost me another hour. First, I tried to put the IRC server behind the Nginx Proxy Manager that I already had running for serving websites, but I couldn’t get the Gamja web interface to connect successfully. From what I found online, IRC requires a few esoteric proxy settings that are easier to configure when you can modify the nginx configuration directly, instead of having to use the limited options of the NginxPM web interface. In the end, I configured Ergo to open a TLS port itself, and installed Certbot to handle the certificate renewal with Let’s Encrypt.

Installing Gamja was a quick job, and here I was able to successfully employ NginxPM. Thanks to Gamja, community members who don’t want to install a local IRC client have the option to access chat from their web browser.

Matterbridge is a nifty self-hosted service that can cross-post messages from one chat service to another. That way, there’s no split between the people adopting IRC and the ones still stuck on Discord. Discord and IRC are only two items in the long list of services that Matterbridge supports. Since I had already been using it to bridge our channel on Libera.Chat, I just had to modify the connection details to switch to our own IRC server. Then, I added bridges for two more channels, just to have the most important ones covered.

I used the Discord API and a few bots to build admin workflows, posting of news and social media messages, and a few other fun features. IRC has been using bots for the same purposes for much longer. With Limnoria, I was able to replace all the features I wanted to have from the start. The list of available Limnoria plugins is long and provides a lot of useful (or simply fun) features. Here are some of the plugins I set up myself:

SedRegex

allows you to correct a previous message using a quick

s/what you typed/what you meant to type/

. (Did you know that Skype had this built in?)
With

Later

, you can leave offline members a message that they’ll see when they post their first message after returning.
The

Karma

plugin keeps tally of up- and downvotes for a member or another thing. A neat way to show appreciation or annoyance.
The 3rd-party

Grapnel hooks

plugin installs a web API that I can use to post messages from the outside, mainly from automations I build on

Make.com

.

In total, I spent less than a workday on getting our new community chat platform to a sufficiently usable state. That’s perfectly okay, don’t you think?

Conclusion

With little effort, I was able to set up an online chat platform for my DevOps community that gives members a choice of either the convenience of Discord, the flexibility of their own IRC client, or the ease of an IRC web interface. In case I have to ditch Discord after all, there’ll be a well-established alternative chat service in place already. It’s nice to be able to focus on our conversations without worries.

And you’re welcome to join them, either on IRC or on Discord!

The systemd age attestation controversy

2026-04-01T13:29:03+01:00

In yesterday’s Office Hour, we discussed systemd, especially systemd timers. Then the question was raised why there is so much backlash against systemd in some circles. Here’s a short summary of my answer:

systemd is becoming kind of an “OS within the OS”. That this runs counter to the Unix philosophy’s “small programs working together” rubs people, especially the more puristic ones, the wrong way.
systemd author Lennart Pöttering has strong opinions, and some people seem to think that they have to offer even stronger counter-arguments, to put it lightly.

Technical differences can and should be debated. We have a wealth of talented developers in the open-source community. By putting their heads together, they have created, and will keep creating, remarkable pieces of software.

What should never, ever, happen, is that developers get death threats for their decisions. That actually happened after Dylan M. Taylor submitted a pull request to add a field to store (NB: and only store) a user’s birthdate in systemd’s JSON user records. Here’s a gift link to an LWN article that lays out the whole story well.

I find behaviour as described in the article absolutely unacceptable. In Open Source, we’re a community, and mob mentality has no place in it.

Launching a free online Linux administration course

2025-09-02T11:58:30+01:00

I had planned to start the cohort-based course “Basic Linux System Administration” later this week. When this cohort unfortunately didn’t fill up, I realized this was the perfect opportunity to do something I’ve wanted to try for a while: offer my Linux System Administration course completely free as a livestreaming experience.

Starting this month, I’ll be delivering my complete “Basic Linux System Administration” course through weekly live lectures on my streaming channels. This isn’t just a recording of classroom sessions—it’s a fully interactive learning experience designed specifically for the online format.

What You’ll Get

The course covers everything you need to get a solid foundation for a system administration or DevOps role. We’ll start with the fundamentals—installing Linux, mastering the command line, and understanding how Linux actually works—then progress through practical skills like user management, process control, networking, and security.

Each week includes live Q&A sessions where you can get direct answers to your questions, plus assessment quizzes to verify your understanding. The hands-on labs use real Linux systems, so you’ll gain practical experience with the same tools used in production environments.

Why I’m Doing This

After decades in software development and managing Linux systems, I’ve seen how transformative these skills can be for someone’s career. The demand for Linux expertise continues to grow, but quality education shouldn’t be locked behind high course fees.

This streaming format lets me reach people who might not otherwise access this training—whether due to location, schedule constraints, or budget limitations. Plus, I’m genuinely excited to experiment with live online education and build a learning community around practical Linux skills.

Get Started

All the details are on my course page. You can sign up there to receive notifications about lecture schedules and other important updates.

The first session covers Linux fundamentals and walks you through your first Fedora installation. Whether you’re entirely new to Linux or have some experience but want to formalize your knowledge, this course will give you the solid foundation you need for a successful career in system administration.

Let’s build something great together. See you in the stream!

—Jochen, The Monospace Mentor

A homelab is a skill booster

2025-08-09T12:50:04+01:00

In her article “What’s the biggest unsolved problem within Site Reliability Engineering?”, Paige Cruz identifies a critical challenge facing our industry: how do we pass the hard-won knowledge of veteran system administrators down to a new generation that has never worked outside cloud infrastructure?

Cruz makes a compelling case that this knowledge transfer gap represents SRE’s biggest unsolved problem. However, I take issue with one of her conclusions. She argues that “It’s also not acceptable or sustainable or accessible in my opinion to continue to expect people set up home-labs and experiment themselves. We don’t ask veterinarians to practice surgery in their free time at home and I don’t think we should expect that from future SRE.”

I respectfully disagree. A home lab represents the best investment a future SRE can make to secure their career growth.

The home lab advantage

The veterinarian comparison doesn’t hold up under scrutiny. Veterinarians need live animals, surgical equipment, and sterile environments – resources that are neither practical nor ethical to maintain at home. Particle physicists need billion-dollar accelerators. But what does a sysadmin or SRE need to experiment with infrastructure? A few decommissioned Lenovo ThinkCentre boxes and a network switch.

The barrier to entry is remarkably low, and the learning potential is enormous.

Scientific research consistently shows that skill development requires more than passive consumption of learning materials. Growing expertise demands practical exploration – running into problems, analyzing failures, and working through solutions. A home lab provides exactly this environment, free from the constraints and pressures of production systems.

You don’t need a data centre in your basement. One or a few small networked PCs will serve you well. These machines consume minimal power and don’t need to run around the clock. Fire them up when you want to experiment with a new service mesh, test disaster recovery procedures, or break something just to see what happens.

For those who can’t or don’t want to maintain physical hardware, alternatives exist. “Pubmox” – the shared Proxmox Virtual Environment I maintain for members of “The Server Room” – offers free access to virtual infrastructure for hands-on learning.

The learning laboratory

A home lab creates space for the kind of curiosity-driven exploration that builds deep technical intuition. When you’re troubleshooting a networking issue at 2 AM in your home lab, you’re not worried about affecting customers or violating change management policies. You can take risks, make mistakes, and learn from the consequences.

This environment encourages the experimental mindset that separates great SREs from those who merely follow runbooks. You learn to ask “what if” questions and then actually test your hypotheses.

Knowledge transfer requires community

While I advocate strongly for home labs as skill boosters, they’re not the complete answer to Cruz’s original question about knowledge preservation. The solution to transferring veteran sysadmin wisdom doesn’t lie in solitary exploration – it requires collaborative knowledge transfer.

Home labs excel at building technical skills, but they can’t replace the mentorship and context that comes from working alongside experienced practitioners. The real magic happens when hands-on experimentation combines with guided learning from those who’ve solved these problems before.

This is why I’m launching “Basic Linux System Administration” – an instructor-led course backed by “The Server Room” community. Participants get free access to Pubmox, allowing them to practice what they learn while benefiting from my decades of system administration experience.

It’s the best of both worlds: the freedom to experiment and break things, combined with the guidance to understand why systems fail and how to make them more reliable.

A home lab won’t solve SRE’s knowledge transfer problem on its own. But for individual practitioners looking to deepen their skills and advance their careers, it remains an invaluable investment. The combination of practical experimentation and collaborative learning creates the foundation for the next generation of systems experts our industry desperately needs.