Nowadays, the majority of production applications run in containerized environments, most commonly on Kubernetes clusters. This approach provides numerous benefits, including portability, scalability, and a standardized API for managing workloads across environments. It has become the industry standard for modern application deployment. However, every rule has its exceptions. There are scenarios where running an application outside of Kubernetes is the more appropriate choice. One simple and reliable alternative for Go applications is systemd. Systemd is the standard system and service manager for most modern Linux distributions, replacing the older SysV init system. It is much more than a process launcher - it provides a rich ecosystem for service management, dependency handling, logging, automatic restarts, resource control, and system initialization. Here are two common scenarios where systemd is preferred over Kubernetes:
In this blog post, I'll show you how to build a production-ready Go application that follows the best practices for running as a systemd service. The good news is that there is nothing inherently different about writing an application that runs under systemd. From the application's perspective, it is just another process. The same principles that apply to any production-grade Go service apply here as well. The main difference lies in how the service communicates its lifecycle and health status to its supervisor. In Kubernetes, this is typically achieved through readiness and liveness probes. The application exposes an HTTP endpoint (for example, /healthz) that returns a 200 OK response when the service is healthy. Kubernetes periodically probes this endpoint and uses the response to determine whether the application is ready to receive traffic or needs to be restarted. Systemd takes a different approach. Instead of polling an HTTP endpoint, it communicates with the service over a Unix domain socket. The application sends status notifications to the systemd process (PID 1), informing it when startup has completed, whether the service is healthy, when it is reloading, and when it is shutting down. It's important to note that not every systemd service type supports these notifications. Systemd defines several service types, such as simple, exec, forking, oneshot, notify, and notify-reload. Only the notify and notify-reload service types support the notification protocol used for health reporting and lifecycle events. The notify-reload type extends notify by also supporting coordinated configuration reloads. Since it is a superset of notify, we'll use notify-reload throughout this article to demonstrate the complete set of production-ready capabilities. To interact with systemd from Go, we'll use the popular go-systemd library from CoreOS, which provides Go bindings for the systemd notification protocol and other systemd APIs.
Below is a minimal Go program that prints hello world on a ticker every second. It will serve as our starting point - from here we'll incrementally extend it into a systemd-aware service.
package main
import (
"log/slog"
"os"
"time"
"github.com/go-logr/logr"
)
func main() {
logger := logr.FromSlogHandler(
slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{
Level: slog.LevelDebug,
},
),
)
ticker := time.NewTicker(1 * time.Second)
defer ticker.Stop()
for range ticker.C {
logger.Info("[systemd-go-app] hello world")
}
}Before we start extending the program, a quick word on sd_notify itself. It is not an HTTP call or a systemd-specific library call - it is a tiny wire protocol built on top of a SOCK_DGRAM Unix domain socket. Each notification is a single datagram of newline-separated KEY=VALUE pairs (for example READY=1, STATUS=Serving requests, or RELOADING=1\nMONOTONIC_USEC=12345). Systemd creates the socket, passes its path to the service via the NOTIFY_SOCKET environment variable, and reads whatever the process writes to it. Anything that can open a Unix datagram socket can speak the protocol - no linking against libsystemd is required, which is why the go-systemd library is a pure-Go implementation. Its daemon.SdNotify(unsetEnvironment, state) function opens the socket, writes the datagram, and returns (sent bool, err error). When NOTIFY_SOCKET is empty, it returns (false, nil) immediately, which makes it safe to call unconditionally - but wrapping the whole systemd path in an explicit guard, as we do below, keeps the intent obvious and lets us skip building any of the surrounding lifecycle machinery when running outside systemd.
One small aside on the logger: we build a logr.Logger on top of a slog.TextHandler via logr.FromSlogHandler. The reason is ergonomic - logger.Error takes the error as its first argument (logger.Error(err, "msg", kv...)), which reads more naturally at every systemd notification site than slog.Error("msg", "err", err), and matches the convention used across a lot of systemd-adjacent Go code.
When a service is of type notify or notify-reload, systemd populates the NOTIFY_SOCKET environment variable with the path to this socket. Our first addition is a check for that variable - if it isn't set, the application is running outside of systemd, so we short-circuit before doing any lifecycle work.
package main
import (
"log/slog"
"os"
"time"
"github.com/go-logr/logr"
)
func main() {
logger := logr.FromSlogHandler(
slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{
Level: slog.LevelDebug,
},
),
)
+ if os.Getenv("NOTIFY_SOCKET") == "" {
+ logger.Info("[systemd-go-app] NOTIFY_SOCKET is not set; sd_notify support is disabled")
+ return
+ }
+
ticker := time.NewTicker(1 * time.Second)
defer ticker.Stop()
for range ticker.C {
logger.Info("[systemd-go-app] hello world")
}
}With the guard in place, we can send our first real notification: READY=1. This tells systemd that startup is complete and the service is prepared to run. Anything registered to depend on this unit (via After= / Requires=) is held back by systemd until this message arrives, so the notification must be sent after any warm-up work is done - not from an init() or the top of main(). In this minimal example there is no warm-up to speak of, but the daemon.SdNotify call site is what matters. We handle all three return shapes explicitly: an error aborts startup, a successful send is logged, and the third case - sent=false with no error - happens when the library falls through the guard, which we already ruled out above but keep here for completeness.
package main
import (
"log/slog"
"os"
"time"
+ "github.com/coreos/go-systemd/v22/daemon"
"github.com/go-logr/logr"
)
func main() {
logger := logr.FromSlogHandler(
slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{
Level: slog.LevelDebug,
},
),
)
if os.Getenv("NOTIFY_SOCKET") == "" {
logger.Info("[systemd-go-app] NOTIFY_SOCKET is not set; sd_notify support is disabled")
return
}
+ sent, err := daemon.SdNotify(false, daemon.SdNotifyReady)
+ switch {
+ case err != nil:
+ logger.Error(err, "sdnotify READY=1: %w", err)
+ return
+ case sent:
+ logger.Info("[systemd-go-app] sent READY=1 to systemd")
+ default:
+ logger.Info("[systemd-go-app] NOTIFY_SOCKET not set; READY=1 was a no-op")
+ }
+
ticker := time.NewTicker(1 * time.Second)
defer ticker.Stop()
for range ticker.C {
logger.Info("[systemd-go-app] hello world")
}
}Next comes the shutdown path. When systemd wants to stop the service - because of systemctl stop, a reboot, or an OnFailure= action - it sends SIGTERM to the main process. We install a signal handler and, on receiving SIGTERM or SIGINT, notify systemd with STOPPING=1 before exiting. The STOPPING=1 message lets systemd distinguish "the process is going away on purpose" from "the process crashed", which matters for restart policies and dependency ordering.
To make room for a signal-driven shutdown, the ticker moves into its own goroutine and gets a reloadCh it selects on alongside its timer - closing reloadCh tells the goroutine to stop. Right now that only fires on shutdown, so a plain done-channel would work just as well; the name pays off in the next step, when the same channel becomes the mechanism for tearing down a reload iteration.
package main
import (
"log/slog"
"os"
+ "os/signal"
+ "syscall"
"time"
"github.com/coreos/go-systemd/v22/daemon"
"github.com/go-logr/logr"
)
func main() {
logger := logr.FromSlogHandler(
slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{
Level: slog.LevelDebug,
},
),
)
if os.Getenv("NOTIFY_SOCKET") == "" {
logger.Info("[systemd-go-app] NOTIFY_SOCKET is not set; sd_notify support is disabled")
return
}
+ termCh := make(chan os.Signal, 1)
+ signal.Notify(termCh, syscall.SIGINT, syscall.SIGTERM)
+
sent, err := daemon.SdNotify(false, daemon.SdNotifyReady)
switch {
case err != nil:
logger.Error(err, "sdnotify READY=1: %w", err)
return
case sent:
logger.Info("[systemd-go-app] sent READY=1 to systemd")
default:
logger.Info("[systemd-go-app] NOTIFY_SOCKET not set; READY=1 was a no-op")
}
- ticker := time.NewTicker(1 * time.Second)
- defer ticker.Stop()
- for range ticker.C {
- logger.Info("[systemd-go-app] hello world")
- }
+ reloadCh := make(chan struct{})
+
+ go func() {
+ ticker := time.NewTicker(1 * time.Second)
+ defer ticker.Stop()
+ for {
+ select {
+ case <-reloadCh:
+ return
+ case <-ticker.C:
+ logger.Info("[systemd-go-app] hello world")
+ }
+ }
+ }()
+
+ <-termCh
+ sent, err = daemon.SdNotify(false, daemon.SdNotifyStopping)
+ if err != nil {
+ logger.Error(err, "STOPPING=1 failed")
+ } else if sent {
+ logger.Info("[systemd-go-app] sent STOPPING=1 to systemd")
+ }
+ close(reloadCh)
}The final lifecycle piece is hot-reload. With Type=notify-reload, systemd sends SIGHUP to the main process on systemctl reload and waits for a RELOADING=1 notification followed by another READY=1 once the reload finishes. RELOADING=1 must be sent in the same datagram as MONOTONIC_USEC=, which is the value of CLOCK_MONOTONIC in microseconds at the moment the reload starts. Systemd uses this timestamp to detect duplicate or stale reload notifications - if a second reload is triggered before the first finishes, the older MONOTONIC_USEC lets systemd drop the outdated READY=1.
A reload is a defined sequence: on SIGHUP, the service must tear down its current state, notify systemd with RELOADING=1 and MONOTONIC_USEC, rebuild its runtime state, and notify systemd with a second READY=1. Only then does systemd consider the reload complete.
The code models this as a labeled restartLoop. Each iteration represents one lifetime of the service: send READY=1, create a fresh reloadCh, start the ticker goroutine (and, in the next step, the watchdog) bound to that channel, then wait on a select over the two signal channels. On SIGHUP, the code emits RELOADING=1, calls close(reloadCh) to terminate the iteration's goroutines through their <-reloadCh case, and executes continue restartLoop; the next iteration would re-read configuration or rotate credentials before its own READY=1. On SIGTERM/SIGINT, the same tear-down runs but the notification is STOPPING=1 and the loop is exited via break. Note that signal.Notify is invoked once, outside the loop, to avoid registering a new channel with the runtime on every reload.
package main
import (
+ "fmt"
"log/slog"
"os"
"os/signal"
"syscall"
"time"
"github.com/coreos/go-systemd/v22/daemon"
"github.com/go-logr/logr"
)
func main() {
logger := logr.FromSlogHandler(
slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{
Level: slog.LevelDebug,
},
),
)
if os.Getenv("NOTIFY_SOCKET") == "" {
logger.Info("[systemd-go-app] NOTIFY_SOCKET is not set; sd_notify support is disabled")
return
}
+ // Signal handlers are registered ONCE outside the restart loop so we don't
+ // leak channels/goroutines across reload iterations.
termCh := make(chan os.Signal, 1)
signal.Notify(termCh, syscall.SIGINT, syscall.SIGTERM)
- sent, err := daemon.SdNotify(false, daemon.SdNotifyReady)
- switch {
- case err != nil:
- logger.Error(err, "sdnotify READY=1: %w", err)
- return
- case sent:
- logger.Info("[systemd-go-app] sent READY=1 to systemd")
- default:
- logger.Info("[systemd-go-app] NOTIFY_SOCKET not set; READY=1 was a no-op")
- }
-
- reloadCh := make(chan struct{})
-
- go func() {
- ticker := time.NewTicker(1 * time.Second)
- defer ticker.Stop()
- for {
- select {
- case <-reloadCh:
- return
- case <-ticker.C:
- logger.Info("[systemd-go-app] hello world")
+ sigHUPCh := make(chan os.Signal, 1)
+ signal.Notify(sigHUPCh, syscall.SIGHUP)
+
+ monotonicEpoch := time.Now()
+
+restartLoop:
+ for {
+ sent, err := daemon.SdNotify(false, daemon.SdNotifyReady)
+ switch {
+ case err != nil:
+ logger.Error(err, "sdnotify READY=1: %w", err)
+ return
+ case sent:
+ logger.Info("[systemd-go-app] sent READY=1 to systemd")
+ default:
+ logger.Info("[systemd-go-app] NOTIFY_SOCKET not set; READY=1 was a no-op")
+ }
+
+ reloadCh := make(chan struct{})
+
+ // The "core work" of this demo: print hello world on a ticker so
+ // there's something visible in the journal between signals.
+ go func() {
+ ticker := time.NewTicker(1 * time.Second)
+ defer ticker.Stop()
+ for {
+ select {
+ case <-reloadCh:
+ return
+ case <-ticker.C:
+ logger.Info("[systemd-go-app] hello world")
+ }
+ }
+ }()
+
+ select {
+ case <-termCh:
+ sent, err := daemon.SdNotify(false, daemon.SdNotifyStopping)
+ if err != nil {
+ logger.Error(err, "STOPPING=1 failed")
+ } else if sent {
+ logger.Info("[systemd-go-app] sent STOPPING=1 to systemd")
+ }
+ close(reloadCh)
+ break restartLoop
+
+ case <-sigHUPCh:
+ monotonicUSec := uint64(max(time.Since(monotonicEpoch), 0) / time.Microsecond)
+ msg := fmt.Sprintf(
+ "%s\nMONOTONIC_USEC=%d",
+ daemon.SdNotifyReloading,
+ monotonicUSec,
+ )
+
+ sent, err := daemon.SdNotify(false, msg)
+ if err != nil {
+ logger.Error(err, "sdnotify RELOADING=1 failed")
+ } else if sent {
+ logger.Info("[systemd-go-app] SIGHUP received, sent RELOADING=1 to systemd")
}
- }
- }()
-
- <-termCh
- sent, err = daemon.SdNotify(false, daemon.SdNotifyStopping)
- if err != nil {
- logger.Error(err, "STOPPING=1 failed")
- } else if sent {
- logger.Info("[systemd-go-app] sent STOPPING=1 to systemd")
+
+ close(reloadCh)
+ continue restartLoop
+ }
}
- close(reloadCh)
}There's one more mechanism worth wiring up: the systemd watchdog. If the unit file sets WatchdogSec=, systemd expects the service to send WATCHDOG=1 notifications at least that often. If a ping doesn't arrive in time, systemd concludes the process is deadlocked and kills it - usually with SIGABRT to produce a core dump - then restarts it according to the unit's Restart= policy. This is fundamentally different from the earlier notifications: READY, STOPPING, and RELOADING are lifecycle events, whereas WATCHDOG is a periodic liveness heartbeat. It's the systemd equivalent of a Kubernetes liveness probe.
Two details make or break a watchdog implementation. First, ping at roughly half the configured interval - the convention documented by sd_watchdog_enabled(3) - so scheduling jitter, GC pauses, or a slow syscall don't push a ping past the deadline. Second, the ping goroutine's lifetime must be scoped to a single reload iteration: we reuse the same reloadCh we already close on SIGHUP/SIGTERM, so the watchdog exits along with the workload goroutine and neither leaks across reloads. The go-systemd library exposes daemon.SdWatchdogEnabled, which reads the interval from the WATCHDOG_USEC environment variable systemd sets and returns 0 when the watchdog is disabled (or when WATCHDOG_PID points at a different process) - a natural way to skip starting the ticker at all when it isn't configured.
package main
import (
"fmt"
"log/slog"
"os"
"os/signal"
"syscall"
"time"
"github.com/coreos/go-systemd/v22/daemon"
"github.com/go-logr/logr"
)
func main() {
logger := logr.FromSlogHandler(
slog.NewTextHandler(
os.Stderr,
&slog.HandlerOptions{
Level: slog.LevelDebug,
},
),
)
if os.Getenv("NOTIFY_SOCKET") == "" {
logger.Info("[systemd-go-app] NOTIFY_SOCKET is not set; sd_notify support is disabled")
return
}
termCh := make(chan os.Signal, 1)
signal.Notify(termCh, syscall.SIGINT, syscall.SIGTERM)
sigHUPCh := make(chan os.Signal, 1)
signal.Notify(sigHUPCh, syscall.SIGHUP)
monotonicEpoch := time.Now()
restartLoop:
for {
sent, err := daemon.SdNotify(false, daemon.SdNotifyReady)
switch {
case err != nil:
logger.Error(err, "sdnotify READY=1: %w", err)
return
case sent:
logger.Info("[systemd-go-app] sent READY=1 to systemd")
default:
logger.Info("[systemd-go-app] NOTIFY_SOCKET not set; READY=1 was a no-op")
}
reloadCh := make(chan struct{})
go func() {
ticker := time.NewTicker(1 * time.Second)
defer ticker.Stop()
for {
select {
case <-reloadCh:
return
case <-ticker.C:
logger.Info("[systemd-go-app] hello world")
}
}
}()
+ duration, err := daemon.SdWatchdogEnabled(false)
+ switch {
+ case err != nil:
+ logger.Error(err, "SdWatchdogEnabled returned error; watchdog disabled")
+ case duration == 0:
+ logger.Info("[systemd-go-app] WATCHDOG_USEC not set; watchdog disabled")
+ default:
+ go func() {
+ // Per sd_watchdog_enabled(3): send a keep-alive every half of
+ // the interval returned here.
+ tickInterval := duration / 2
+ ticker := time.NewTicker(tickInterval)
+ defer ticker.Stop()
+ for {
+ select {
+ case <-reloadCh:
+ return
+ case <-ticker.C:
+ if _, err := daemon.SdNotify(false, daemon.SdNotifyWatchdog); err != nil {
+ logger.Error(err, "WATCHDOG=1 failed")
+ } else {
+ logger.Info("[systemd-go-app] sent WATCHDOG=1 to systemd")
+ }
+ }
+ }
+ }()
+ }
+
select {
case <-termCh:
sent, err := daemon.SdNotify(false, daemon.SdNotifyStopping)
if err != nil {
logger.Error(err, "STOPPING=1 failed")
} else if sent {
logger.Info("[systemd-go-app] sent STOPPING=1 to systemd")
}
close(reloadCh)
break restartLoop
case <-sigHUPCh:
monotonicUSec := uint64(max(time.Since(monotonicEpoch), 0) / time.Microsecond)
msg := fmt.Sprintf(
"%s\nMONOTONIC_USEC=%d",
daemon.SdNotifyReloading,
monotonicUSec,
)
sent, err := daemon.SdNotify(false, msg)
if err != nil {
logger.Error(err, "sdnotify RELOADING=1 failed")
} else if sent {
logger.Info("[systemd-go-app] SIGHUP received, sent RELOADING=1 to systemd")
}
close(reloadCh)
continue restartLoop
}
}
}With this in place, a corresponding unit file entry such as WatchdogSec=30s is all systemd needs to enable the mechanism - the process will get pinged every 15 seconds, and systemd will restart it if 30 seconds pass without a ping. Pairing this with Restart=on-watchdog or Restart=always gives you automatic recovery from soft hangs that a plain process-crash policy would never catch.
I hope this walkthrough gave you a concrete feel for what a production-ready notify-reload service looks like in Go. We started from a plain hello-world loop and layered on the full systemd lifecycle piece by piece: guarding on NOTIFY_SOCKET, announcing READY=1, restructuring around a labeled restartLoop with a fresh reloadCh per iteration, handling SIGTERM/SIGINT with a clean STOPPING=1, doing coordinated SIGHUP reloads with the mandatory MONOTONIC_USEC stamp, and finally wiring in the WATCHDOG=1 heartbeat scoped to the same reload iteration so nothing leaks across reloads. The complete, runnable version of the code from this article, together with a matching notify-reload unit file, is available at github.com/iypetrov/go-playground/tree/master/systemd-go-app.