Pipeline Deployment & Container Lifecycle with Background Polling: Building Debezium's Host Path

I'm contributing to the Debezium Platform as part of Google Summer of Code 2026. In my previous post, I covered the host provisioning engine — how the system discovers SSH hosts and installs Docker on them. This post covers what happens next: actually deploying pipelines to those hosts and keeping them alive.

Where We Left Off

From my previous POST, the platform could watch ~/.ssh/config, provision discovered hosts via Ansible, and track host status in the database. But a provisioned host sitting idle is useless. We needed the platform to:

Pick the best host for a new pipeline
Assign a unique port
Deploy a Debezium Server container
Monitor the container's health continuously
Detect if someone tampered with the config on disk

The Container Runtime Abstraction

The first design decision: how do we manage containers on remote hosts? I could have wired Ansible commands directly into the pipeline controller. My Mentor pushed back on that during review. The right approach was an interface:

public interface HostContainerRuntime {
    void deploy(HostAllocation allocation, String containerName,
                String configContent, String image);
    void undeploy(String host, String containerName);
    void stop(String host, String containerName);
    void start(String host, String containerName);
    String logs(String host, String containerName);
}

The initial implementation, AnsibleContainerRuntime, translates each method into an Ansible ad-hoc command. For example, deploy() runs these steps over SSH:

mkdir -p /opt/debezium/configs/<name>/
mkdir -p /opt/debezium/data/<name>/
Write application.properties to the config directory
docker rm -f <name> (remove any stale container)
docker run -d --name <name> -p <port>:8080 -v config -v data <image>

Why did this interface matter? Because in future impls where I have to create a lightweight host Agent, I replaced the entire implementation with an AgentContainerRuntime that calls a REST API instead of running Ansible commands. The controller code didn't change at all. CDI just injected a different bean.

Enterprise Clean Architecture & Code Review Refactorings

During code review on PR #493, my mentor shared valuable feedback on enterprise architecture design patterns. Refactoring the code based on these comments transformed the implementation into production-grade Clean Architecture:

1. JPA Entity Hiding & Boundary Decoupling

In early iterations, HostDeploymentService returned JPA @Entity instances (HostDeploymentEntity). Passing ORM entities across service boundaries introduces major enterprise anti-patterns:

Hibernate Dirty Checking: Unintended SQL UPDATE queries can trigger outside transaction boundaries.
LazyInitializationException: Accessing lazy-loaded relations outside active EntityManager sessions causes runtime failures.
Tight Coupling: Controllers and background pollers become tightly bound to database column definitions.

Solution: We created lightweight, immutable Domain Records (Deployment and HostStatusReference). Service query methods map entities at the boundary using Deployment.from(entity) or Blaze-Persistence view models. Controllers, strategy selectors, and pollers never touch JPA entities directly:

// Boundary mapping: JPA entities never leak past the service layer
public Optional<Deployment> findByPipelineId(Long pipelineId) {
    return findEntityByPipelineId(pipelineId).map(Deployment::from);
}

Concurrency-Safe Host Selection

When a pipeline is created, the system needs to answer two questions: which host should it run on, and which port should it use?

This is harder than it sounds. If two pipelines deploy at the same time, they could both pick the same host and the same port. The fix is pessimistic database locking:

@Transactional(REQUIRES_NEW)
public HostAllocation allocateHostAndPort() {
    List<HostStatusEntity> readyEntities = lockAllReadyHosts();
    // ... strategy selects host, MAX(serverPort)+1 allocates port
}

Three things make this safe:

Lock ordering: All READY hosts are locked sorted by id ASC. If two transactions try to lock the same set of rows, they'll acquire them in the same order. No ABBA deadlock.

Live port query: Port allocation uses MAX(serverPort) + 1 against the host_deployment table, not a cached counter. Under a pessimistic lock, this value is always current.

Short lock window: @Transactional(REQUIRES_NEW) means the locks are released as soon as host selection finishes — before the actual Docker deploy begins. We don't hold database locks for 2 minutes while Docker pulls images.

The Deploy Strategy

Host selection is pluggable via the DeployStrategy interface:

public interface DeployStrategy {
    HostStatusReference select(List<HostStatusReference> readyHosts);
}

The first implementation, LeastLoadedDeployStrategy, picks the host with the fewest active containers. The live COUNT(*) query runs inside the pessimistic lock, so the count is always accurate.

Background Status Polling

Once a container is deployed, we can't assume it stays running. Docker processes crash. SSH connections drop. Someone might manually edit application.properties on the remote host. The HostDeploymentStatusPoller watches for all of this:

@Scheduled(every = "${platform.host.status-poll-interval:30s}",
           identity = "host-deployment-status-poller")
void pollDeploymentStatus() {
    if (!isHostMode()) return; // skip in Kubernetes mode

    List<HostDeployment> active = deploymentService.findByStatuses(
            DeploymentStatus.DEPLOYING, DeploymentStatus.RUNNING);
    active.forEach(this::checkDeployment);
}

The state machine handles five transitions:

Current State	Condition	New State
`DEPLOYING`	Container running	`RUNNING`
`DEPLOYING`	Not running, grace period elapsed	`FAILED`
`DEPLOYING`	Not running, within grace period	(skip)
`RUNNING`	Container stopped/removed	`FAILED`
`RUNNING`	Config hash mismatch	`CONFIG_DRIFT`

The Grace Period

When a container is first deployed, Docker might need to pull the image (~500MB). On a slow connection, this can take several minutes. If the poller checked immediately and saw "not running," it would incorrectly mark the deployment as failed.

The solution: a 5-minute grace period from the deployedAt timestamp. The poller skips the "not running" state check until the grace period has elapsed.

Config Drift Detection

This was one of the more interesting pieces. The Conductor stores a SHA-256 hash of the application.properties file when it deploys a container. On every poll cycle, the poller checks the remote file's hash:

expected = deployment.getConfigHash()     // stored at deploy time
actual   = SHA-256(remote application.properties)
if (expected != actual) → mark CONFIG_DRIFT

This catches cases where someone SSH'd into the host and edited the config directly. The platform flags it so the operator can decide whether to redeploy with the correct config or accept the change.

Retry Logic

The Agent REST call (or the original Ansible SSH command) can fail due to transient network issues. But a single TCP timeout doesn't mean the container is down. The poller wraps each status check in RetryingRunnable:

RetryingRunnable.<RuntimeException> builder()
    .retries(hostConfig.statusPollMaxRetries())      // default 3
    .doRun(() -> result[0] = checkContainerStatus())
    .delayStrategy(DelayStrategy.exponential(
        Duration.ofSeconds(1), Duration.ofSeconds(8)))
    .retriableExceptions(DebeziumException.class)
    .build()
    .run();

Only DebeziumException (connectivity failures) is retried. A definitive answer — like Docker reporting running=false — is trusted immediately. No retry.

Deployment Mode Guard

The @Scheduled annotation doesn't support conditional activation. It fires regardless of whether you're in host or operator mode. I added a simple guard:

private boolean isHostMode() {
    return "host".equals(deploymentMode);
}

The poller checks this on every tick and returns immediately if we're running in Kubernetes mode. Wasteful? Slightly. But it avoids the complexity of programmatic scheduler registration, and a no-op check every 30 seconds has zero measurable impact.

What I Learned

Interfaces are worth the upfront cost. When my Mentor suggested abstracting the container runtime, it felt like over-engineering for a single implementation. Two weeks later, when I built the Agent-based implementation, the controller didn't need a single line changed. The interface paid for itself immediately.

Database locks have a blast radius. My first version held the pessimistic lock during the entire deploy operation — including the Ansible call. That meant other pipelines waited 2+ minutes to deploy. Moving to REQUIRES_NEW fixed this: select quickly, release, then deploy.

Polling is surprisingly tricky. Getting the state machine right took several iterations. The grace period logic, the retry behavior, the deployment mode guard — each one seems simple in isolation, but the interactions between them required careful thought.

Never leak database entities across service boundaries. Mapping JPA entities to immutable domain records (Deployment.from(entity)) at service boundaries prevents lazy-loading exceptions, eliminates dirty-checking side effects, and keeps presentation logic decoupled from DB schema changes.

Design patterns elevate code quality. Applying the GoF Command Pattern to ad-hoc Ansible commands turned procedural scripts into clean, self-documenting, unit-testable Java code with sealed result interfaces.

PR: https://github.com/debezium/debezium-platform/pull/493

Below is a full end-to-end walkthrough of the host-based pipeline deployment.