====== RDU3 Jenkins ======

===== Overview =====

The RDU3 Jenkins deployment is the new Jenkins environment running in the RDU3 OpenShift cluster.

Jenkins URL:

https://jenkins.apps.prod-scale-spoke1-dc-rdu3.itup.redhat.com

OpenShift cluster:

prod-scale-spoke1-dc-rdu3.itup.redhat.com

OpenShift Console:

https://console-openshift-console.apps.prod-scale-spoke1-dc-rdu3.itup.redhat.com

Production OpenShift project:

ceph-downstream-tenant--pipeline


===== Architecture =====

The Jenkins environment consists of:

  * Jenkins controller running in OpenShift
  * OpenShift Virtualization (KubeVirt/CNV) VMs used as Jenkins build agents
  * RHEL 9 builder VMs
  * On-demand Jenkins agent provisioning
  * Jenkins agents connect back to the controller using WebSocket

Unlike a traditional Kubernetes Jenkins deployment, builds are not executed inside Kubernetes pods. Dedicated KubeVirt VMs are created on demand and used as Jenkins agents.

This provides Jenkins jobs with a full RHEL virtual machine while still allowing the builders to be dynamically created and removed as required.


===== Jenkins Configuration Repository =====

The Jenkins configuration is maintained in the following Git repository:

<code>
git@gitlab.cee.redhat.com:ccit/jenkins-csb-customers/storage-casc-ceph-downstream.git
</code>

During the RDU3 migration, the new Jenkins instance is using the following branch:

<code>
rdu3-migration
</code>

This is a temporary migration branch used for the RDU3 Jenkins configuration while the existing Jenkins environment is still in use.

Changes specific to the new RDU3 Jenkins deployment should currently be made against the ''rdu3-migration'' branch.

Once the migration to the RDU3 Jenkins instance is complete, the temporary migration branch can be retired and the normal production configuration workflow can be used.


===== Jenkins Agents =====

The Jenkins build agents are RHEL 9 virtual machines running through OpenShift Virtualization.

Example agent types include:

  * rhel9medium
  * rhel9large

These builders are not expected to be permanently running Jenkins nodes. They are provisioned on demand when Jenkins needs capacity to execute a job.

Each VM is configured as a Jenkins agent and connects back to:

https://jenkins.apps.prod-scale-spoke1-dc-rdu3.itup.redhat.com

The Jenkins agent uses WebSocket connectivity rather than the traditional JNLP TCP agent port.


===== On-Demand Builder Lifecycle =====

The RDU3 Jenkins builders are dynamically created using OpenShift Virtualization.

Instead of maintaining a large pool of permanently running builder VMs, Jenkins creates builders when jobs require them and removes the builders when they are no longer needed.

The lifecycle of a builder is:

  - A Jenkins job is submitted and requests a particular builder type/label.
  - Jenkins checks whether an appropriate agent is already available.
  - If no suitable agent is available, Jenkins requests a new builder.
  - A new KubeVirt VirtualMachine is created in the OpenShift tenant.
  - OpenShift Virtualization starts the VM and creates the corresponding VirtualMachineInstance (VMI).
  - The RHEL 9 VM boots.
  - The VM is provisioned with the required Jenkins agent configuration.
  - The Jenkins agent connects back to the Jenkins controller using WebSocket.
  - Jenkins detects the new agent and marks it online.
  - The queued Jenkins job is assigned to the new builder.
  - The job runs inside the VM.
  - After the builder is no longer required, Jenkins removes the agent.
  - The associated KubeVirt VM is destroyed, releasing the CPU and memory resources back to the OpenShift tenant.

This makes the builders effectively disposable.

A builder exists only for the period in which Jenkins requires the capacity. A later job can cause a new clean VM to be created instead of depending on the state left behind by an older build.


==== Builder Creation ====

When Jenkins determines that additional capacity is required, it requests a new builder based on the configuration for the requested node type.

For example, a job requiring a ''rhel9large'' builder causes Jenkins to request a VM matching the rhel9large configuration.

OpenShift Virtualization then creates the VM and allocates the configured resources, including:

  * CPU
  * Memory
  * Storage
  * Network interfaces

The VM boots from the configured RHEL 9 image.

As part of provisioning, the VM is configured with the Jenkins agent information required to connect back to the controller.


==== Jenkins Agent Registration ====

Inside the VM, the Jenkins agent is started using ''agent.jar''.

An example of the agent command is:

<code>
java -jar /home/jenkins-build/agent.jar \
  -url https://jenkins.apps.prod-scale-spoke1-dc-rdu3.itup.redhat.com \
  -secret @/etc/jenkins-agent/secret \
  -name <node-name> \
  -webSocket \
  -workDir /home/jenkins-build
</code>

The important part of the RDU3 configuration is:

<code>
-webSocket
</code>

The agent establishes an outbound WebSocket connection to the Jenkins controller over HTTPS.

Once the connection is established, Jenkins marks the node online and can schedule the waiting job onto it.


==== Running the Build ====

Once the builder is online, Jenkins assigns the queued job to the VM.

From the Jenkins job's perspective, the VM behaves like a regular Jenkins node.

The build has access to the RHEL environment and any tools installed as part of the builder provisioning process.


==== Builder Destruction ====

The builders are disposable and are not intended to remain running indefinitely after their work has completed.

After Jenkins determines that the dynamically created agent is no longer required, the agent is removed and the associated KubeVirt VM is deleted.

Deleting the VM also releases the compute resources that were allocated to it.

This is particularly important in RDU3 because the OpenShift tenants have resource limits. Dynamically removing unused builders prevents idle Jenkins nodes from continuously consuming the tenant's CPU and memory allocation.


===== Jenkins / KubeVirt Integration =====

The Jenkins integration uses customized logic for managing the KubeVirt builder nodes.

The Sorter Jenkins plugin contains Java customizations used as part of the KubeVirt Jenkins agent implementation.

The integration allows Jenkins to dynamically request KubeVirt builder VMs when jobs require additional capacity and remove those VMs when they are no longer needed.


==== KubeVirt Builder Flow ====

The following diagram shows the lifecycle of an on-demand Jenkins builder:

<code>
                  +-----------------------+
                  |      Jenkins Job      |
                  |   enters the queue    |
                  +-----------+-----------+
                              |
                              v
                  +-----------------------+
                  |  Jenkins Controller   |
                  |                       |
                  | Builder capacity      |
                  | required              |
                  +-----------+-----------+
                              |
                              v
                  +-----------------------+
                  |   Sorter / KubeVirt   |
                  |      Integration      |
                  +-----------+-----------+
                              |
                              | Create VM
                              v
        +------------------------------------------------+
        |              OpenShift RDU3                    |
        |                                                |
        |  ceph-downstream-tenant--pipeline              |
        |                                                |
        |       +-----------------------------+          |
        |       | OpenShift Virtualization    |          |
        |       |        / KubeVirt           |          |
        |       +-------------+---------------+          |
        |                     |                          |
        |                     | Start VM                 |
        |                     v                          |
        |       +-----------------------------+          |
        |       |     RHEL 9 Builder VM       |          |
        |       |                             |          |
        |       | rhel9medium / rhel9large    |          |
        |       +-------------+---------------+          |
        |                     |                          |
        +---------------------|--------------------------+
                              |
                              | WebSocket / HTTPS
                              v
                  +-----------------------+
                  |  Jenkins Controller   |
                  |                       |
                  | Agent becomes online  |
                  +-----------+-----------+
                              |
                              v
                  +-----------------------+
                  |       Job Runs        |
                  |   on the Builder VM   |
                  +-----------+-----------+
                              |
                              v
                  +-----------------------+
                  |     Job Completes     |
                  +-----------+-----------+
                              |
                              v
                  +-----------------------+
                  | Jenkins removes the   |
                  | dynamic agent         |
                  +-----------+-----------+
                              |
                              | Delete VM
                              v
                  +-----------------------+
                  | KubeVirt VM Destroyed |
                  |                       |
                  | CPU / Memory released |
                  +-----------------------+
</code>

The important point is that the builder VM is temporary:

<code>
Job queued
    |
    v
VM created
    |
    v
VM boots
    |
    v
Agent connects to Jenkins
    |
    v
Job runs
    |
    v
Job finishes
    |
    v
Agent removed
    |
    v
VM destroyed
</code>


===== Networking =====

Outbound traffic from the RDU3 tenant is restricted.

When Jenkins or its builder VMs require access to a new external destination, that destination must be allowed by the tenant egress configuration.

The tenant uses egress rules to control which external networks and services workloads are allowed to reach.


==== Adding a New Outbound Destination ====

For example, if Jenkins builders need to reach a service at:

<code>
192.0.2.50
</code>

on TCP port 22, an EgressFirewall rule can be added:

<code yaml>
apiVersion: network.openshift.io/v1
kind: EgressFirewall
metadata:
  name: default
  namespace: ceph-downstream-tenant--pipeline
spec:
  egress:
    - type: Allow
      to:
        cidrSelector: 192.0.2.50/32
      ports:
        - protocol: TCP
          port: 22
</code>

For an HTTPS service:

<code yaml>
    - type: Allow
      to:
        cidrSelector: 192.0.2.50/32
      ports:
        - protocol: TCP
          port: 443
</code>

The destination should be restricted as much as possible. Prefer allowing the specific destination IP/network and required port instead of opening broad outbound access.

Before adding a new external dependency, identify:

  * Destination hostname/IP
  * Destination network/CIDR
  * Protocol
  * Required port

The appropriate egress rule can then be added to the tenant configuration.

After the rule is applied, connectivity can be verified from a Jenkins builder VM.


===== OpenShift Access =====

The RDU3 environment provides four tenants for our use.

Each tenant has limited CPU, memory, and other resources, so workloads should be placed in the appropriate tenant.

The two primary tenants used by Jenkins are:

  * ''ceph-downstream-tenant--pipeline'' - Production Jenkins workloads.
  * ''ceph-downstream-tenant--runtime-int'' - Staging and testing environment.

The ''ceph-downstream-tenant--pipeline'' tenant hosts the production Jenkins environment and production builder workloads.

The ''ceph-downstream-tenant--runtime-int'' tenant should be used for testing changes before they are introduced into production.

Examples include:

  * Testing new Jenkins plugins
  * Testing plugin upgrades
  * Testing Jenkins configuration changes
  * Testing builder changes
  * Testing Jenkins controller changes
  * Validating other infrastructure changes

Changes should first be validated in ''ceph-downstream-tenant--runtime-int'' before being applied to ''ceph-downstream-tenant--pipeline''.


===== Migration =====

The RDU3 Jenkins environment is intended to become the new home for the downstream Jenkins controller and builder workloads.

During the migration period, the RDU3-specific Jenkins configuration is maintained in the:

<code>
rdu3-migration
</code>

branch of:

<code>
git@gitlab.cee.redhat.com:ccit/jenkins-csb-customers/storage-casc-ceph-downstream.git
</code>

This allows the RDU3 environment to be configured and validated without affecting the existing production Jenkins deployment.

Once the migration is complete and the RDU3 Jenkins instance becomes the primary production environment, the temporary migration configuration can be consolidated into the normal production configuration.


===== Related Components =====

  * OpenShift
  * OpenShift Virtualization / KubeVirt
  * RHEL 9 builder VMs
  * Jenkins Sorter plugin
  * RDU3 TenantEgress / EgressFirewall