Install OpenShift Logging Operator with LokiStack

Elasticsearch and Kibana are gone from OpenShift logging. The current stack is three operators: the Red Hat OpenShift Logging Operator (the old “Cluster Logging Operator”, package name cluster-logging) runs Vector on every node, the Loki Operator runs a LokiStack that keeps logs in object storage, and the Cluster Observability Operator puts a Logs page back in the web console.

Original content from computingforgeeks.com - post 74727

This guide installs all three, creates a LokiStack on S3 storage, writes a ClusterLogForwarder that sends application, infrastructure and audit logs to it, queries the result from the CLI and the console, forwards one namespace to an external syslog receiver, and sets per-tenant retention. Along the way it covers the four failures that bite almost everyone on a first install, with the exact error text. Every command below was run on OpenShift Local (OpenShift 4.21.14) with Logging 6.6.1, Loki Operator 6.6.1 and Cluster Observability Operator 1.5.3 in October 2026.

How the pieces fit together

Vector tails pod logs, the node journal and the API audit log, then pushes each log type to its own tenant on the LokiStack gateway. The gateway checks the service account token, the distributor applies rate limits, and the ingester flushes compressed chunks to the bucket. Queries from the console or the CLI take the same gateway in the other direction.

Two kinds of storage are involved, and both are mandatory. A StorageClass gives the ingester, compactor and index gateway their block PVCs (write-ahead log and index cache). The S3 bucket holds the actual log data. Miss either one and the operators still install cleanly, which is what makes the failure confusing.

Prerequisites and sizing

• OpenShift 4.19 or newer. Logging 6.6.1 supports 4.19 and later (the 6.6.0 release needed 4.20).
• Cluster admin access and the oc CLI logged in as a cluster-admin user.
• A StorageClass with dynamic provisioning (ReadWriteOnce).
• An S3-compatible bucket with credentials that can list, get, put and delete objects plus get and put object tags. Without delete, retention never frees space.

Sizing is driven by two numbers: how many GB of logs the cluster produces per day, and how many days you keep them. Daily volume sets the LokiStack size, which in turn sets CPU and memory requests. Retention sets the bucket size. The Red Hat rule of thumb is daily volume divided by 6 (Snappy compression) times retention days, so 100 GB/day kept for 30 days needs about 500 GB of object storage.

LokiStack size	Ingest per day	CPU requests	Memory requests	PVC requests
1x.demo	Demo only	None	None	40Gi
1x.pico	50 GB	7 vCPU	17Gi	590Gi
1x.extra-small	100 GB	14 vCPU	31Gi	590Gi
1x.small	500 GB	34 vCPU	67Gi	590Gi
1x.medium	2 TB	54 vCPU	139Gi	590Gi

Those figures come from the Red Hat OpenShift Logging planning guide. Red Hat suggests 1x.small as the starting point for production clusters, 1x.extra-small for dev and test, and 1x.pico for single-node OpenShift. Our lab ran 1x.demo on a single-node OpenShift Local (CRC) cluster with 7 vCPU and 20 GiB of RAM. The whole logging stack used about 1.4 GiB of memory there (cgroup reading, page cache included), with the Vector collector at 425 MiB and the ingester at 412 MiB. Treat that as a floor for following along, not a production recommendation.

Confirm the cluster has a StorageClass before you start:

oc get storageclass

Note the name. The LokiStack manifest later needs it:

NAME                                     PROVISIONER                        RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
crc-csi-hostpath-provisioner (default)   kubevirt.io.hostpath-provisioner   Retain          WaitForFirstConsumer   false                  146d

No StorageClass at all is common on single-node and bare-metal clusters. Create one with the Local Storage Operator or LVM Storage before going further.

Install the Loki Operator

Install the log store first. The Logging Operator and the Loki Operator must track the same minor version, so both subscriptions below use the stable-6.6 channel.

There is a trap in the catalog. Two packages are named loki-operator, one from Red Hat and one from Community Operators:

oc get packagemanifests -n openshift-marketplace | grep -E "^loki-operator"

On our cluster, oc get packagemanifest loki-operator without a label selector returned the community one, with a single alpha channel. Both entries are listed here:

loki-operator                                           Red Hat Operators     147d
loki-operator                                           Community Operators   147d

Filter on the Red Hat catalog to see the supported channels:

oc get packagemanifests -n openshift-marketplace -l catalog=redhat-operators \
  -o jsonpath='{range .items[?(@.metadata.name=="loki-operator")]}{.status.defaultChannel}{" "}{.status.channels[*].name}{"\n"}{end}'

The default channel prints first, followed by every channel the catalog carries:

stable-6.6 stable-6.4 stable-6.5 stable-6.6

Create the manifest for the namespace, OperatorGroup and Subscription. The source: redhat-operators line is what keeps you off the community build:

vim loki-operator.yaml

Paste the following:

apiVersion: v1
kind: Namespace
metadata:
  name: openshift-operators-redhat
  labels:
    openshift.io/cluster-monitoring: "true"
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: loki-operator
  namespace: openshift-operators-redhat
spec:
  upgradeStrategy: Default
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: loki-operator
  namespace: openshift-operators-redhat
spec:
  channel: stable-6.6
  installPlanApproval: Automatic
  name: loki-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace

Apply it:

oc apply -f loki-operator.yaml

The CSV reached Succeeded in about a minute on our cluster. Check it:

oc get csv -n openshift-operators-redhat | grep -E "NAME|loki"

The Red Hat build shows the Loki Operator display name and a Succeeded phase:

NAME                                    DISPLAY                          VERSION   REPLACES                                PHASE
loki-operator.v6.6.1                    Loki Operator                    6.6.1     loki-operator.v6.6.0                    Succeeded

If the DISPLAY column says something other than Loki Operator, or the channel list was just alpha, the community package got installed. Delete the subscription and CSV, then reapply with source: redhat-operators.

Install the Red Hat OpenShift Logging Operator

The Logging Operator lives in openshift-logging, the same namespace that will hold the LokiStack and the S3 secret. Installing it now creates that namespace in one step:

vim logging-operator.yaml

Add the namespace, OperatorGroup and Subscription:

apiVersion: v1
kind: Namespace
metadata:
  name: openshift-logging
  labels:
    openshift.io/cluster-monitoring: "true"
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: cluster-logging
  namespace: openshift-logging
spec:
  upgradeStrategy: Default
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cluster-logging
  namespace: openshift-logging
spec:
  channel: stable-6.6
  installPlanApproval: Automatic
  name: cluster-logging
  source: redhat-operators
  sourceNamespace: openshift-marketplace

Apply it and wait for the CSV:

oc apply -f logging-operator.yaml
oc get csv -n openshift-logging | grep -E "NAME|cluster-logging"

It sits in Pending for a minute or two before it settles on Succeeded:

NAME                                    DISPLAY                          VERSION   REPLACES                                PHASE
cluster-logging.v6.6.1                  Red Hat OpenShift Logging        6.6.1     cluster-logging.v6.6.0                  Succeeded

Installing the operator deploys nothing that collects logs. Collection starts only when a ClusterLogForwarder exists.

Create the S3 secret for Loki

Loki reads its bucket details from a secret in openshift-logging. The key names (bucketnames, endpoint, access_key_id, access_key_secret, region) are specific to the Loki Operator, so do not rename them to the AWS SDK style. Set your bucket values first. The defaults below match the throwaway lab store described in the next section:

export S3_BUCKET="loki"
export S3_ENDPOINT="http://seaweedfs.loki-s3.svc:8333"
export S3_ACCESS_KEY="lokiadmin"
export S3_SECRET_KEY="LokiStrongPass123"
export S3_REGION="us-east-1"

For AWS, the endpoint is https://s3.<region>.amazonaws.com and the bucket must already exist. The Red Hat install guide also shows --from-literal=forcepathstyle="true" for S3-compatible stores that only serve path-style URLs. Our SeaweedFS endpoint worked without it. Create the secret:

oc create secret generic logging-loki-s3 -n openshift-logging \
  --from-literal=bucketnames="${S3_BUCKET}" \
  --from-literal=endpoint="${S3_ENDPOINT}" \
  --from-literal=access_key_id="${S3_ACCESS_KEY}" \
  --from-literal=access_key_secret="${S3_SECRET_KEY}" \
  --from-literal=region="${S3_REGION}"

If your endpoint uses a self-signed or internal CA (Ceph RGW, ODF, on-prem S3 appliances), the LokiStack also needs the CA in a ConfigMap referenced under spec.storage.tls.caName. Without it the LokiStack reports Ready while ingesters log x509: certificate signed by unknown authority and drop data.

No bucket yet? A test store inside the cluster

Readers with a real S3 bucket can skip this. For a lab, we ran the SeaweedFS S3 gateway inside the cluster. MinIO was the usual choice for this, but in October 2026 both quay.io/minio/minio and docker.io/minio/minio refused pulls on our cluster (unauthorized: access to the requested resource is not authorized and requested access to the resource is denied). This store has one replica on one PVC. It is a test fixture, not a backup target.

Create the S3 identity file first:

vim s3.json

One identity with the actions Loki needs, using the same keys as the secret above. If you changed S3_ACCESS_KEY or S3_SECRET_KEY, change them here too:

{"identities":[{"name":"loki","credentials":[{"accessKey":"lokiadmin","secretKey":"LokiStrongPass123"}],"actions":["Admin","Read","List","Tagging","Write"]}]}

Now the namespace, PVC, Deployment and Service:

vim seaweedfs-s3.yaml

The enableServiceLinks: false line matters. Our first attempt named the Service s3, Kubernetes injected S3_PORT=tcp://10.217.4.184:8333 into the pod, and SeaweedFS crashed with invalid value "tcp://10.217.4.184:8333" for environment variable s3.port. The image is pinned to the SeaweedFS release we tested (newer tags are listed on the SeaweedFS releases page):

apiVersion: v1
kind: Namespace
metadata:
  name: loki-s3
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: s3-data
  namespace: loki-s3
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: seaweedfs
  namespace: loki-s3
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: seaweedfs
  template:
    metadata:
      labels:
        app: seaweedfs
    spec:
      enableServiceLinks: false
      containers:
        - name: seaweedfs
          image: docker.io/chrislusf/seaweedfs:4.48
          args:
            - server
            - -dir=/data
            - -s3
            - -s3.config=/etc/seaweedfs/s3.json
          ports:
            - containerPort: 8333
          volumeMounts:
            - name: data
              mountPath: /data
            - name: s3-config
              mountPath: /etc/seaweedfs
              readOnly: true
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: s3-data
        - name: s3-config
          secret:
            secretName: seaweedfs-s3-config
---
apiVersion: v1
kind: Service
metadata:
  name: seaweedfs
  namespace: loki-s3
spec:
  selector:
    app: seaweedfs
  ports:
    - name: s3
      port: 8333
      targetPort: 8333

Create the namespace and identity secret, deploy, then create the loki bucket with weed shell:

oc create namespace loki-s3
oc create secret generic seaweedfs-s3-config -n loki-s3 --from-file=s3.json=s3.json
oc apply -f seaweedfs-s3.yaml
oc rollout status deploy/seaweedfs -n loki-s3 --timeout=240s
oc exec -n loki-s3 deploy/seaweedfs -- sh -c "echo s3.bucket.create -name loki | weed shell"

The last command confirms the bucket:

create bucket under /buckets
created bucket loki

Our S3_ENDPOINT default already points at this Service, so the secret created earlier works as is.

Create the LokiStack

The LokiStack resource turns into the gateway, distributor, ingester, querier, query frontend, index gateway and compactor. Open the manifest:

vim lokistack.yaml

This is the exact file from our lab. Change storageClassName to the name from oc get storageclass, and size to 1x.extra-small or larger outside a lab:

apiVersion: loki.grafana.com/v1
kind: LokiStack
metadata:
  name: logging-loki
  namespace: openshift-logging
spec:
  size: 1x.demo
  storage:
    schemas:
      - version: v13
        effectiveDate: "2024-10-01"
    secret:
      name: logging-loki-s3
      type: s3
  storageClassName: crc-csi-hostpath-provisioner
  tenants:
    mode: openshift-logging
  limits:
    global:
      ingestion:
        ingestionRate: 8
        ingestionBurstSize: 16
      retention:
        days: 7

The ingestion block raises the per-tenant limit from the operator default of 4 MB/s (burst 6 MB) to 8 MB/s (burst 16 MB). We added it after the collector hit 429 errors, covered further down. Apply the LokiStack:

oc apply -f lokistack.yaml

If oc apply returned an error instead of lokistack.loki.grafana.com/logging-loki created, check the API group first.

Error: no matches for kind “LokiStack” in version “loki.storage.openshift.io/v1”

Use apiVersion: loki.grafana.com/v1. Some snippets in the current Red Hat install guide show loki.storage.openshift.io/v1, and that API group does not exist on the cluster. Applying one of those snippets fails with:

error: resource mapping not found for name: "logging-loki" namespace: "openshift-logging" from "lokistack.yaml": no matches for kind "LokiStack" in version "loki.storage.openshift.io/v1"
ensure CRDs are installed first

oc api-resources | grep -i lokistack shows the real group. Once the correct manifest is applied, the pods come up in about 90 seconds:

oc get pods -n openshift-logging -l app.kubernetes.io/instance=logging-loki

On 1x.demo every component runs as a single replica except the gateway:

NAME                                           READY   STATUS    RESTARTS   AGE
logging-loki-compactor-0                       1/1     Running   0          7m34s
logging-loki-distributor-7fbdcb9d5d-5rbn2      1/1     Running   0          8m6s
logging-loki-gateway-57d85955f5-dhbz4          2/2     Running   0          31m
logging-loki-gateway-57d85955f5-t2qcr          2/2     Running   0          31m
logging-loki-index-gateway-0                   1/1     Running   0          7m34s
logging-loki-ingester-0                        1/1     Running   0          7m35s
logging-loki-querier-7f6d6c9848-jdr6t          1/1     Running   0          8m6s
logging-loki-query-frontend-5cc4dbd997-9qxtt   1/1     Running   0          8m5s

Check the conditions too. If PVCs are stuck in Pending, the StorageClass name is wrong:

oc get lokistack logging-loki -n openshift-logging \
  -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{"\n"}{end}'

A healthy stack reports Ready with no failed or pending components:

Ready=True
Failed=False
Pending=False

Ready=True here means the components are running. It does not prove that logs are arriving, which the next sections cover.

Grant the collector its roles

Vector runs under a service account you create, and that account needs one cluster role to write to Loki plus one per log type it reads. The Logging Operator creates the roles. You bind them. The concepts are the same as any Kubernetes service account and ClusterRole binding:

oc create sa logging-collector -n openshift-logging
oc adm policy add-cluster-role-to-user logging-collector-logs-writer -z logging-collector -n openshift-logging
oc adm policy add-cluster-role-to-user collect-application-logs -z logging-collector -n openshift-logging
oc adm policy add-cluster-role-to-user collect-infrastructure-logs -z logging-collector -n openshift-logging
oc adm policy add-cluster-role-to-user collect-audit-logs -z logging-collector -n openshift-logging

Each binding prints a confirmation line such as clusterrole.rbac.authorization.k8s.io/collect-audit-logs added: "logging-collector". Skip the audit role only if you will never forward audit logs.

Error: ClusterRoleMissing, not authorized to collect [“audit”] logs

Bind the role before you add the input. We added audit to a running forwarder while the service account had only the application and infrastructure roles. The operator didn’t just skip audit. It deleted the whole collector DaemonSet, so application and infrastructure collection stopped too. The forwarder status showed:

observability.openshift.io/Authorized=False ClusterRoleMissing: insufficient permissions on service account, not authorized to collect ["audit"] logs
observability.openshift.io/Valid=False ValidationFailure: one or more conditions [inputs, outputs, pipelines, filters] have failed validation
Ready=False ValidationFailure: collector not ready

Binding collect-audit-logs brings the DaemonSet back within a minute. We re-tested this with the bind alone: the forwarder generation stayed the same and the collector pod came back. The reverse is also true. Removing any collect role that an input relies on deletes the DaemonSet straight away, so treat those bindings as part of the forwarder.

Create the ClusterLogForwarder

The ClusterLogForwarder (observability.openshift.io/v1) replaces both the ClusterLogging and the old logging.openshift.io ClusterLogForwarder from the 5.x releases. One output points at the LokiStack, and one pipeline sends all three log types to it:

vim clusterlogforwarder.yaml

Do not drop the tls block. The next section shows what happens without it:

apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  serviceAccount:
    name: logging-collector
  outputs:
    - name: lokistack-out
      type: lokiStack
      lokiStack:
        target:
          name: logging-loki
          namespace: openshift-logging
        authentication:
          token:
            from: serviceAccount
      tls:
        ca:
          key: service-ca.crt
          configMapName: logging-loki-gateway-ca-bundle
  pipelines:
    - name: all-logs-to-loki
      inputRefs:
        - application
        - infrastructure
        - audit
      outputRefs:
        - lokistack-out

Apply it:

oc apply -f clusterlogforwarder.yaml

The operator creates a DaemonSet named after the forwarder, one collector per node:

oc get daemonset instance -n openshift-logging

Our single-node cluster shows one pod. On a real cluster DESIRED equals the Linux node count:

NAME       DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
instance   1         1         1       1            1           kubernetes.io/os=linux   27m

A running DaemonSet is not proof of delivery either, as the next error shows.

Error: certificate verify failed: self-signed certificate in certificate chain

The gateway certificate is signed by the OpenShift service CA, which Vector does not trust by default. Our first forwarder had no tls block, and its status looked perfect: Authorized=True, Valid=True, Ready=True. Meanwhile nothing reached Loki. Only the collector logs tell the truth:

oc logs -n openshift-logging ds/instance --tail=200 | grep -i certif

Every push attempt failed like this:

WARN sink{component_kind="sink" component_id=output_lokistack_out_infrastructure component_type=loki}: vector::internal_events::http_client: HTTP error. error=error trying to connect: error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:2124:: self-signed certificate in certificate chain

The fix is the tls.ca block above, pointing at the logging-loki-gateway-ca-bundle ConfigMap the Loki Operator creates. Re-apply and the warnings stop within a minute.

Error: Server responded with an error: 429 Too Many Requests

Right after the audit input went live, Vector replayed the backlog and Loki pushed back:

WARN sink{component_kind="sink" component_id=output_lokistack_out_infrastructure component_type=loki}: vector::sinks::util::retries: Retrying after error. error=Server responded with an error: 429 Too Many Requests

Vector retries, so nothing was lost, but sustained 429s mean the tenant limit is below your real ingest rate. The rendered Loki config showed the default limits:

oc get cm logging-loki-config -n openshift-logging -o jsonpath='{.data.config\.yaml}' | grep -E "ingestion_rate_mb|ingestion_burst_size_mb"

Before the change it printed ingestion_rate_mb: 4 and ingestion_burst_size_mb: 6. With the spec.limits.global.ingestion block from the LokiStack manifest applied, it reads:

  ingestion_rate_mb: 8
  ingestion_burst_size_mb: 16

Changing limits restarts every Loki pod except the gateway, so do it in a quiet window. We saw zero 429s in the collector log after the restart finished.

Query the logs from the CLI

The LokiStack gateway has a Route, and it accepts your oc token. That gives you LogQL from any terminal without the console. The examples query a test app in a demo-app namespace that prints an info line every 2 seconds and an error line every 20. Create the same app if you want identical results, or swap demo-app for one of your own namespaces:

oc new-project demo-app
oc create deployment log-generator -n demo-app --image=registry.access.redhat.com/ubi9/ubi-minimal:latest -- /bin/sh -c 'i=0; while true; do i=$((i+1)); if [ $((i % 10)) -eq 0 ]; then echo "level=error msg=\"payment API timeout\" order_id=$i"; else echo "level=info msg=\"order processed\" order_id=$i"; fi; sleep 2; done'

The gateway authenticates with an OAuth token, so oc whoami -t only works after oc login as a user (kubeadmin or an identity provider account). A certificate-based kubeconfig such as the installer’s system:admin has no token, and the command fails with no token is currently in use for this session. Give the app a minute to log something, then search for the errors:

LOKI_HOST=$(oc get route logging-loki -n openshift-logging -o jsonpath='{.spec.host}')
TOKEN=$(oc whoami -t)
curl -sk -G -H "Authorization: Bearer ${TOKEN}" \
  "https://${LOKI_HOST}/api/logs/v1/application/loki/api/v1/query_range" \
  --data-urlencode 'query={kubernetes_namespace_name="demo-app"} |= "level=error" | json | line_format "{{.message}}"' \
  --data-urlencode 'limit=5' | python3 -c 'import sys,json; d=json.load(sys.stdin); [print(v[1]) for r in d["data"]["result"] for v in r["values"]]'

Only the error lines come back, with the Kubernetes metadata stripped by line_format:

level=error msg="payment API timeout" order_id=1140
level=error msg="payment API timeout" order_id=1150
level=error msg="payment API timeout" order_id=1160
level=error msg="payment API timeout" order_id=1170
level=error msg="payment API timeout" order_id=1180

The tenant is part of the path: application, infrastructure or audit. A metric query against the infrastructure tenant shows which namespaces are the noisiest, which is the first thing to check when the bucket grows faster than expected:

curl -sk -G -H "Authorization: Bearer ${TOKEN}" \
  "https://${LOKI_HOST}/api/logs/v1/infrastructure/loki/api/v1/query" \
  --data-urlencode 'query=sum by (kubernetes_namespace_name) (count_over_time({log_type="infrastructure"}[5m]))' \
  | python3 -c 'import sys,json; r=json.load(sys.stdin)["data"]["result"]; [print(x["metric"].get("kubernetes_namespace_name","(journal)"), x["value"][1]) for x in sorted(r, key=lambda x: -int(x["value"][1]))[:8]]'

The eight busiest sources over the last 5 minutes print in order. Node journal entries have no namespace label and print as (journal), and on our idle node they ranked third:

openshift-marketplace 478
openshift-network-operator 477
(journal) 440
openshift-cluster-version 431
openshift-operator-lifecycle-manager 290
openshift-kube-apiserver 173
openshift-dns 160
openshift-ovn-kubernetes 118

Stream labels available for selectors include kubernetes_namespace_name, kubernetes_pod_name, kubernetes_container_name, kubernetes_host and log_type. For one-off checks on a single pod or node, oc logs and oc adm node-logs are still faster than a LogQL round trip.

Add the Logs page to the OpenShift console

The Observe > Logs page in the console comes from a UIPlugin managed by the Cluster Observability Operator. Without it, logs are stored and queryable from the CLI but invisible in the console:

vim coo.yaml

Add the namespace, OperatorGroup and Subscription:

apiVersion: v1
kind: Namespace
metadata:
  name: openshift-cluster-observability-operator
  labels:
    openshift.io/cluster-monitoring: "true"
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: cluster-observability-operator
  namespace: openshift-cluster-observability-operator
spec: {}
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cluster-observability-operator
  namespace: openshift-cluster-observability-operator
spec:
  channel: stable
  installPlanApproval: Automatic
  name: cluster-observability-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace

Apply it, then wait for the UIPlugin CRD before creating a plugin. Applying the plugin too early fails with no matches for kind "UIPlugin":

oc apply -f coo.yaml
oc wait --for=condition=Established crd/uiplugins.observability.openshift.io --timeout=300s
oc get csv -n openshift-cluster-observability-operator | grep -E "NAME|cluster-observability"

The CRD is established and the operator CSV has succeeded:

customresourcedefinition.apiextensions.k8s.io/uiplugins.observability.openshift.io condition met
NAME                                    DISPLAY                          VERSION   REPLACES                                PHASE
cluster-observability-operator.v1.5.3   Cluster Observability Operator   1.5.3     cluster-observability-operator.v1.5.2   Succeeded

Now create the plugin manifest:

vim uiplugin.yaml

The plugin is cluster-scoped and only needs the LokiStack name:

apiVersion: observability.openshift.io/v1alpha1
kind: UIPlugin
metadata:
  name: logging
spec:
  type: Logging
  logging:
    lokiStack:
      name: logging-loki

Apply it and confirm the console picked it up:

oc apply -f uiplugin.yaml
oc get uiplugin logging -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{"\n"}{end}'
oc get consoles.operator.openshift.io cluster -o jsonpath='{.spec.plugins}{"\n"}'

The plugin is Available and logging-view-plugin is now in the console plugin list:

Reconciled=True
Degraded=False
Available=True
["networking-console-plugin","logging-view-plugin"]

If the console is managed by Argo CD or another GitOps tool, add logging-view-plugin to spec.plugins in Git as well, or the next sync removes it. All three operators now show as Succeeded under Ecosystem > Installed Operators:

Reload the console and open Observe > Logs. Pick the tenant from the drop-down (application, infrastructure or audit), filter by namespace, and toggle the histogram. This is the demo-app namespace, where the red slices are the error lines:

The Severity filter works on plain-text logs too. Vector parsed level=error out of our unstructured lines, so selecting error narrows the view to the failures without writing LogQL:

One quirk: typing "level=error" with quotes into the Content box made the console wrap it in a second pair of quotes, and the query matched nothing. Type the text without quotes. The Logs page is separate from the pod metrics views, which still live under Observe and the pod details pages.

Forward logs to an external syslog or SIEM

A forwarder can feed Loki and an external system at the same time. The syslog output below sends only the demo-app namespace to a receiver on TCP 5140, while everything still goes to Loki. Other output types (splunk, elasticsearch, kafka, cloudwatch, http, otlp, loki, googleCloudLogging, azureLogsIngestion, s3) follow the same output plus pipeline pattern. The older azureMonitor output is deprecated because Microsoft retired the HTTP Data Collector API it uses, so new Azure setups should use azureLogsIngestion. Splunk HEC specifics are in our OpenShift to Splunk guide.

If you have no syslog server to test against, run a throwaway TCP listener in the cluster first. It prints every received line to its pod log. Create the script:

vim recv.py

Paste this Python listener:

import socketserver
class H(socketserver.StreamRequestHandler):
    def handle(self):
        for line in self.rfile:
            print(line.decode(errors="replace").rstrip(), flush=True)
socketserver.ThreadingTCPServer.allow_reuse_address = True
socketserver.ThreadingTCPServer(("0.0.0.0", 5140), H).serve_forever()

Run it from a ConfigMap on the UBI Python image and expose it on TCP 5140:

oc new-project syslog-test
oc create configmap receiver -n syslog-test --from-file=recv.py=recv.py
oc create deployment syslog-receiver -n syslog-test --image=registry.access.redhat.com/ubi9/python-312:latest --port=5140 -- python3 /opt/recv/recv.py
oc set volume deploy/syslog-receiver -n syslog-test --add --type=configmap --configmap-name=receiver --mount-path=/opt/recv
oc expose deploy/syslog-receiver -n syslog-test --port=5140
oc rollout status deploy/syslog-receiver -n syslog-test --timeout=240s

With the receiver running, edit the forwarder again:

vim clusterlogforwarder.yaml

Replace the file with this version. Change the url to your receiver:

apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  serviceAccount:
    name: logging-collector
  outputs:
    - name: lokistack-out
      type: lokiStack
      lokiStack:
        target:
          name: logging-loki
          namespace: openshift-logging
        authentication:
          token:
            from: serviceAccount
      tls:
        ca:
          key: service-ca.crt
          configMapName: logging-loki-gateway-ca-bundle
    - name: siem-syslog
      type: syslog
      syslog:
        url: tcp://syslog-receiver.syslog-test.svc:5140
        rfc: RFC5424
        appName: '{.kubernetes.namespace_name||"none"}'
        payloadKey: '{.message}'
        enrichment: KubernetesMinimal
  inputs:
    - name: demo-app-only
      type: application
      application:
        includes:
          - namespace: demo-app
  pipelines:
    - name: all-logs-to-loki
      inputRefs:
        - application
        - infrastructure
        - audit
      outputRefs:
        - lokistack-out
    - name: demo-app-to-siem
      inputRefs:
        - demo-app-only
      outputRefs:
        - siem-syslog

Apply it with oc apply -f clusterlogforwarder.yaml. The collector pod restarts, and oc logs -n syslog-test deploy/syslog-receiver shows one RFC5424 line per log entry:

<14>1 2026-10-07T22:13:18.183162Z crc demo-app ab2c6913-b0a4-4ae2-8e01-c043c2f9a5dc container - namespace_name=demo-app, container_name=ubi-minimal, pod_name=log-generator-577b98c498-9cnwb, message=level=info msg="order processed" order_id=559
<11>1 2026-10-07T22:13:20.186836Z crc demo-app ab2c6913-b0a4-4ae2-8e01-c043c2f9a5dc container - namespace_name=demo-app, container_name=ubi-minimal, pod_name=log-generator-577b98c498-9cnwb, message=level=error msg="payment API timeout" order_id=560

Three details from that output. The priority follows the detected level: info lines arrive as <14> (user.info) and error lines as <11> (user.err), so SIEM severity rules work without parsing. appName carries the namespace. Without payloadKey, our first run sent the entire JSON record (annotations, pod IPs, labels) as the message body, which most SIEMs index as one opaque string.

Set retention per tenant

The global retention.days: 7 in the LokiStack applies to every tenant. Override it per tenant, and per stream inside a tenant, with a patch. Red Hat supports local Loki retention of up to 30 days, so anything longer belongs in an external store:

vim retention-patch.yaml

This keeps audit logs for 30 days, application logs for 3, and the chatty demo-app namespace for 1:

spec:
  limits:
    global:
      retention:
        days: 7
    tenants:
      application:
        retention:
          days: 3
          streams:
            - days: 1
              priority: 1
              selector: '{kubernetes_namespace_name="demo-app"}'
      audit:
        retention:
          days: 30

Merge it into the LokiStack and read back the runtime overrides Loki actually uses:

oc patch lokistack logging-loki -n openshift-logging --type=merge --patch-file retention-patch.yaml
oc get cm logging-loki-config -n openshift-logging -o jsonpath='{.data.runtime-config\.yaml}'

The overrides show up per tenant, with the stream rule under application:

---
overrides:
  application:
    retention_period: 3d
    retention_stream:
    - selector: '{kubernetes_namespace_name="demo-app"}'
      priority: 1
      period: 1d
  audit:
    retention_period: 30d

The merge patch kept the ingestion limits set earlier. The compactor does the deleting, so the bucket credentials must allow DeleteObject or the bucket grows forever while the LokiStack still says Ready.

Moving from the lab stack to a production LokiStack

The 1x.demo size has no resource requests and no replication. In its first 30 minutes on a single idle node, with audit logging on and the initial backlog replay included, our bucket collected 34 chunks and 43.6 MB. Use that rate only to sanity-check the compression math, never to size a real cluster. Before pointing production nodes at Loki, change these:

• Set size to 1x.extra-small or 1x.small. From Logging 6.6, both run three ingesters, so one ingester can restart without interrupting ingestion.
• Use a real bucket (S3, GCS, Azure Blob, ODF) and let the compactor handle expiry through the retention settings above. On AWS, prefer STS short-term credentials over static keys.
• Re-measure ingest on the LokiStack Writes dashboard (Observe > Dashboards) after a week, then set ingestionRate above the observed peak instead of guessing.
• Keep the forwarder inputs to what someone will actually read. Check the per-tenant volume with the metric query above before deciding. If audit logs only matter to the security team, forwarding them to the SIEM alone keeps them out of Loki entirely.
• Pin both logging subscriptions to the same stable-6.x channel and move them together. A Logging Operator one minor ahead of the Loki Operator is an unsupported pair.

If you also run Loki outside OpenShift, the upstream Helm route in our Loki on Kubernetes guide uses the same storage schema and LogQL, so queries carry over unchanged. The operator-managed stack is the one Red Hat supports on OpenShift, and the official install guide is the reference for STS, Workload Identity and disconnected catalogs.