KOBIL Shift services initial verification
Shift Installation Verification
Audience: Admin-Expert (DevOps) to validate the KOBIL Shift deployment.
KOBIL Shift deployment is a 2 step deployment:
- First install the KOBIL Shift-Operator Charts to the target namespace
- Secondly install the KOBIL Shift Charts to the target namespace.
Validation of KOBIL Shift-Operator deployment
-
Validate the created image pull secret first - and ensure image download works
Find more info for Repo access credentials here: KOBIL Repository Access In case the image pull secret covers invalid access credentials, the image pull action fails and creates events. -
When running the
helm install shift-operator ....you may use debug option for helm and to save the output for review.
Samplehelm install shift-operator -n shift -f ./shift-operator.yaml --debug 2>&1 | tee shift-op.install.out -
When the KOBIL Shift Operator is successfully deployed you will find appropriate deployment and created KOBIL CRD. Verify by running:
kubectl get deployment- here we expect to see the KOBIL Shift Operator deployment is available
kubectl get crd | grep -i kobil- here we expect to see the KOBIL servicegroup CRD was created and is listed
Validation of KOBIL Shift deployment
-
Validate the Shift Operator pod is running - and validate events for its runtime.
-
When running the
helm install shift ....you may use debug option for helm and to save the output for review (see Operator above). -
Running the
helm install shift ....deployment will create the KOBIL Shift CRD "servicegroup" objects which are then processed by the KOBIL Shift Operator. -
When the KOBIL Shift helm install is successfully processed you will find appropriate "servicegroup" objects. Verify by running:
kubectl get servicegroups- here we expect to see multiple servicegroup objects (depending on the deployed services)
helm list [-n namespace]- here we expect to see the KOBIL Shift services applied into this namespace -
The KOBIL Shift Operator will process the KOBIL Shift servicegroup objects and update the READY status (which is on TRUE once the servicegroup is successfully processed).
-
The KOBIL Shift Operator will run the
helm install <service> .....for the specified services per servicegroup object settings. Therefore running ahelm listwill show the installed KOBIL Shift microservices.
Sample data
Checking the Kubernetes resources for deployt KOBIL Shift.
Due to the 2stage deployment implementation - first doing the KOBIL Shift-Operator -
then the KOBIL Shift, it is important to check for the pod runtime and servicegroups.
Commands:
# verify pod runtime (Item 1 - find below more info)
kubectl get pods
# verify READY status for KOBIL Shift "servicegroups" (Item 2)
kubectl get servicegroups
# checking for servicegroup events (Item 3)
kubectl describe servicegroup <servicegroup-name>
# checking for deployments (Item 4)
kubectl get deployments
# checking for the Kafka-resouces (Item 5)
kubectl get kafka
kubectl get kafkatopics
# checking for the helm installed "releases" (Item 6)
helm list
helm list --all
The expected info is as follows:
Results:
Item 1:
kubectl get pods should show all pods at stable runtime (READY status) and no (high) restart counts:
NAME READY STATUS RESTARTS AGE
shift-ast-client-properties-76bc8d48-zs4r7 1/1 Running 0 3h14m
shift-ast-key-protection-5699f5b8-cpjhp 1/1 Running 0 49d
shift-ast-localization-56fc7767-7f2wb 1/1 Running 0 42d
shift-ast-stream-7bd47cb-vkj87 1/1 Running 0 42d
shift-ast-tms-5fbddff8-x56pr 1/1 Running 0 3h46m
shift-ast-version-7957588f-slxvt 1/1 Running 0 42d
shift-ast-webhooks-65d49456-cgtnr 1/1 Running 0 42d
shift-astca-699f7f7c-zqtf8 1/1 Running 0 3h46m
shift-astclientmanagement-56c77fd4-nnq58 1/1 Running 0 42d
shift-astcpb-5d7fd968-cfsqj 1/1 Running 0 47d
shift-astlogin-58fbb9fb-st58v 1/1 Running 0 42d
shift-idp-core-6f4c5779-kkbx6 1/1 Running 0 6d8h
shift-idp-scp-connector-6c6cc694-kkxv2 1/1 Running 0 42d
shift-scp-notifier-54568f6d-4cbrx 3/3 Running 0 42d
shift-operator-g-shift-operator-69bbcf75-wdqmk 1/1 Running 0 3h47m
Item 2:
kubectl get servicegroup should show all servicegroups are at READY status "true" (not on "false")
NAME READY SERVICES
shift-gondor-pt-shift-ast true {"ast-client-properties":{"chart":"ast-client-properties","version":"7.4.3"},....}
shift-gondor-pt-shift-idp true {"idp-core":{"chart":"idp-core-ibm-pkv","version":"2.5.0"},....}
shift-gondor-pt-shift-kafka true {"kafka":{"chart":"kafka","version":"0.15.0"}}
shift-gondor-pt-shift-scp true {"scp-notifier":{"chart":"scp-notifier","version":"0.43.1"}}
Item 3:
kubectl describe servicegroup <servicegroup> will provide you more details about servicegroup in case of READY:false status.
NAME READY SERVICES
shift-gondor-pt--shift-ast false {"ast-client-properties".......}
Name: shift-shift-ast
Namespace: env-gondor-rqqyv
Labels: app.kubernetes.io/component=shift-chart
app.kubernetes.io/instance=shift-gondor-rqqyv
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=shift
app.kubernetes.io/part-of=shift
app.kubernetes.io/version=0.147.0
helm.sh/chart=shift-0.147.0
Annotations: app.shift.kobil.com/release-prefix: shift-
API Version: shift.kobil.com/v1alpha1
Kind: ServiceGroup
Metadata:
Creation Timestamp: 2023-01-26T15:36:22Z
....
...
..
Status:
Failure: Installation ast-tms:
Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
Installation astca:
Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress
Ready: false
Item 4:
kubectl get deployment should show all deployments being READY and started pods are matching
NAME {color:#172b4d}READY{color}READY UP-TO-DATE AVAILABLE AGE
shift--ast-client-properties 1/1 1 1 281d
shift--ast-key-protection 1/1 1 1 281d
shift--ast-localization 1/1 1 1 281d
shift--ast-stream 1/1 1 1 281d
shift--ast-tms 2/2 1 1 281d
shift--ast-version 2/2 1 1 281d
shift--ast-webhooks 1/1 1 1 281d
shift--astca 1/1 1 1 281d
shift--astclientmanagement 1/1 1 1 281d
shift--astcpb 1/1 1 1 281d
shift--astlogin 2/2 1 1 281d
shift--idp-core 3/3 1 1 281d
shift--idp-scp-connector 1/1 1 1 274d
shift--kafka-cluster-entity-operator 1/1 1 1 152d
shift--scp-notifier 2/2 1 1 281d
shift-operator-g-shift-operator 1/1 1 1 348d
{color:#172b4d}True{color}
Item 5:
kubectl get kafkatopics should show all kafkatopics being READY "true"
NAME CLUSTER PARTITIONS REPLICATION FACTOR READY
com.kobil.ast.app.info shift-kafka-cluster 10 1 {color:#172b4d}True{color}
com.kobil.ast.architecture.info shift-kafka-cluster 10 1 True
com.kobil.ast.ca.info shift-kafka-cluster 10 1 True
....
...
..
com.kobil.ast.stream.client.status shift-kafka-cluster 10 1 True
com.kobil.ast.stream.event shift-kafka-cluster 10 1 True
com.kobil.ast.stream.sse shift-kafka-cluster 10 1 True
com.kobil.ast.tms.status shift-kafka-cluster 10 1 True
com.kobil.ast.version.info shift-kafka-cluster 10 1 True
...
..
Item 6:
helm list will provide details about helm deployed releases (services) in this namespace
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
shift-oc-ibmpkv--ast-client-data-generator env-oc-mpkv-gf4wr 9 2023-12-18 14:38:00.081755736 +0000 UTC deployed ast-client-data-generator-0.1.0-rc.712711 0.1.0-rc.712711
shift-oc-ibmpkv--ast-client-properties env-oc-mpkv-gf4wr 10 2023-12-18 14:38:12.821127605 +0000 UTC deployed ast-client-properties-8.20.6 8.20.6
shift-oc-ibmpkv--ast-key-protection env-oc-mpkv-gf4wr 10 2023-12-18 14:38:16.719239255 +0000 UTC deployed ast-key-protection-0.9.2 0.9.2
shift-oc-ibmpkv--ast-localization env-oc-mpkv-gf4wr 10 2023-12-18 14:38:19.925392715 +0000 UTC deployed ast-localization-0.11.5 0.11.5
shift-oc-ibmpkv--ast-stream env-oc-mpkv-gf4wr 10 2023-12-18 14:38:22.658442902 +0000 UTC deployed ast-stream-5.4.4 5.4.4
shift-oc-ibmpkv--ast-tms env-oc-mpkv-gf4wr 10 2023-12-18 14:38:25.814348565 +0000 UTC deployed ast-tms-0.23.3 0.23.3
shift-oc-ibmpkv--ast-version env-oc-mpkv-gf4wr 10 2023-12-18 14:38:30.072663711 +0000 UTC deployed ast-version-11.9.0 11.9.0
shift-oc-ibmpkv--ast-webhooks env-oc-mpkv-gf4wr 10 2023-12-18 14:38:33.389010512 +0000 UTC deployed ast-webhooks-5.17.1 5.17.1
shift-oc-ibmpkv--astca env-oc-mpkv-gf4wr 10 2023-12-18 14:38:37.824305236 +0000 UTC deployed astca-5.7.5 5.7.5
shift-oc-ibmpkv--astclientmanagement env-oc-mpkv-gf4wr 10 2023-12-18 14:38:41.712266343 +0000 UTC deployed astclientmanagement-7.20.7 7.20.7
shift-oc-ibmpkv--astcpb env-oc-mpkv-gf4wr 10 2023-12-18 14:38:44.449769287 +0000 UTC deployed astcpb-4.6.1 4.6.1
shift-oc-ibmpkv--astlogin env-oc-mpkv-gf4wr 10 2023-12-18 14:38:47.016123026 +0000 UTC deployed astlogin-4.16.4 4.16.4
shift-oc-ibmpkv--idp-core env-oc-mpkv-gf4wr 376 2024-01-10 13:53:00.814291772 +0000 UTC deployed idp-core-ibm-pkv-2.5.1-rc.795623 2.5.1-rc.795623
shift-oc-ibmpkv--idp-scheduler env-oc-mpkv-gf4wr 66 2024-01-10 13:43:48.184633517 +0000 UTC deployed idp-scheduler-0.1.4 0.1.4
shift-oc-ibmpkv--idp-scp-connector env-oc-mpkv-gf4wr 376 2024-01-10 13:53:05.131951525 +0000 UTC deployed idp-scp-connector-1.5.0 1.5.0
shift-oc-ibmpkv--kafka env-oc-mpkv-gf4wr 11 2023-12-18 14:38:00.209407762 +0000 UTC deployed kafka-0.15.0 0.15.0