# Deployment Troubleshooting

# 1. Known issues

Refer to the Mojaloop Helm Chart Release notes for more information on known issues: https://github.com/mojaloop/helm/releases (opens new window).

# 1.1. Nginx-Ingress Controller support for Mojaloop Helm release v13.x - v14.0x.x support for Kubernetes v1.20 - v1.21

If you are using Mojaloop v13.x - v14.0.x, and want to install the Nginx-Ingress controller, then it is recommended that you install Nginx-Ingress Controller v0.47.0 along with Kubernetes v1.20 - v1.21 due to breaking changes introduce in Kubernetes v1.22.

If you are using helm, this can be done as follows:

helm install ingress-nginx ingress-nginx --version="3.33.0" --repo https://kubernetes.thub.io/ingress-nginx

# 1.2. Nginx-Ingress Controller support for Mojaloop Helm release v12.x

If you are installing Mojaloop v12.x with an Nginx-Ingress controller version newer than v0.22.0, ensure you create a custom Mojaloop v12.0.0 values config (opens new window) with the following changes prior to stall:**

## **LOOK FOR THIS LINE IN mojaloop/values.yaml CONFIG FILE**
mojaloop-simulator:
  ingress:
   ## nginx ingress controller >= v0.22.0 <-- **COMMENT THE FOLLOWING THREE LINES BELOW:**
   # annotations: <-- COMMENTED
   #  nginx.ingress.kubernetes.io/rewrite-target: '/$2' <-- COMMENTED
   # ingressPathRewriteRegex: (/|$)(.*) <-- COMMENTED
   ## nginx ingress controller < v0.22.0 <-- **UNCOMMENT THE FOLLOWING THREE LINES LOW:**
   annotations:
     nginx.ingress.kubernetes.io/rewrite-target: '/'
   ingressPathRewriteRegex: "/"

Note: This is NOT necessary if you are installing Mojaloop v13.x or newer.

# 1.3. Docker Desktop Kubernetes support for Mojaloop v13.x - v14.0.x

If you are installing Mojaloop v13.x - v14.0.x, on Windows or MacOS, it is recommend that you install Docker Desktop v4.2.0 (opens new window) version as it comes packaged with Kubernetes v1.21.5 which meets Mojaloop v13.x - v14.0.x version which matches Deployment Guide (1. Pre-requisites) recommendations.

# 1.4. Mojaloop Helm release v10.x or less does not support Kubernetes v1.16 or greater

# 1.4.1 Description

Note: This is only applicable to Mojaloop Helm v10.x or less release.

When installing mojaloop helm charts, the following error occurs:

Error: validation failed: [unable to recognize "": no matches for kind "Deployment" in version "apps/v1beta2", unable to recognize "": no matches for kind "Deployment" in version "extensions/v1beta1", unable to recognize "": no matches for kind "StatefulSet" in version "apps/v1beta2", unable to recognize "": no matches for kind "StatefulSet" in version "apps/v1beta1"]

# 1.4.2 Reason

In version 1.16 of Kubernetes breaking change has been introduced (more about it in "Deprecations and Removals" of Kubernetes release notes (opens new window). The Kubernetes API versions apps/v1beta1 and apps/v1beta2are no longer supported and and have been replaced by apps/v1.

Mojaloop helm charts v10 or less refer to deprecated ids, therefore it's not possible to install v10- on Kubernetes version above 1.15 without manually modification.

Refer to the following issue for more info: mojaloop/helm#219 (opens new window)

# 1.4.3 Fixes

Ensure that you are deploying Mojaloop Helm charts v10.x or less on v1.15 of Kubernetes.

# 2. Deployment issues

# 2.1. `ERR_NAME_NOT_RESOLVED` Error

# 2.1.1 Description

The following error is displayed when attempting to access an end-point (e.g. central-ledger.local) via the Kubernetes Service directly in a browser: ERR_NAME_NOT_RESOLVED

# 2.1.2 Fixes

Verify that Mojaloop was deployed correctly (by checking that the helm chart(s) was installed) by executing:
```
helm list
```
If the helm charts are not listed, see the Deployment Guide - 5.1. Mojaloop Helm Deployment section to install a chart.
Ensure that all the Mojaloop Pods/Containers have started up correctly and are available through the Kubernetes dashboard.
Note that the Mojaloop deployment via Helm can take a few minutes to initially startup depending on the system's available resources and specification. Expect this to take anything between 2-10 minutes.

# 2.3. MicroK8s - Connectivity Issues

# 2.3.1 Description

My pods can’t reach the internet or each other (but my MicroK8s host machine can).

An example of this is that the Central-Ledger logs indicate that there is an error with the Broker transport as per the following example:

2019-11-05T12:28:10.470Z - info: Server running at: 
2019-11-05T12:28:10.474Z - info: Handler Setup - Registering {"type":"prepare","enabled":true}!
2019-11-05T12:28:10.476Z - info: CreateHandler::connect - creating Consumer for topics: [topic-transfer-prepare]
2019-11-05T12:28:10.515Z - info: CreateHandler::connect - successfully connected to topics: [topic-transfer-prepare]
2019-11-05T12:30:20.960Z - error: Consumer::onError()[topics='topic-transfer-prepare'] - Error: Local: Broker transport failure)

# 2.3.2 Fixes

Make sure packets to/from the pod network interface can be forwarded to/from the default interface on the host via the iptables tool. Such changes can be made persistent by installing the iptables-persistent package:

sudo iptables -P FORWARD ACCEPT
sudo apt-get install iptables-persistent

or, if using ufw:

sudo ufw default allow routed

The MicroK8s inspect command can be used to check the firewall configuration:

microk8s.inspect

# 3. Ingress issues

# 3.1. Ingress rules are not resolving for Nginx Ingress v0.22 or later when installing Mojaloop Helm v12.x or less

# 3.1.1 Description

Note: This is only applicable to Mojaloop Helm v12.x or less release.

Ingress rules are unable to resolve to the correct path based on the annotations specified in the values.yaml (opens new window) configuration files when using Nginx Ingress controllers v0.22 or later.

This is due to the changes introduced in Nginx Ingress controllers that are v0.22 or later as per the following link: https://kubernetes.github.io/ingress-nginx/examples/rewrite/#rewrite-target.

# 3.1.2 Fixes

Make the following change to Ingress annotations (from --> to) in the values.yaml files:

nginx.ingress.kubernetes.io/rewrite-target: '/'` --> `nginx.ingress.kubernetes.io/rewrite-target: '/$1'

# 3.2. Ingress rules are not resolving for Nginx Ingress earlier than v0.22

# 3.2.1 Description