Category: Culture

[RFC] Helm Template

A. HelmChart Template

Prerequirements

  • Helm
### Linux ###
$ curl -fsSL -o get_helm.sh <https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3>
$ chmod 700 get_helm.sh
$ ./get_helm.sh

### MacOS ###
$ brew install helm
  • Helmfile
$ wget https://github.com/roboll/helmfile/releases/download/v0.139.7/helmfile_linux_amd64
$ chmod +x helmfile_linux_amd64
$ sudo mv helmfile_linux_amd64 /usr/local/bin/helmfile
  • Helm Plugins
$ helm plugin install https://github.com/databus23/helm-diff
$ helm plugin install https://github.com/hypnoglow/helm-s3.git
  • Added Mandatory Repository
$ helm repo add stable https://charts.helm.sh/stable
$ helm repo update

Helm Repository

  • Check Repository Helm
$ helm repo list
----
NAME            URL
stable          https://charts.helm.sh/stable
  • Adding Repository Helm
### LAB ###
AWS_REGION=ap-southeast-1 helm repo add devopscorner-lab s3://devopscorner-charts/lab

### STAGING ###
AWS_REGION=ap-southeast-1 helm repo add devopscorner-staging s3://devopscorner-charts/staging

### PRODUCTION ###
AWS_REGION=ap-southeast-1 helm repo add devopscorner s3://devopscorner-charts/prod

helm repo update

Creating HelmChart Template

$ helm create [helmchart_name]
---
eg: 
$ helm create myhelm

$ tree myhelm
myhelm
├── Chart.yaml
├── charts
├── templates
│   ├── NOTES.txt
│   ├── _helpers.tpl
│   ├── deployment.yaml
│   ├── hpa.yaml
│   ├── ingress.yaml
│   ├── service.yaml
│   ├── serviceaccount.yaml
│   └── tests
│       └── test-connection.yaml
└── values.yaml

3 directories, 10 files

Structure HelmChart Template (Multi Environment)

.
├── template
│   ├── lab
│   │   ├── api
│   │   │   ├── Chart.yaml
│   │   │   ├── api.yaml
│   │   │   ├── templates
│   │   │   │   ├── _helpers.tpl
│   │   │   │   └── serviceaccount.yaml
│   │   │   └── values.yaml
│   │   ├── backend
│   │   │   ├── Chart.yaml
│   │   │   ├── backend.yaml
│   │   │   ├── templates
│   │   │   │   ├── _helpers.tpl
│   │   │   │   └── serviceaccount.yaml
│   │   │   └── values.yaml
│   │   ├── frontend
│   │   │   ├── Chart.yaml
│   │   │   ├── frontend.yaml
│   │   │   ├── templates
│   │   │   │   ├── _helpers.tpl
│   │   │   │   └── serviceaccount.yaml
│   │   │   └── values.yaml
│   │   └── svcrole
│   │       ├── Chart.yaml
│   │       ├── templates
│   │       │   ├── _helpers.tpl
│   │       │   ├── clusterrole.yaml
│   │       │   ├── rolebinding.yaml
│   │       │   └── serviceaccount.yaml
│   │       └── values.yaml
... ... ...   
│   ├── [environment]
│   │   ├── api
│   │   │   └── values.yaml
│   │   ├── backend
│   │   │   └── values.yaml
│   │   ├── frontend
│   │   │   └── values.yaml
│   │   └── svcrole
│   │       └── values.yaml
└── test
    ├── lab
    │   ├── helmfile.yaml
    │   └── values
    │       ├── api
    │       │   └── api.yaml
    │       ├── backend
    │       │   └── backend.yaml
    │       ├── frontend
    │       │   └── frontend.yaml
    │       └── svcrole
    │           ├── account.yaml
    │           ├── api.yaml
    │           ├── backend.yaml
    │           └── frontend.yaml
    └── staging
        ├── helmfile.yaml
        └── values
            ├── api
            │   └── api.yaml
            ├── backend
            │   └── backend.yaml
            ├── frontend
            │   └── frontend.yaml
            └── svcrole
                ├── account.yaml
                ├── api.yaml
                ├── backend.yaml
                └── frontend.yaml

HelmChart In Repository

  • Structure on services repository
_infra/
   dev/
      helmfile.yaml
      values/
            api/values.yaml
            backend/values.yaml
            svcrole/values.yaml
            frontend/values.yaml

Testing Helm

  • Testing the Chart Template
helm template ./api -f values/api/values.yaml
helm template ./backend -f values/backend/values.yaml
helm template ./svcrole -f values/svcrole/values.yaml
helm template ./frontend -f values/frontend/values.yaml

Packing HelmChart

  • Create zip Packate of HelmChart
helm package api
helm package backend
helm package svcrole
helm package frontend

Update HelmChart Template

  • Push chart into private repository
### LAB ###
helm s3 push api-[version].tgz devopscorner-lab --force
helm s3 push backend-[version].tgz devopscorner-lab --force
helm s3 push frontend-[version].tgz devopscorner-lab --force
helm s3 push svcrole-[version].tgz devopscorner-lab --force
---
### STAGING ###
helm s3 push api-[version].tgz devopscorner-staging --force
helm s3 push backend-[version].tgz devopscorner-staging --force
helm s3 push frontend-[version].tgz devopscorner-staging --force
helm s3 push svcrole-[version].tgz devopscorner-staging --force
---
### PRODUCTION ###
helm s3 push api-[version].tgz devopscorner --force
helm s3 push backend-[version].tgz devopscorner --force
helm s3 push frontend-[version].tgz devopscorner --force
helm s3 push svcrole-[version].tgz devopscorner --force

B. Versioning HelmChart

  • Change Version HelmChart
$ vi api/Chart.yaml

# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: "1.1.0-rc"
  • Repacking HelmChart template
  • Repush HelmChart into private repository

C. Using Versioning HelmChart in helmfile.yaml

  • Repository Lab
---
repositories:
  - name: devopscorner-lab
    url: s3://devopscorner-charts/lab

templates:
  default: &default
    namespace: devopscorner
    version: "1.0.0-rc"

releases:
  - name: devopscorner-api
    chart: devopscorner-lab/api
    values:
      - ./values/api/values.yaml
    <<: *default

  - name: devopscorner-backend
    chart: devopscorner-lab/backend
    values:
      - ./values/backend/values.yaml
    <<: *default

  - name: devopscorner-frontend
    chart: devopscorner-lab/frontend
    values:
      - ./values/frontend/values.yaml
    <<: *default

  - name: devopscorner-svcaccount
    chart: devopscorner-lab/svcrole
    values:
      - ./values/svcrole/account.yaml
    <<: *default

[RFC] Logging

A. Concepts

Standardization export log path and name

eg:
----
/var/log/[microservice-name]/[microservice-name]-error.log   # error only
/var/log/[microservice-name]/[microservice-name].log         # info, warning & debug

Log using JSON formatted

Severity logs & formatting logs

eg: INFO
---
{
  "datetime": "2020-10-10 20:01:59TZ+0700"
  "severity": "info",
  "message": "yes, this is info"
}

eg: WARNING
---
{
  "datetime": "2020-10-10 20:01:59TZ+0700"
  "severity": "warning",
  "message": "this is warning"
}

eg: ERROR
---
{
  "datetime": "2020-10-10 20:01:59TZ+0700"
  "severity": "error",
  "code": 404
  "message": "not found"
}

eg: DEBUG (optional)
---
{
  "datetime": "2020-10-10 20:01:59TZ+0700"
  "severity": "debug",
  "code": 100
  "message": "describe debug information (criteria by number) "
}

Logrotation & compression

# /etc/logrotate.d/[microservice-name]
---
/var/log/[microservice-name]/[microservice-name].log {
        rotate 12
        weekly
        missingok
        notifempty
        compress
        delaycompress
        size 50M
        notifempty
        sharedscripts
        postrotate
           /usr/bin/killall -HUP [microservice-name]
        endscript
}

/var/log/[microservice-name]/[microservice-name]-error.log {
        rotate 12
        weekly
        missingok
        notifempty
        compress
        delaycompress
        size 50M
        notifempty
        sharedscripts
        postrotate
           /usr/bin/killall -HUP [microservice-name]
        endscript
}

Log4j (JAVA)

# log4j.properties
---
log4j.rootLogger=INFO, fileLogger
log4j.appender.fileLogger=org.apache.log4j.RollingFileAppender
log4j.appender.fileLogger.layout=org.apache.log4j.PatternLayout
log4j.appender.fileLogger.layout.ConversionPattern=%d [%t] %-5p (%F:%L) - %m%n
log4j.appender.fileLogger.File=example.log
log4j.appender.fileLogger.MaxFileSize=50MB
log4j.appender.fileLogger.MaxBackupIndex=12

Schedule logging (log exporter)

  • Schedule with cron (crontab)
/etc/cron.d/[microservice-name]
  • Schedule with systemd
/etc/systemd/system/[microservice-name].service
/etc/systemd/system/[microservice-name].timer

B. Tools

  • GO

https://github.com/sirupsen/logrus

  • Python
from datetime import datetime
import logging
import time
import json
       
def main():
    print("--- Staring Log Exporter Agent ---")
    logging.basicConfig(level=logging.INFO, filename="/var/log/[microservice-name]/[microservice-name].log", format="%(message)s")

if __name__ == '__main__':
    main()

[RFC] Load Testing

A. Overview

Load Testing / Stress Test is a mechanism for flooding the network with a certain requests during some period time.

To have successfully loadtest we need:

  • Preparation
  • How many request will be use ?
  • How much time execute this loadtesting ?
  • How many microservices involved ?
  • Where is the environment from this loadtest will be impact (onprem, staging, production) ?
  • Cost effective estimation for this loadtesting event ?

B. Technical Documentation

Define your technical documentation for this loadtest (estimation and after loadtesting event)

  • Environment
  • Number of Microservices
  • Number of Request
  • Period Time (Begin – End Snapshot Datetime)
  • Target Metrics
  • Snapshot Result (Metrics Dashboard & Logs)