Introduction / Context

Recently I had an interesting opportunity to work on building a self-service platform with Knative in center. Knative is a Kubernetes-based platform to build, deploy, and manage modern serverless workloads. It provides a set of middleware components that are essential to building modern, source-centric, and container-based applications that can run anywhere: on-premises, in the cloud, or even in a third-party data-center. It is a platform that is built on top of Kubernetes and is designed to be portable across different cloud providers.

In this post, I'll document the whole process, so that someone from the future (potentially me) may find it useful. I'll also try to include some debugging tips, that I found useful while working on this project. The official documentation of Knative is good, but not great. The official manifests are too inconsistent, and the documentation is not always up-to-date. So here it goes.

Caution: The point of this project was to have a basic setup up and running, so I didn't bother with some ideal best practices - like using proper RBAC, bootstrapping with EKS Blueprints, or parameterising the Terrafrom + K8s manifests, using a secret store / vault, etc.

Prerequisites

Before diving into a managed Kubernetes solution like EKS, that may cost you a lot of money along the way, I'd recommend using Minikube with --driver=docker. I'm a big fan of DevContainers, you may find detail about it in my previous post: https://audacioustux.com/posts/getting-started-devcontainer/ - it's optional though.

The whole setup may require at least 4 vCPUs and 8GB of RAM. On my EKS setup, I used 3 nodes of m5.large (2 vCPU, 8GB RAM, AL2_x86_64).

Also, you'll need to have the following tools installed:

Setup

Let's first create a bash script to automate some of the repetitive tasks and make things reproducible. Create a file named up.sh with the following content:

#!/usr/bin/env bash

set -eax
# -e = exit immediately if a command exits with a non-zero status.
# -a = each variable or function that is modified or created is given the export attribute. Usefull for using with `parallel` command.
# -x = print a trace of simple commands and their arguments after they are expanded and before they are executed. Useful for debugging, but bad for security.

# get the remote repo url
export REPO=`git remote get-url origin`
# argocd cli options - every argocd command will create a port-forward to argocd server, and close it after the command is finished.
export ARGOCD_OPTS='--port-forward --port-forward-namespace argocd'

deploy-argocd(){
    # Note: This is not ideal - but it's good enough for a quick setup
    echo "Deploying argocd..."
    # create argocd namespace declaratively, so the script can be run multiple times without any error
    kubectl create namespace argocd --dry-run=client -o yaml | kubectl apply -f -

    kubectl apply -k k8s/kustomize/argocd -n argocd
    # wait for all argocd pods to be ready. this may take a while.
    echo "Waiting for all argocd pods to be ready..."
    kubectl wait --for=condition=ready pod \\\\
        --all \\\\
        -n argocd \\\\
        --timeout=300s
}

login-argocd(){
    echo "Logging in to argocd..."
    # Note: ideally, password should be stored in a secret store / vault, argocd-secret should be patched with bcrypt hash of the password:
    # local password=`argocd account bcrypt --password ${ARGOCD_PASSWORD:?}`
    # kubectl patch secret argocd-secret \\\\
    #     -n argocd \\\\
    #     -p '{"stringData": {"admin.password": "'$password'"}}'
    # and argocd-initial-admin-secret should be deleted
    local password=`kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d`

    argocd login --username admin --password $password
}

add-private-repos(){
    # add private repos to argocd so that argocd can access them
    echo "Adding private repo..."
    local repo_url=($REPO)

    for repo in "${repo_url[@]}"
    do
        argocd repo add $repo --username ${GIT_USERNAME:?} --password ${GIT_TOKEN:?}
    done
}

deploy-argocd-apps(){
    # deploy all the Application.argoproj.io manifests in k8s/apps directory
    # Note: ideally, it should be done in Apps of Apps pattern.
    echo "Deploying argocd apps..."
    kubectl apply --recursive -f k8s/apps
}

deploy-secrets(){
    # deploy secrets, that will be used for CI/CD pipelines.
    echo "Deploying secrets..."
    local git=`kubectl create secret generic git-config --from-literal=username=$GIT_USERNAME --from-literal=password=$GIT_TOKEN --dry-run=client -o yaml`
    local docker=`kubectl create secret generic docker-config --from-file=$HOME/.docker/config.json --dry-run=client -o yaml`

    echo "$git" | kubectl apply -n argo -f -
    echo "$docker" | kubectl apply -n argo -f -
}

deploy-argocd
login-argocd
add-private-repos
deploy-argocd-apps

# ebort - is a script that re-tries a command until it succeeds. This is useful as some of the namespaces are created by argocd apps, and it may take a while for the namespaces to be created.
# 2> /dev/null is used to suppress the error messages.
ebort -- deploy-secrets 2> /dev/null

Okay, this script will fail, because we don't have the manifests yet. Let's create the manifests now.

ArgoCD