DARE Deployment instructions

Deployment on ubuntu

Kubernetes setup

    # Install docker 
	sudo apt install docker.io
	
    # Enable docker 
	sudo systemctl enable docker

    # Add Kubernetes signing key 
	curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
	
    # Add Xenial Kubernetes Repository 
	sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
	
    # Install Kubeadm 
	sudo apt install kubeadm=1.15.3-00 kubectl=1.15.3-00 kubelet=1.15.3-00
	
    # Initialize Kubernetes on the master node
	sudo kubeadm init

    # start using your cluster 
	mkdir -p $HOME/.kube & sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config & sudo chown $(id -u):$(id -g) $HOME/.kube/config

    # Deploy a Pod Network through the master node 
	kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

    # Run 
	kubectl taint nodes --all node-role.kubernetes.io/master-

MPI-operator (v 0.1.0)

wget https://github.com/kubeflow/mpi-operator/archive/0.1.0.tar.gz
tar -xvzf 0.1.0.tar.gz
cd mpi-operator-0.1.0/deploy/
In 3-mpi-operator.yaml, change the two images (mpi-operator & kubectl-delivery) from latest to 0.1.0

	- name: mpi-operator
        image: mpioperator/mpi-operator:0.1.0
        args: [
          "-alsologtostderr",
          "--gpus-per-node", "8",
          "--kubectl-delivery-image",
          "mpioperator/kubectl-delivery:0.1.0"

Deploy mpi-operator:

   kubectl create -f 0-crd.yaml
   kubectl create -f 1-namespace.yaml
   kubectl create -f 2-rbac.yaml
   kubectl create -f 3-mpi-operator.yaml

Rook Shared file system (release-0.8)

git clone https://github.com/rook/rook.git
cd rook/
git checkout release-0.8
cd cluster/examples/kubernetes/ceph

   kubectl create -f operator.yaml
   kubectl create -f cluster.yaml
   kubectl create -f filesystem.yaml
   kubectl create -f storageclass.yaml

NOTE: rook-ceph-block storageclass is not default. To set as default:

   kubectl patch storageclass rook-ceph-block -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Ingress

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/130af33510882ae62c89277f2ad0baca50e0fafe/deploy/static/mandatory.yaml
Check ingress controller: kubectl get pods -n ingress-nginx
mkdir ingress-deployment && cd ingress-deployment
vi nginx-ingress.yaml
Copy-Paste inside the file the below:

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  externalTrafficPolicy: Local
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: http
      port: 80
      targetPort: http
    - name: https
      port: 443
      targetPort: https

kubectl apply -f nginx-ingress.yaml
Check the created service

kubectl get svc -n ingress-nginx

Install Helm & Tiller

DARE uses the Helm package manager (https://helm.sh/) for Kubernetes to install and manage some of the external packages it uses. This facilitates the installation and upgrade of external components and prevents duplicated work on Kubernetes descriptors for well-known applications. To use Helm, we need to install the helm command and corresponding service (Tiller) first. Summary:

Download 3.1.1 release from https://github.com/helm/helm/releases
Install helm binary to /usr/local/bin
Initialize helm and Tiller
Update package sources

## Download & unpack release package
$ wget https://get.helm.sh/helm-v3.1.1-linux-amd64.tar.gz
$ tar xf helm-v3.1.1-linux-amd64.tar.gz
## move to path
$ sudo mv linux-amd64/helm /usr/local/bin/
## create service account
$ kubectl create serviceaccount -n kube-system tiller
## create role for RBAC
$ kubectl create clusterrolebinding tiller-binding --clusterrole=cluster-admin --serviceaccount kube-system:tiller
## Update package sources
$ helm repo update

Install cert-manager

To maintain certificates for the externally reachable services and pages, DARE uses the Let’s Encrypt Certification Authority (https://letsencrypt.org/) through its ACME protocl (https://tools.ietf.org/html/rfc8555). The Cert-Manager addon (https://cert-manager.io) automates this process even further so that certificates are automatically issued, configured in the ingress and updated based on annotations in our Kubernetes descriptors. For installation, we use the official Helm package from Helm hub (https://hub.helm.sh/charts/jetstack/cert-manager). Summary:

Install the custom resource definitions required for cert-manager
Activate the Jetstack Helm repository
Install the Cert-Manager package
Add a clusterissue for letsencrypt

## install custom resource definitions
$ kubectl apply --validate=false -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.14/deploy/manifests/00-crds.yaml
## Add the Jetstack Helm repository
$ helm repo add jetstack https://charts.jetstack.io
## Install the cert-manager helm chart
$ helm install cert-manager --version v0.14.0 jetstack/cert-manager
## Install the letsencrypt ClusterIssuer (careful: needs customization!)
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/k8s.io/master/cert-manager/letsencrypt-prod.yaml

Install Keycloak (WIP)

## Add codecentric package source and update package sources.
$ helm repo add codecentric https://codecentric.github.io/helm-charts
$ helm repo update
## Install Keycloak Helm Chart
$ helm install keycloak -f keycloak-values.yaml --version 8.0.0 codecentric/keycloak 

## To redeploy keycloak:
$ helm upgrade keycloak -f keycloak-values.yaml codecentric/keycloak

Note: in order to find keycloak-values.yaml, go to k8s directories

Once Keycloak is deployed:

Login to the Admin panel in keycloak UI and create a new realm named dare
In the new realm, register the dare-login component as client. Use confidential strategy and after saving the client copy the secret key
Paste the secret key in the dare-login-dp.yaml
Create user accounts
Create a user with username admin and some password. Copy the password in the dare-login-dp.yaml in readiness and liveness prob

DARE platform

git clone https://gitlab.com/project-dare/dare-platform.git
cd dare-platform/k8s/
execute the script deploy.sh
Expose deployments:

    kubectl expose deployment d4p-registry --type=NodePort --name=d4p-registry-public
    kubectl expose deployment dare-login --type=NodePort --name=dare-login-public
    kubectl expose deployment exec-api --type=NodePort --name=exec-api-public
    kubectl expose deployment exec-registry --type=NodePort --name=exec-registry-public
    kubectl expose deployment playground --type=NodePort --name=playground-public
    kubectl expose deployment semantic-data --type=NodePort --name=semantic-data-public
    kubectl expose deployment workflow-registry --type=NodePort --name=workflow-registry-public

If volumes are not mounted –> systemctl restart kubelet

JupyterHub

This is an optional step. If you have multiple users and you want them to create an environment with existing notebooks, you can create a JupyterHub. We have a demo yaml in k8s (jupyterhub-config.yaml) but since it contained sensitive information we have comments to the fields that need to be completed.

Create secret token for jupyterhub-config.yaml

    openssl rand -hex 32

Copy the token and paste it in the aforementioned yaml file in the secretToken field
Run the following

    helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/
    helm repo update
    kubectl create namespace jupyterhub
    helm upgrade --install jupyterhub jupyterhub/jupyterhub --namespace jupyterhub --version=0.9.0 --values jupyterhub-config.yaml

Installation