1. Introduction

2. Why Use kubeadm

3. System Requirements

4. Cluster Architecture Overview

5. Networking Requirements

6. Why Disable Swap

7. Understanding Cgroups and Drivers

8. Step-by-Step Setup

   • a. Launch EC2 Instances

   • b. Prepare All Nodes

   • c. Install and Configure Containerd

   • d. Install Kubernetes Binaries

   • e. Initialize Control Plane

   • f. Join Worker Nodes

   • g. Install Network Add-On

9. Validation and Testing

10. Cleanup Reminder

11. Conclusion

📌 Introduction

Managed Kubernetes services such as EKS are convenient, but they hide many of the components that make Kubernetes work. In this guide you'll build a Kubernetes v1.36 cluster from scratch on AWS using kubeadm and containerd, giving you hands-on experience with the control plane, worker nodes, networking, and cluster bootstrapping.

🛠️ Why Use kubeadm ?

kubeadm is a tool designed to simplify Kubernetes cluster setup.

It automates:

• Control plane component setup

• Certificate generation

• Worker node joining

• Bootstrapping configs

Without kubeadm, you’d need to manually configure:

• kube-apiserver

• etcd

• kube-scheduler

• controller-manager

• kubelet

• kube-proxy

• A container runtime (e.g., containerd)

That’s a heavy lift—and error-prone. So let’s do it the smart way.

🧱 System Requirements

Each node should have:

⚠️ Kubernetes cannot install the control plane on Windows. Worker nodes can be Windows, but only Linux supports the control plane.

Note: For learning environments you may allow all TCP within the security group. For production environments restrict traffic to only required Kubernetes ports.

This rule only allows internal traffic — for example:

• Pods talking to each other across nodes

• kubelet or containerd communicating

• Control plane → worker communication

✅ Good for Kubernetes node-to-node communication

❌ NOT enough for traffic from outside AWS (like your laptop)

🌐 Cluster Topology (for this setup)

• Control Plane Node: t2.medium (2 vCPU, 4GB RAM)

• Worker Node 1: t2.medium (2 vCPU, 4GB RAM)

• Worker Node 2: t2.medium (2 vCPU, 4GB RAM)

This is a cost-optimized test cluster for learning—not for heavy workloads.

🔌 Networking Between Nodes

Kubernetes requires:

• Full network connectivity between all nodes (ping from any to any)

• Open ports for components to talk (API server, kubelet, etc.)

• Seamless Pod-to-Pod communication, even across nodes

Without this, your cluster may initialize but won’t function correctly.

🚫 Why Disable Swap?

Kubernetes uses real-time memory metrics to schedule pods. If swap is enabled:

• The OS may “fake” available memory

• Scheduler makes bad decisions

• Pods crash unexpectedly

So we must disable swap to ensure scheduling is predictable.

🧠 Cgroups and Why They Matter

Cgroups (Control Groups) are a Linux kernel feature that:

• Isolate and limit CPU/RAM for processes (like pods)

• Help enforce resource requests/limits in Kubernetes

Cgroup Drivers:

• systemd: Recommended for modern Ubuntu

• cgroupfs: Older or alternate driver

⚠️ Both the container runtime and kubelet must use the same cgroup driver—or you’ll get errors. Thankfully, kubeadm sets kubelet to use systemd by default on modern distros.

⚙️ Step-by-Step Setup

a. Launch EC2 Instances

In AWS:

• Launch 3 EC2 instances

• Ubuntu 24.04 LTS AMI or latest

• SG allows internal communication (use custom SG, allow all TCP from self)

Label your instances clearly: control-node, worker1, worker2.

b. Prepare All Nodes

SSH into each node and run:

Set Hostname

hostnamectl set-hostname control-node # or worker1 / worker2

Enable IPv4 Forwarding

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system

Disable Swap

sudo swapoff -a
sudo sed -i '/ swap / s/^/#/' /etc/fstab

c. Install and Configure Containerd

Install containerd

sudo apt-get update && sudo apt-get install -y containerd

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

Generate Default Config

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml

Enable SystemdCgroup

Update SystemdCgroup to True:

nano /etc/containerd/config.toml

# search for below then update to true
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

Restart containerd

sudo systemctl restart containerd
sudo systemctl enable containerd

d. Install Kubernetes Binaries

Run on all 3 nodes:

# Install prerequisites.Update apt and install the packages needed for repository setup 
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gpg

# Download and save the new signing key.The same key is used for all versions
sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.36/deb/Release.key \
  | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

# Add the new Kubernetes repository. This overwrites any previous Kubernetes list file and points apt to the new repo
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] \
https://pkgs.k8s.io/core:/stable:/v1.36/deb/ /" \
| sudo tee /etc/apt/sources.list.d/kubernetes.list

# Update apt and install the Kubernetes binaries
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

Note: Executing apt-mark hold command on kubernetes components (like kubeadm, kubelet, kubectl) will stop the components from automatically upgrading so it will always be in sync with the cluster’s version.

If they upgrade unexpectedly:

• You may break compatibility.

• Your nodes could fail to join the cluster.

• Upgrades must be coordinated carefully using kubeadm upgrade.

1. (You can substitute v1.36 with another minor release, e.g. v1.30, if you plan to install a different Kubernetes version.)

e. Initialize Control Plane

On control-node only:

# replace <PRIVATE-IP> the the actual Private-IP of your server
sudo kubeadm init \
  --apiserver-advertise-address=<PRIVATE-IP> \
  --pod-network-cidr=10.244.0.0/16 \
  --cri-socket=unix:///run/containerd/containerd.sock

Once your Kubernetes control-plane has initialized successfully

# start Cluster
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown \((id -u):\)(id -g) $HOME/.kube/config

# Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf

f. Join Worker Nodes

Copy the kubeadm join command shown in the output and run it on worker1 and worker2:

sudo kubeadm join <CONTROL-PLANE-IP>:6443 --token <TOKEN> \
  --discovery-token-ca-cert-hash sha256:<HASH> \
  --cri-socket unix:///run/containerd/containerd.sock

g. Install Network Add-On

Use Flannel (compatible with CIDR above):

kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

🔍 Validate Setup

View Nodes

kubectl get nodes

# you should see if you setup correctly 
NAME           STATUS   ROLES           AGE   VERSION
control-node   Ready    control-plane   28m   v1.36.2
worker-01      Ready    <none>          20m   v1.36.2
worker-02      Ready    <none>          20m   v1.36.2

# You can label the workers so it reads worker(purely cosmetic)
kubectl label node worker-01 node-role.kubernetes.io/worker=worker
kubectl label node worker-02 node-role.kubernetes.io/worker=worker

NAME           STATUS   ROLES           AGE   VERSION
control-node   Ready    control-plane   38m   v1.36.2
worker-01      Ready    worker          30m   v1.36.2
worker-02      Ready    worker          30m   v1.36.2

Create Namespace & Pod

kubectl create ns test
kubectl run nginx --image=nginx -n test
kubectl get pods -n test

Your Kubernetes cluster is now fully up and running with:

• ✅ control-node: Ready

• ✅ worker1: Ready

• ✅ worker2: Ready

You’re all set to start deploying workloads!

🧹 Cleanup Reminder

Terminate the EC2 instances if you’re not using them to avoid AWS charges.

📌 Conclusion

You now have:

• Full control plane setup

• Worker nodes registered

• Networking configured

• A working kubectl setup

This setup is learning-grade in structure, even if you’re using smaller instances for testing.

In my next blog post I will setup our kubernetes cluster created with kubeadm to be production grade with the following;

• HA control plane

• Load balancer

• External etcd

• Backup strategy

• Monitoring

• Logging

• Pod security standards

• RBAC hardening

• Ingress controller

• Certificate management

After which we’re all set to start deploying workloads.

• Deploy a voter application

• Set up monitoring (Prometheus, Grafana)

• Configure persistent storage

• Automate with GitOps or CI/CD

In this guide, we'll walkthrough how to set up DuckDNS with Nginx as a reverse proxy and secure it with Let's Encrypt SSL certificates.

Let's get started!!!

Prerequisites

Before we begin, ensure you have the following:

✅ An AWS EC2 instance running Linux (Amazon Linux, Ubuntu, or CentOS) with Jenkins installed.
✅ Nginx installed as a reverse proxy.
✅ A DuckDNS subdomain (e.g., jenkins-projectlab1.duckdns.org).
✅ A security group that allows inbound traffic on ports 8080 (Jenkins), 80 (HTTP) and 443 (HTTPS).

If you haven’t set up Jenkins yet. You will have to install OpenJDK 17 or newer which is required by Jenkins. For this tutorial I’ll launch an Amazon linux 2023 EC2 instance (t2 medium). You can use any instance you like by make sure you there is enough storage and memory (2 CPU minimum).

First install java:

sudo dnf install -y java-17-amazon-corretto
#verify installation
java -version

Now Proceed with Jenkins Installation

After installing Java, continue with these steps:

a. Add the Jenkins repository:

sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key

b. Install Jenkins:

sudo dnf install -y jenkins

c. Enable and start Jenkins:

sudo systemctl enable --now jenkins

#Check if Jenkins is running
sudo systemctl status jenkins

By default, Jenkins runs on http://your-ec2-ip:8080. Let’s make it accessible via https://jenkins-projectlab1.duckdns.org.

Step 1: Set Up DuckDNS for Your Domain

a. Go to DuckDNS and create an account if you don’t have one then sign in using your GitHub or Google account.
b. Under “subdomains” enter a name (e.g., jenkins-projectlab1) and click add domain.
c. Your full domain will be: jenkins-projectlab1.duckdns.org
d. Copy your EC2 Public IP from AWS then go to DuckDNS and update the IP field with your EC2 public IP.

Set Up Automatic Updates (Recommended)

Since EC2 instances often get dynamic IPs, use a script to update DuckDNS automatically, so you don’t have to manually copy the EC2 instance IP and paste in DuckDNS when the IP changes.

To write a script to automatically update DuckDNS with your EC2 instance IP follow the steps below:

a. Connect to your EC2 instance via SSH.

b. Create and cd into the directory**:**

mkdir -p ~/duckdns && cd ~/duckdns

c. Create an executable script to automatically update a new IP in DuckDNS when it changes:

nano duck.sh

d. Login into DuckDNS to get your token and domain name. Add the following lines (replace <YOUR-DUCKDNS-TOKEN> and <YOUR-DOMAIN\>) :

echo url="https://www.duckdns.org/update?domains=<YOUR-DOMAIN>&token=<YOUR-DUCKDNS-TOKEN>&ip=" | curl -k -o ~/duckdns/duck.log -K -

e. Save and exit.

f. Make the duck.sh file executable:

chmod +x duck.sh

g. Test the script:

./duck.sh

#Check duck.log to confirm the update:
cat duck.log

Automate Updates with Cron

Cron is a time-based job scheduler in Unix/Linux systems used to automate tasks at specified intervals. It runs scheduled commands or scripts without manual intervention.

First install Cron on your EC2 Instance:

#For Amazon Linux / Amazon Linux 2
sudo yum install cronie -y

#For Ubuntu/Debian:
sudo apt update && sudo apt install cron -y

#Start and Enable the Cron Service
sudo systemctl start crond
sudo systemctl enable crond

#Verify Cron is Running
systemctl status crond

Next, Open the cron editor:

crontab -e

Add your cron job (e.g., updating DuckDNS every 5 minutes or however you choose):

#This updates your IP every 5 minutes.
*/5 * * * * ~/duckdns/duck.sh >/dev/null 2>&1

Verify Your DuckDNS Domain

Open your browser and go to: http://jenkins-projectlab1.duckdns.org:8080

Note: Ensure your EC2 security group allows traffic on port 8080, 80 & 443 is for Jenkins, HTTP & HTTPS respectively.

Step 2: Install & Configure Nginx as a Reverse Proxy

A reverse proxy is a server that sits between client devices and backend servers, forwarding client requests to the appropriate backend service.

For a better understanding, a reverse proxy is like a middleman between users and a website or app. When you visit a website, instead of connecting directly to the main server, your request first goes through the reverse proxy. The proxy then forwards your request to the correct backend server and sends the response back to you.

Why Use a Reverse Proxy?

Security – Hides the real server, protecting it from attacks. Faster Loading – Stores and delivers frequently used content quickly. Load Balancing – Spreads traffic across multiple servers to prevent overload. HTTPS Support – Handles encryption to secure connections.

To access Jenkins over HTTPS using Nginx as a reserve proxy follow these steps.

First install Nginx:

sudo yum install -y nginx  # For Amazon Linux/CentOS
sudo apt install -y nginx  # For Ubuntu/Debian

#Start and enable Nginx
sudo systemctl enable --now nginx

Configure Nginx for Jenkins

Create a new Nginx configuration file:

sudo nano /etc/nginx/conf.d/jenkins.conf

Add the following content:

server {
    listen 80;
    server_name jenkins-projectlab1.duckdns.org;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Save and exit (‘:wq’ if using vim editor).

Test and restart Nginx:

sudo nginx -t
sudo systemctl restart nginx

After execution of sudo nginx -t command, you should get “syntax is ok” & “test is successful” confirmation on your terminal.

At this point, your Jenkins should be accessible via http://jenkins-projectlab1.duckdns.org.

Step 3: Secure Jenkins with ‘Let's Encrypt’ SSL

Now, let’s install Certbot to get a free SSL certificate so we can access Jenkins via HTTPS:

First, install Certbot:

#For Amazon Linux/CentOS:
sudo yum install -y certbot python3-certbot-nginx

#For Ubuntu/Debian
sudo apt install -y certbot python3-certbot-nginx

Generate an SSL Certificate

Next step, run the codes below to generate an SSL Certificate for our DuckDNS domain:

sudo certbot --nginx -d jenkins-projectlab1.duckdns.org

Follow the prompts:

• Enter your email address.

• Agree to the terms of service.

• Certbot will configure Nginx for HTTPS automatically.

If it doesn’t, manually update your Nginx config:

sudo nano /etc/nginx/conf.d/jenkins.conf

Replace with:

server {
    listen 80;
    server_name your-subdomain.duckdns.org;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name jenkins-projectlab1.duckdns.org;

    ssl_certificate /etc/letsencrypt/live/jenkins-projectlab1.duckdns.org/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/jenkins-projectlab1.duckdns.org/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Save and restart Nginx:

sudo nginx -t
sudo systemctl restart nginx

Note: Certbot will most likely configure Nginx for HTTPS automatically hence you may not need to update manually.

Step 4: Enable Auto-Renewal for SSL

Let's Encrypt certificates expire every 90 days. To renew them automatically, set up a cron job:

sudo crontab -e

Add this line at the bottom:

0 0 * * * certbot renew --quiet && systemctl reload nginx

This will renew the SSL certificate daily at midnight.

Final Test 🎉

Now, open your browser and visit https://jenkins-projectlab1.duckdns.org

Your Jenkins is now running securely over HTTPS! 🚀

Wrapping Up

By following this guide, you:

✅ Configured DuckDNS for dynamic DNS.
✅ Set up Nginx as a reverse proxy for Jenkins.
✅ Secured Jenkins with Let's Encrypt SSL.
✅ Enabled auto-renewal for SSL certificates.

Your Jenkins instance is now secure and accessible over HTTPS from anywhere, and you can use this process to assign a free domain to any IP or resource accessed via HTTPS.

Have questions? Let me know in the comments! 😊

Cloud computing is being adopted vastly by more than 90 percent of the fortune 500 companies with Amazon Web Services (AWS) getting a large slice of the pie while it continues on it's quest for world domination in the cloud computing space by a very wide margin. Amazon Web Services (AWS) offers a vast array of services that empowers developers to configure, build, deploy and manage applications seamlessly with second to none flexibility and scale. Among these services Amazon EC2 (Elastic Cloud Compute) is a frontrunner, providing resizable compute capacity in the cloud, which has revolutionised the way companies approach IT infrastructure. However, with great power comes the need for secure, efficient management tools, especially when it comes to accessing these virtual servers. Traditional methods, like SSH (Secure Shell), while powerful, come with their own set of challenges, including the management of SSH keys and the security risks associated with open ports.

Enter AWS Session Manager, a game-changer for developers and IT administrators looking for a more secure, streamlined way to access and manage their EC2 instances. This blog post will dive into the world of AWS Session Manager, exploring its benefits over traditional SSH access methods, such as eliminating the need for SSH keys, reducing the attack surface by keeping ports closed, and providing an audit trail of access and command execution. Whether you're a seasoned AWS user or new to cloud computing, this guide will walk you through everything you need to know about connecting to your AWS EC2 instance using AWS Session Manager, ensuring that you can leverage this powerful tool to enhance your cloud management strategy.

What is AWS Session Manager?

AWS Session Manager is a fully managed service provided by Amazon Web Services (AWS) that allows users to manage their Amazon Elastic Compute Cloud (EC2) instances, containers, and virtual machines (VMs) securely and efficiently. It's part of the AWS Systems Manager suite, which offers a unified interface for automating tasks across AWS resources.

The key functionality of Session Manager is to provide a secure browser-based interactive shell and command-line interface (CLI) access to your instances without the need to open inbound ports, manage SSH keys, or use bastion hosts. By eliminating the need for these traditional access mechanisms, Session Manager enhances security and simplifies the management of your compute resources.

Here are some of the notable features and benefits of using AWS Session Manager:

• Secure Access: It uses IAM (Identity and Access Management) roles and policies to control access to instances, ensuring that only authorized users can initiate sessions. All session data is encrypted, providing an additional layer of security.

• Simplified Management: With Session Manager, there's no need to manage SSH keys or SSL certificates. This simplification reduces the operational burden on teams and minimizes the risk of misconfiguration or unauthorized access.

• Audit and Compliance: All sessions can be logged and audited. Session Manager integrates with AWS CloudTrail and Amazon S3 to log session activity and store session output, aiding in compliance and audit processes.

• Seamless Integration: It integrates with other AWS services and features, like Amazon CloudWatch for logs, AWS S3 for session output storage, and AWS CloudTrail for audit trails. This integration allows for a cohesive and robust management ecosystem.

• Cost-Effective: As a feature of AWS Systems Manager, there are no additional costs for using Session Manager beyond the standard costs for Systems Manager and any associated resources, such as S3 for log storage or CloudWatch logs.

By leveraging AWS Session Manager, organisations can streamline their operations, enhance security, and comply with audit requirements, making it an essential tool for managing cloud resources efficiently and securely.

Prerequisites

To effectively use AWS Session Manager for managing your EC2 instances, you'll need to ensure that several prerequisites are in place. These prerequisites encompass AWS account setup, EC2 instance configuration, and necessary IAM role configurations. Here’s a comprehensive list to get you started:

1. AWS Account: You will need an active AWS account, if you don't one, you can create it by signing up on the AWS website. You'll need to have a beginner level knowledge of AWS.

2. AWS Systems Manager Agent (SSM Agent): The AWS Systems Manager Agent (SSM Agent) must be installed and running on the EC2 instances you want to manage. The SSM Agent facilitates communication between your instances and the Systems Manager, enabling features like Session Manager. Most Amazon Machine Images (AMIs) come with the SSM Agent pre-installed, but you should verify and update it to the latest version for the best performance and security. More details can be found here https://docs.aws.amazon.com/systems-manager/latest/userguide/ami-preinstalled-agent.html

3. IAM Role Configurations: Create an IAM role with permissions that allow the instance to communicate with the Systems Manager service. Attach the AmazonSSMManagedInstanceCore managed policy to this role. This step is crucial for the SSM Agent on your EC2 instances to function correctly. Ensure your IAM user account has the necessary permissions to access Systems Manager and initiate sessions. You can use managed policies like AmazonSSMFullAccess or create a custom policy that suits your security requirements.

4. Network Configuration: Your EC2 instances must have internet access to communicate with the AWS Systems Manager endpoint. This can be achieved through a public subnet with an internet gateway or through a private subnet with NAT gateway or VPC endpoint configuration. To configure, secure and facilitate access to your EC2 instance, it's essential to configure your instance's security group settings correctly. This involves adjusting the inbound traffic rules to allow connections through TCP protocol on three key ports: 443 for HTTPS, 80 for HTTP, and 8080 for custom TCP traffic. To ensure universal accessibility, set the source IP range to 0.0.0.0/0, which effectively permits any IP address across the globe to interact with your instance via these specified ports or set it to your IP for enhance security. This setup is crucial for enabling secure web access and supporting any specialised applications running on port 8080.

5. Operating System Compatibility: Ensure that your EC2 instances are running an operating system that is compatible with the SSM Agent. AWS provides a list of supported operating systems on their official documentation.

By ensuring these prerequisites are met, you'll be in a strong position to leverage AWS Session Manager for secure, efficient, and hassle-free management of your EC2 instances.

Setting up your EC2 Instance for Session Manager

The simple solution to configure session manager on your EC2 Instance starts with creating an IAM role for the Instance and attach the AmazonSSMManagedInstanceCore managed policy to this role.

Session manager tends to have some advantages that makes it highly encouraged and it's also a security best practice. With session manager agent installed on your EC2 instance there is no need for SSH keys which can be compromised. As a cloud computing expert, It's always best practice to apply the principles of least privileged when building/designing or configuring cloud infrastructure.

1. Launch an EC2 instance, for this blog post I will use Amazon Linux 2023 AMI & t2.micro instance type to eliminate instance cost which has SSM Agent preinstalled. Make sure to configure your network components correctly most especially the inbound traffic on your security group.

For testing purposes I assigned IPv4 on launch of my Instance and opened SSH port 22 to connect to my instance via instance connect and programmatically confirmed if SSM Agent is installed and running on Amazon Linux 2023 operating system.

We can confidently confirm that the SSM Agent is pre-installed and actively running as you can clearly see from above.

sudo systemctl status amazon-ssm-agent

The code above is used to verify the installation of the SSM Agent on Amazon Linux 2023. Refer to Amazon's official documentation to get a list of commands to verify the installation status of the SSM Agent of your instance's operating system type https://docs.aws.amazon.com/systems-manager/latest/userguide/ami-preinstalled-agent.html. Delete SSH inbound rule after confirmation before proceeding to the next step.

NOTE: There is no requirement to allocate an IPv4 address to your instance upon initialisation, and you can effectively establish a connection to your EC2 instance using Session Manager while operating within a private subnet.

Creating an IAM Role for the EC2 Instance

To utilize AWS Session Manager for securely connecting to your EC2 instances without requiring SSH keys, it's essential to have the appropriate IAM role attached to your instances. This guide will walk you through creating an IAM role with the AmazonSSMManagedInstanceCore managed policy, which grants the necessary permissions for your instances to communicate with AWS Systems Manager.

Step 1: Navigate to the IAM Console

• Log in to your AWS Management Console.

• Open the IAM console by finding the IAM (Identity and Access Management) service in the "Services" menu or by using the search bar.

Step 2: Create a New Role

• In the IAM dashboard, click on "Roles" on the left sidebar.

• Click the "Create role" button.

• Choose "AWS service" as the type of trusted entity.

• Select "EC2" under the “Common use cases” section, as this role will be used by EC2 instances. Then, click "Next: Permissions".

Step 3: Attach the AmazonSSMManagedInstanceCore Policy

• In the "Attach permissions policies" screen, use the search bar to find the AmazonSSMManagedInstanceCore policy.

• Check the box next to AmazonSSMManagedInstanceCore to select it.

• Click "Next: Tags" (optional step, you can add tags as per your organizational needs).

• Click "Next: Review".

Step 4: Review and Create the Role

• Assign a name to your role, such as "EC2SessionManagerRole". You can also add a description for clarity.

• Review the details to ensure the AmazonSSMManagedInstanceCore policy is attached.

• Click "Create role" to finalize the creation process.

Attach the Role to Your EC2 Instance

• Navigate to the EC2 dashboard from the AWS Management Console.

• Select the "Instances" link on the left sidebar to view your list of instances.

• Choose the instance you wish to attach the role to and select "Actions".

• Navigate to "Security" and select "Modify IAM role".

• In the "IAM role" dropdown menu, select the role you created earlier ("EC2-SSM-ACCESS-ROLE").

• Click to apply the changes.

Return to the EC2 dashboard and select the recently modified instance. Right-click on it, then choose the "Connect" option. If you attempt to initiate a session immediately, an error message may appear. This is normal, as it typically takes between 10 to 15 minutes for the newly attached IAM role to become active on your EC2 instance.

After successfully attaching the IAM role to the EC2 Instance, you will observe the following on your screen within the management console:

Upon clicking "Connect," you will have successfully established a connection with your EC2 instance via Session Manager. The display on your screen will then confirm that you've executed the steps correctly, leading you into a session where you can directly interact with your EC2 instance.

Conclusion

Your EC2 instance is now equipped with the necessary IAM role to facilitate secure connections via AWS Session Manager. This setup enhances your instance's security by eliminating the need for SSH keys and simplifies access management.

Remember to regularly review and update your IAM policies and roles to adhere to the principle of least privilege, ensuring that your AWS resources are accessible only as necessary.

Thank you for taking the time to read through this guide. I hope it has been informative and helpful in your journey to harness the power of AWS Session Manager for your EC2 instances. Stay tuned for more content like this, designed to empower you on your cloud learning journey.

Here's a brief overview of the process of building the monitoring app and what the app does:

In this blog post, we will be building a monitoring application using Python and Flask that can monitor the CPU and memory utilization of your computer. The application makes use of the psutil library to retrieve the CPU and memory utilization data.

We start by creating a basic Flask application that renders an HTML page with two gauges showing the CPU and memory utilization using Plotly.com. We use the psutil library to retrieve the CPU and memory utilization data and display it on the gauges. We also add a message to be displayed on the page if the CPU or memory utilization goes above 80%.

Next, we containerize the application using Docker. We create a Dockerfile that installs the required Python libraries and copies the application code to the container. We then build a Docker image from the Dockerfile and run a Docker container from the image.

We then deploy the application on Kubernetes using Amazon EKS. We create an ECR repository and push the Docker image to the repository. We then create an Amazon EKS cluster and nodes, and deploy the application on the cluster using a python boto3 library writing codes for kubernetes Deployment and Service.

Finally, we port forward and expose the Kubernetes application to our local machine, allowing us to access the monitoring application from a web browser.

Overall, the monitoring application allows us to monitor the CPU and memory utilization of our computer in a scalable and efficient manner, and with the use of Plotly.com, we can display the utilization data in a visually rich format.

Prerequisites

Before we begin, make sure you have the following:

• Python 3.10 or higher installed on your machine

• Flask web framework

• Docker installed on your machine

• AWS CLI installed on your machine

• You should have eksctl command-line tool installed on your machine

• Access to an AWS account

• You should have kubectl command-line tool installed on your machine

• Access to an EKS cluster

• VSCode installed

• Basic Docker and Kubernetes knowledge

Project Architecture

I have used LucidChart to design a sketch architectural diagram that represents what we intend to build. The diagram provides an overview of the system's components, their relationships, and how they fit together.

Step 1: Create the Flask application

Create a new Python file and name it app.py. Copy and paste the following code:

import psutil
# render - add style in flask application
from flask import Flask, render_template

app = Flask(__name__)


@app.route("/")
def index():
    cpuGauge = psutil.cpu_percent()
    memGauge = psutil.virtual_memory().percent
    Message = None
    if cpuGauge > 80 or memGauge > 80:
        Message = " High CPU or Memory utilization detected. Please scale up"
    return render_template("index.html", cpu_metric=cpuGauge, mem_metric=memGauge, message=Message)


if __name__ == "__main__":
    app.run(debug=True, host='0.0.0.0')

This code defines a Flask application that listens on the root URL / and returns the CPU and memory utilization of your computer in HTML rich format using render_template.

The psutil library is used to retrieve the CPU and memory utilization data. The CPU utilization is measured as a percentage using the cpu_percent method, and the memory utilization is also measured as a percentage using the virtual_memory().percent method.

If the CPU or memory utilization goes above 80%, a message is displayed on the page asking you to scale up.

Running the Application Locally

To run the application locally, save the code in a file called app.py and run it using the following command:

python3 app.py

This will start the Flask development server and you can access the application by opening a web browser and navigating to http://localhost:5000/

If you are using a MacBook you will encounter an error "Port 5000 already in use". This happens because the port 5000 we are using for localserver is being used by MacOS.

The solution to resolving this: Goto system preferences --> sharing --> uncheck Airplay Receiver. Airplay Receiver is using port 5000.

Run the server again you should be able to open http://localhost:5000/ on your browser:

Next: Create Folder 'templates' and a file inside the folder 'index.html' with the codes below:

<!DOCTYPE html>
<html>

<head>
    <title>System Monitoring</title>
    <script src="https://cdn.plot.ly/plotly-latest.min.js"></script>
    <style>
        .plotly-graph-div {
            margin: auto;
            width: 50%;
            background-color: rgba(151, 128, 128, 0.688);
            padding: 20px;
        }

        .alert {
            color: red;
        }
    </style>
</head>

<body>
    <div class="container">
        <h1>System Monitoring</h1>
        <div id="cpu-gauge"></div>
        <div id="mem-gauge"></div>
        {% if message %}
        <div class="alert">{{ message }}</div>
        {% endif %}
    </div>
    <script>
        var cpu_metric = {{ cpu_metric }}; // Render the dynamic value here
        var mem_metric = {{ mem_metric }}; // Render the dynamic value here

        var cpuGauge = {
            type: "indicator",
            mode: "gauge+number",
            value: cpu_metric,
            gauge: {
                axis: { range: [null, 100] },
                bar: { color: "#1f77b4" },
                bgcolor: "white",
                borderwidth: 2,
                bordercolor: "#ccc",
                steps: [
                    { range: [0, 50], color: "#d9f0a3" },
                    { range: [50, 85], color: "#ffeb84" },
                    { range: [85, 100], color: "#ff5f5f" }
                ],
                threshold: {
                    line: { color: "red", width: 4 },
                    thickness: 0.75,
                    value: cpu_metric,
                }
            }
        };

        var memGauge = {
            type: "indicator",
            mode: "gauge+number",
            value: mem_metric,
            gauge: {
                axis: { range: [null, 100] },
                bar: { color: "#1f77b4" },
                bgcolor: "white",
                borderwidth: 2,
                bordercolor: "#ccc",
                steps: [
                    { range: [0, 50], color: "#d9f0a3" },
                    { range: [50, 85], color: "#ffeb84" },
                    { range: [85, 100], color: "#ff5f5f" }
                ],
                threshold: {
                    line: { color: "red", width: 4 },
                    thickness: 0.75,
                    value: mem_metric
                }
            }
        };

        var cpuGaugeLayout = { title: "CPU Utilization" };
        var memGaugeLayout = { title: "Memory Utilization" };

        Plotly.newPlot('cpu-gauge', [cpuGauge], cpuGaugeLayout);
        Plotly.newPlot('mem-gauge', [memGauge], memGaugeLayout);
    </script>
</body>

</html>

After creating the file 'index.html' we should have a rich HTML visual of the CPU and Memory Utilization Metrics.

Run python3 app.py Open your browser you should see below:

Step 2: Containerize the application using Docker

Create a new file in the same directory as app.py name it requirements.txt. These are the dependency libraries and appropriate versions that are required to be used for the project. Copy and paste the following:

Flask==2.2.3
MarkupSafe==2.1.2
Werkzeug==2.2.3
itsdangerous==2.1.2
psutil==5.8.0
plotly==5.5.0
tenacity==8.0.1
boto3==1.9.148
kubernetes==10.0.1

Create a new file in the same directory as app.py and name it Dockerfile. Copy and paste the following code:

FROM python:3.9-slim-buster

WORKDIR /app

COPY requirements.txt .

# Update package manager and install required dependencies
RUN apt-get update && apt-get install -y gcc python3-dev

# Update pip
RUN python3 -m pip install --upgrade pip

#install the required python packages
RUN pip3 install --no-cache-dir -r requirements.txt

#copy the application code to the working directory
COPY . .

#set the environment variables for the flask app
ENV FLASK_RUN_HOST=0.0.0.0

# Expose the port on which the flask app wil run
EXPOSE 5000

# start the Flask app when the conainer is run
CMD [ "flask", "run"]

This Dockerfile defines the steps to build a Docker image for a Python application that uses Flask web framework. Here's what each instruction does:

• FROM python:3.9-slim-buster: This specifies the base image for the container. In this case, the base image is python:3.9-slim-buster.

• WORKDIR /app: This sets the working directory to /app in the container.

• COPY requirements.txt .: This copies the requirements.txt file from the local directory to the container's working directory.

• RUN apt-get update && apt-get install -y gcc python3-dev: This updates the package manager and installs required dependencies such as gcc and python3-dev.

• RUN python3 -m pip install --upgrade pip: This updates pip to the latest version.

• RUN pip3 install --no-cache-dir -r requirements.txt: This installs the Python packages specified in requirements.txt.

• COPY . .: This copies the application code from the local directory to the container's working directory.

• ENV FLASK_RUN_HOST=0.0.0.0: This sets the environment variable FLASK_RUN_HOST to 0.0.0.0, which allows the Flask app to listen on all network interfaces.

• EXPOSE 5000: This exposes port 5000 on the container, which is the port on which the Flask app will run.

• CMD [ "flask", "run"]: This specifies the command that will be executed when the container starts. In this case, it starts the Flask app using the command flask run.

To build the Docker image, run the following command in the same directory as the Dockerfile:

docker build -t my-flask-app .

This will build a Docker image with the tag my-flask-app.

To run the Docker container from the image, run the following command:

docker run -d -p 5000:5000 my-flask-app

This command starts a Docker container and maps port 5000 on the host to port 5000 on the container. The command '-d' also runs the container in a detached mode.

Open a web browser and navigate to http://localhost:5000 to see the render_template HTML output:

Step 3: Create ECR and push image to the Repo

Assuming you have an AWS account, you can use the AWS CLI to create an Amazon ECR repository and push the Docker image to the repository.

First, create an ECR repository by running the following command:

aws ecr create-repository --repository-name cloud-native-repo

The next step is to access the AWS management console and search for "ECR" in the search bar to navigate to the Elastic Container Registry. Once there, you can view the recently created repository.

To push the Docker image that was created on your local machine to the AWS ECR Repository, click on "View Push Commands" to get instructions on how to proceed. Follow the instructions to complete the image push.

OR Use the following steps to authenticate and push an image to your repository.

1. Retrieve an authentication token and authenticate your Docker client to your registry.

   Use the AWS CLI:

2.    aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin <your-aws-account-id>.dkr.ecr.us-east-1.amazonaws.com
   

   Note: if you receive an error using the AWS CLI, make sure that you have the latest version of the AWS CLI and Docker installed.

3. After the build is completed, tag your image so you can push the image to this repository:

    docker tag my-cloud-native-repo:latest <your-aws-account-id>.dkr.ecr.us-east-1.amazonaws.com/my-cloud-native-repo:latest
   

4. Run the following command to push this image to your newly created AWS repository:

    docker push <your-aws-account-id>.dkr.ecr.us-east-1.amazonaws.com/my-cloud-native-repo:latest
   

This command pushes the my-flask-app image to the ECR repository.

Step 4: Create EKS cluster and nodes

You can use the AWS CLI to create an Amazon EKS cluster and nodes.

First, create an Amazon EKS cluster by running the following command:

eksctl create cluster --name my-flask-app --region <your-aws-region> --nodegroup-name standard-workers --node-type t2.micro --nodes 2 --nodes-min 1 --nodes-max 3 --ssh-access --ssh-public-key <your-public-ssh-key>

Replace <your-aws-region> and <your-public-ssh-key> with the appropriate values for your AWS account.

This command creates a new EKS cluster named monitoring-app and launches two worker nodes of type t2.micro with minimum one node and maximum three nodes.

I have chosen to use an AWS free tier t2.micro instance type to minimize expenses. However, this instance type is equipped with only 1 CPU and 1GB memory, which may be inadequate for certain Kubernetes deployments to operate efficiently. If you encounter performance issues or errors, it is recommended to upgrade to an instance type with at least 2 CPUs and 2GB memory, such as the t3.small instance type. Please note that the specific instance type required for your deployment may vary depending on its requirements.

Overall, eksctl automates the process of creating and configuring the necessary AWS resources for an EKS cluster and node groups, including the required IAM roles and policies.

Step 5: Create Kubernetes Deployment and Service using Python

Now, create a Kubernetes Deployment and Service that use the my-flask-app Docker image.

Create a new file in the same directory as app.py and name it eck.py. Copy and paste the following code:

from kubernetes import client, config

# Load kubernetes configuration
config.load_kube_config()

# create a kubernetes API Client
api_client = client.ApiClient()

# Define the deployment
deployment = client.V1Deployment(
    metadata=client.V1ObjectMeta(name="my-flask-app"),
    spec=client.V1DeploymentSpec(
        replicas=3,  # set the desired number of replicas
        min_ready_seconds=30,  # set the minimum number of seconds for a pod to become ready
        strategy=client.V1DeploymentStrategy(
            type="RollingUpdate",
            rolling_update=client.V1RollingUpdateDeployment(
                max_unavailable=1,  # set the maximum number of unavailable pods during a rolling update
                max_surge=1  # set the maximum number of pods that can be created above the desired number of replicas during a rolling update
            )
        ),
        selector=client.V1LabelSelector(
            match_labels={"app": "my-flask-app"}
        ),
        template=client.V1PodTemplateSpec(
            metadata=client.V1ObjectMeta(
                labels={"app": "my-flask-app"}
            ),
            spec=client.V1PodSpec(
                containers=[
                    client.V1Container(
                        name="my-flask-container",
                        image="<your-aws-account-id>.dkr.ecr.us-east-1.amazonaws.com/my-cloud-native-repo:latest",
                        ports=[client.V1ContainerPort(container_port=5000)]
                    )
                ]
            )
        )
    )
)

# Create the deployment
api_instance = client.AppsV1Api(api_client)

api_instance.create_namespaced_deployment(
    namespace="default",
    body=deployment
)

# Define the service
service = client.V1Service(
    metadata=client.V1ObjectMeta(name="my-flask-service"),
    spec=client.V1ServiceSpec(
        selector={"app": "my-flask-app"},
        ports=[client.V1ServicePort(port=5000)]
    )
)

# Create the service
api_instance = client.CoreV1Api(api_client)
api_instance.create_namespaced_service(
    namespace="default",
    body=service
)

This Python code is for deploying a Flask application on a Kubernetes cluster.

Firstly, the code imports the required Kubernetes client and configuration modules from the Kubernetes Python library.

Then it loads the Kubernetes configuration using config.load_kube_config(). This step ensures that the code is using the correct Kubernetes cluster for deployment.

Next, the code defines a V1Deployment object that describes the deployment of the Flask application. It includes the desired number of replicas, minimum number of seconds for a pod to become ready, and the rolling update strategy. The deployment object also specifies the label selector, pod template, and container image for the application.

After defining the deployment object, the code creates the deployment by calling api_instance.create_namespaced_deployment() method of the AppsV1Api class, passing in the deployment object and the namespace where it should be created.

Finally, the code defines a V1Service object that describes how to access the Flask application. The service object specifies the label selector and the port where the Flask application is listening. The code creates the service object by calling the api_instance.create_namespaced_service() method of the CoreV1Api class, passing in the service object and the namespace where it should be created.

To apply these Kubernetes manifests, run the following command:

python3 eks.py

confirm if the kubernetes deployment and service was created

kubectl get deploy,svc

Step 6: Port forward and expose the Kubernetes application

To access the Kubernetes application from your local machine, you need to forward the Kubernetes Service port to your local machine.

To do this, run the following command:

kubectl port-forward service/my-flask-service 5000:5000

This command forwards port 5000 of the Kubernetes Service named my-flask-service to port 5000 on your local machine.

You can now access the my-flask-app application by opening a web browser and navigating to http://localhost:5000/.

Cleanup

To delete the EKS cluster and associated resources created by eksctl to avoid incurring charges, you can use the following command:

eksctl delete cluster --name <cluster-name> --region <your-aws-region>

Replace <cluster-name> with the name of the EKS cluster you want to delete, and <your-aws-region> with the AWS region where the cluster is located.

This command will delete the CloudFormation stack that was created by eksctl, including the control plane, worker nodes, and networking resources. It will also delete the IAM roles and policies created for the EKS cluster and worker nodes.

Note that deleting an EKS cluster and its associated resources is an irreversible operation, and you will lose all data stored on the worker nodes. Before running the eksctl delete cluster command, make sure you have backed up any data that you want to keep and that you have terminated any services or applications running on the cluster.

Conclusion

In this blog post, you learned how to build a cloud-native monitoring application using Python and Flask, monitor CPU and memory utilization of your computer, containerize the application using Docker, deploy the application on Kubernetes using Amazon EKS, and port forward and expose the Kubernetes application to your local machine.

By following these steps, you can create a scalable and efficient monitoring application that can be run in the cloud. You can also extend the application to monitor other aspects of your system, such as network usage, disk usage, and more.

I hope this blog post was helpful to you in building your own cloud-native monitoring application.

Before we begin, it's important to understand the legal and ethical considerations when web scraping. Be sure to check the website's terms of service to ensure that web scraping is allowed, and be respectful of the website's resources and bandwidth by not overloading their server with requests. Additionally, some websites may have anti-scraping measures in place, so proceed with caution and avoid any actions that may be seen as malicious or harmful.

You can check by inspecting the robots.txt file of the website 'https://example.com/robots.txt'

Now, let's dive into the code.

Let's walk through the process of setting up MongoDB and Mongo Express using Docker.

Prerequisites

Before we begin, you will need to have Docker installed on your machine. You can download and install Docker from the official website: https://www.docker.com/products/docker-desktop.

First, let's create a Docker Network which will be used by both MongoDB and Mongo Express to enable communication between them. Then we test to see if the network has been created successfully.

docker network create mongo-network
docker network ls

Next, let's create a Docker container for MongoDB. To do this, we can run the following command:

docker run -d --name mongodb --network mongo-network -p 27017:27017 mongo

This command will download the latest MongoDB image from Docker Hub and start a container named 'mongo' while a network is connected to it. We are also mapping port 27017 from the container to port 27017 on our host machine.

You can verify that the container is running by running the command:

docker ps

This should list all running containers, and you should see the 'mongo' container listed.

docker run -d \
    --network mongo-network \
    --name mongo-express \
    -p 8081:8081 \
    -e ME_CONFIG_MONGODB_SERVER="mongodb" \
    -e ME_CONFIG_BASICAUTH_USERNAME="admin" \
    -e ME_CONFIG_BASICAUTH_PASSWORD="1234567890" \
    mongo-express

This command will download the latest Mongo Express image from Docker Hub and start a container named 'mongo-express'. We are also mapping port 8081 from the container to port 8081 on our host machine.

The environment variables ME_CONFIG_MONGODB_SERVER, ME_CONFIG_BASICAUTH_USERNAME, and ME_CONFIG_BASICAUTH_PASSWORD are used to configure Mongo Express to connect to our MongoDB container and set up basic authentication.

Accessing Mongo Express

Once the container is running, you can access Mongo Express by opening a web browser and navigating to http://localhost:8081. You should see a login screen where you can enter the username and password you set up in the previous step.

After logging in, you should see a dashboard where you can view and manage your MongoDB databases.

We have three options to create a database. First, you can create it from the mongo-express UI by simply clicking 'create database'. Secondly, you can write python code. Thirdly, you can create it on the command line or terminal with the following commands:

docker exec -it mongodb mongosh

The above command is used to interactively login into the MongoDB container. The command mongosh is the mongo interactive shell.

Next, we will create a DataBase and a Collection where our data will be stored, using the following commands:

#create a database named web_scraper_db.
use web_scraper_db 

#create new user
db.createUser({ user: "User1", pwd: "1234567890", roles: [{ role: "readWrite", db: "web_scraper_db" }] })   

#create a collection named web_scraper_collection.
db.createCollection('web_scraper_collection') 

#add a document in the web_scraper_collection colection.
db.web_scraper_collection.insert({item: "apple", cost: 3}) 

#add a document in the web_scraper_collection colection.
db.web_scraper_collection.insert({item: "pear", qty: 5}) 

#select documents in the web_scraper_collection collection. It should display the two created documents with the _id property.
db.web_scraper_collection.find(): 

show users #list all users.
exit #exit.

#log back into mongo container with a specific user
docker exec -it mongodb mongosh --username User1 --password 1234567890 --authenticationDatabase web_scraper_db 

#check the current user.
db.runCommand({connectionStatus : 1}) 

#confirm if User1 can query documents.
db.web_scraper_collection.find() 

exit #exit the mongodb shell.

You can check out mongodb.com for additional info on more MongoDB commands.

Next step we write some python code to bring our idea to life.

Importing Libraries

The first step is to import the necessary libraries. In this case, we will need requests, BeautifulSoup, and pymongo

import requests
from bs4 import BeautifulSoup
from pymongo import MongoClient

Setting up the MongoDB Connection

Next, we need to set up a connection to the MongoDB database where we will be storing our scraped data. We will use the MongoClient class to connect to the database and create a collection to store our data.

MONGO_DB_URL = 'mongodb://User1:1234567890@localhost:27017/web_scraper_db'
client = MongoClient(MONGO_DB_URL)
db = client['web_scraper_db']
collection = db['web_scraper_collection']

Here, we are connecting to a local MongoDB instance with the URL 'mongodb://User1:1234567890@localhost:27017/web_scraper_db' You will need to modify this URL to match your own MongoDB instance. We are then creating a database called web_scraper_db and a collection called web_scraper_collection

Scraping the Website

Now that we have our MongoDB connection set up, we can begin scraping the website. In this example, we will be scraping the website https://books.toscrape.com/ to extract information about books.

URL = 'https://books.toscrape.com/'
response = requests.get(URL)
soup = BeautifulSoup(response.content, 'html.parser')

Here, we are using the requests library to send a GET request to the website and retrieve the HTML content. We then use BeautifulSoup to parse the HTML content and create a BeautifulSoup object that we can use to extract information.

books = soup.find_all('article', class_='product_pod')
book_list = []
for book in books:
    title = book.h3.a.attrs['title']
    link = book.h3.a.attrs['href']
    price = book.select_one('p.price_color').get_text()
    book_list.append({'title': title, 'link': link, 'price': price})

In this section of the code, we are using BeautifulSoup to extract information about the books on the website. We are searching for all HTML elements with the tag 'article' and the class 'product_pod', which represents the individual books on the website. We then iterate over each book and extract the title, link, and price information. We store this information in a list of dictionaries called book_list.

Inserting the Data into MongoDB

Now that we have our data in the book_list variable, we can insert it into our MongoDB collection.

collection.insert_many(book_list)

Here, we are using the insert_many method to insert all the books in the book_list variable into the MongoDB collection.

num_books = collection.count_documents({})
print(f"Total number of books in the collection: {num_books}")

Save the python codes in a file web_scrapper.py You can run the python script to execute the commands. All things being equal you should get the results below:

Next, let's log in to mongo express using the URL http://localhost:8081 Once logged in click the database created earlier, then click the collection we created which will lead to a page where we can visualize the data scrapped from 'https://books.toscrape.com/'

You can log into the MongoDB container created earlier to query the database with the commands I provided earlier to see the data we saved in it.

docker exec -it mongodb mongosh

#mongodb commands to view web scrape data 
show dbs
use web_scraper_db
show collections
db.web_scraper_collection.find()

You should get the results below

Conclusion

In this blog, we have covered how to set up MongoDB and Mongo Express using Docker, allowing us to easily create and manage MongoDB databases and access a visual interface for managing our data. Then we wrote a python code to run the command to web scrape the website. With these tools, we can easily create web scrapers like the one we discussed earlier and store the data in our MongoDB databases.

Hope you enjoyed this TUTORIAL !!! Good Luck and Happy coding !!!

Serverless Architecture

Just in case you are a beginner in cloud, I will explain the term 'SERVERLESS' so you can have a brief understanding of what it means.

Serverless architecture is a cloud computing model where a third-party provider manages the infrastructure and handles server operations automatically. This means that developers do not have to manage servers or infrastructure themselves, and they can focus on writing code and building applications.

In a serverless architecture, the cloud provider automatically provisions and manages the server resources needed to run the code, and bills the developer based on the actual usage of resources, rather than a fixed amount. This can lead to cost savings, as the developer only pays for what is used.

Serverless architectures typically involve writing small, single-purpose functions that are triggered by events, such as a user uploading a file or a message being sent to a queue. These functions are run on demand and scaled automatically based on the number of requests received.

Serverless from the name doesn't mean that servers are not involved, it means the cloud developer shouldn't worry about provisioning, deploying and maintaining servers. Just focus on the codes that will be used for deployment. The best part is that, in a serverless architecture, you are billed based on the actual usage of resources rather than a fixed amount. This means that you are only charged for the number of computing resources and time that your code uses.

For example, if you create a function that runs for 100 milliseconds and is triggered 100 times in a month, you will only be billed for the total time that your function runs, which would be 10 seconds (100 x 0.1 seconds). The exact pricing model varies by cloud provider, but generally, you will be charged based on the number of requests, the duration of each request, and the amount of memory used.

I will be utilizing Infrastructure as Code tools, such as Serverless Framework and AWS Serverless Application Model (SAM), throughout this article. By doing so, deploying the entire architecture on AWS will only require a few simple commands. Before we dive in, let's take a moment to familiarize ourselves with Infrastructure as Code (IaC), AWS SAM (Serverless Application Model) and CRUD REST API.

Infrastructure as code (IaC)

Infrastructure as Code (IaC) is the practice of managing and provisioning technology infrastructure using machine-readable definition files, rather than manual configuration. This approach enables teams to automate the process of deploying and managing infrastructure, resulting in faster and more consistent deployments.

At a low level, IaC involves writing code to define the resources and configurations needed for a particular infrastructure, such as servers, databases, load balancers, and other components. The code is typically written using a domain-specific language (DSL) that is specific to the infrastructure provider, such as AWS CloudFormation, Terraform, or Azure Resource Manager.

AWS SAM (Serverless Application Model)

AWS Serverless Application Model (SAM) is an open-source framework for building serverless applications using AWS services. SAM extends AWS CloudFormation to provide a simplified way of defining the Amazon API Gateway APIs, AWS Lambda functions, and Amazon DynamoDB tables needed by your serverless application. SAM also provides tools for locally testing and debugging serverless applications, as well as deploying them to AWS. With SAM, developers can quickly and easily build and deploy serverless applications on AWS.

CRUD REST API

A CRUD REST API is a RESTful API that allows for Create, Read, Update, and Delete operations on resources through HTTP methods. REST stands for Representational State Transfer, and is a set of architectural principles for designing web applications.

In a CRUD REST API, each resource is identified by a unique URL, and the HTTP methods are used to perform operations on the resource. The HTTP methods used are:

• POST: used to create a new resource

• GET: used to retrieve an existing resource or a collection of resources

• PUT: used to update an existing resource

• DELETE: used to delete an existing resource

Now we have had some brief explanations of the technologies we will use. It's about time we use Serverless Framework and AWS SAM (Serverless Application Model) to deploy our application on AWS using VSCode as our cloud development environment (CDE).

AWS SAM will be creating a Lambda function, DynamoDB table and API Gateway (REST API) using our CDE VSCode. We will be issuing some commands on our terminal to deploy mentioned resources.

Prerequisites

Before we get started, you'll need the following:

• An AWS account

• VSCode and the AWS Toolkit extension installed

• The AWS SAM CLI installed

• Basic knowledge of Node.js and the AWS SDK

Confirm installation of AWS CLI and AWS SAM using commands:

$ aws --version
-> aws-cli/1.27.32 Python/3.10.10 Darwin/21.6.0 botocore/1.29.32
$ sam --version
->SAM CLI, version 1.66.0

Creating an AWS SAM project

First, we'll create a new AWS SAM project using the sam init command. Open a terminal window on VSCode and navigate to a directory where you want to create the project. You will get a prompt like below:

Choose the options from the prompt as below:

You should get these files automatically added by AWS SAM. You can inspect the files to have a better understanding of what SAM is building.

Next Step: Input sam build in your terminal which will build and package serverless applications before they are deployed to AWS.

Next command: sam deploy --guided

"Sam deploy --guided" is a command used in the AWS Serverless Application Model (SAM) that initiates a guided deployment process for your serverless application. When you run this command, SAM will prompt you to answer a series of questions about your deployment, such as the AWS Region to deploy to, the name of the stack to create, and any input parameters required by the stack.

This command sam deploy --guided provides a step-by-step guided experience that helps to ensure that your deployment is configured correctly and follows best practices. It also helps to identify and resolve any potential issues or errors before the deployment process begins.

After SAM has deployed all the resources required for the CRUD REST API, you will see the endpoint URL below. Copy and put in a text file as we will be testing it later.

Confirmation of SAM deployment

Log into the AWS Management Console, and search for DynamoDB, API Gateway, CloudFormation and Lambda one after the other to investigate if it was deployed successfully.

The YAML template file generated by the sam init command in AWS Serverless Application Model (SAM) is used to deploy your serverless application using AWS CloudFormation. if you inspect CloudFormation you will see the template.yaml SAM generated was used to deploy our resources in AWS.

TESTING THE CRUD API

There are several API testers, but for this tutorial, the Talend API tester will be used.

Talend API Tester is a tool that is designed to test the API of web services and RESTful services. With this tool, developers and testers can simulate API calls, generate test cases, and perform functional and performance testing of APIs.

Talend API tester is a Chrome extension that can be added to your Chrome browser. You can get it from the Chrome store.

Open the Talend extension and input the endpoint URL you saved in a text file earlier, select the 'GET' method, then click send. You should get an empty string ' [ ] ' the simple reason being that DynamoDB is empty.

To populate DynamoDB we will use the 'POST' method, then input some code in BODY. You should get a 200 response confirmation. After you click send, you can change the values in the body to add more items to DynamoDB.

If the 'POST' Method worked perfectly, change to the 'GET' Method to confirm if we successfully added items to DynamoDB. You should get a 200 response.

Goto DynamoDB to confirm

Now we can confirm that we have successfully built a CRUD RESTful Microservice API using AWS SAM. This method saves a lot of time, code debugging and troubleshooting. To understand this fully, you have to configure or build it manually which I will do in my next blog post.

ADDING API KEYS

Add API Keys that will restrict and grant access to selected users.

To add API keys to your CRUD API on AWS API Gateway, you can follow these steps:

1. Create an API key: In the AWS Management Console, navigate to the API Gateway service and select "API Keys" from the left-hand menu. Click on "Create API Key" and follow the prompts to create a new API key.

2. Enable API key usage: In the same "API Keys" menu, select "Usage Plans" and create a new usage plan if you haven't already. Select the usage plan and click on "Add API Key" to add the API key you just created to the plan.

3. Deploy the API: To deploy your API, first click on 'Resources' in the left-hand menu. Then, click on 'GET' and select 'Method Request'. Set 'API Key Required' to 'True'. To deploy the API, click on 'Actions' and select 'Deploy API'. Make sure to deploy the API whenever you make any changes for the changes to take effect.

4. Test the API: Test the URL on your browser to confirm that it requires API Key for access. If you Test on Talend API tester you will get a 403 error response.

To add our API keys and enable access, click on 'API Keys' in the left-hand menu of the API created by SAM. Then, click on the previously created key name and click on 'Show' next to 'API Key'. Copy the key into a text file and save it for later.

Next, go to Talend API Tester, select the 'GET' method and input the endpoint URL. Scroll down one step to the headers section and input the default header for API Keys, X-Api-Key and paste in the API key as the value. Finally, click 'Send'. You should receive a 200 response.

MONITORING API CALLS

Monitoring API calls using CloudWatch involves configuring CloudWatch to capture and log relevant API calls data, such as API Gateway requests, Lambda function invocations, and other metrics.

To view CloudWatch logs, open the Lambda function created by SAM, click "Monitor", then "Logs". Select the logstream of the recent invocations, which will take you to CloudWatch to view logs.

From here, you can use CloudWatch to analyze and visualize your API call data. For example, you can create CloudWatch dashboards to track key metrics, set up CloudWatch alarms to alert you when specific metrics cross a threshold, and use CloudWatch Insights to perform ad-hoc queries and analysis on your log data.

In addition to CloudWatch, AWS provides other tools for monitoring and analyzing your API calls, such as AWS X-Ray and Amazon CloudTrail. AWS X-Ray is a service that lets you trace requests made to your API and identify performance bottlenecks or issues. Amazon CloudTrail is a service that records AWS API calls and events and provides visibility into user and resource activity.

By using CloudWatch, AWS X-Ray, and Amazon CloudTrail together, you can gain comprehensive visibility into your API calls and ensure that your application is running optimally and securely.

CLEANUP

You need to clean up your AWS account to avoid incurring unnecessary fees.

You can clean up in VSCode using either of the codes below:

$ sam destroy

$ aws cloudformation delete-stack --stack-name <stack-name> --region <aws-region>

'sam destroy' is a command in the AWS SAM CLI that removes the AWS CloudFormation stack associated with your SAM application. This command deletes all resources that were created when the stack was created, including the API Gateway, Lambda functions, DynamoDB tables, and any other resources defined in the CloudFormation template.

The command "AWS CloudFormation delete-stack" is used to delete an existing stack. The "<stack-name>" parameter should be replaced with the name of the stack you want to delete, and the "<aws-region>" parameter should be replaced with the AWS region where the stack is located.

By running this command, the AWS CloudFormation service will remove all resources that were created by the stack, and the stack will be deleted. Note that this is an irreversible action, and once a stack is deleted, all associated resources will be terminated and cannot be recovered. Therefore, it is important to use caution when using this command and ensure that you are deleting the correct stack before proceeding.

SUMMARY

In conclusion, deploying a CRUD RESTful API using AWS SAM is a straightforward process that allows you to quickly and easily build and deploy your API. With AWS SAM, you can easily manage your serverless applications and scale your infrastructure as your needs grow.

The process involves designing and creating the API using an API design tool and AWS SAM, developing and testing the API locally with the AWS SAM CLI, packaging and deploying the application using AWS CloudFormation, testing the deployed API using a tool like Postman or Talend API Tester, and monitoring and troubleshooting the API using AWS CloudWatch.

By following the steps outlined in this post, you can get your API up and running in no time. So, whether you're building a new API or migrating an existing one, AWS SAM is a powerful tool that can help you get the job done quickly and efficiently.

Hope you enjoyed the tutorial. Good luck and happy coding...