Categories
Technically Speaking

UPchieve remote tutoring platform deployment

With the ongoing Covid19 situation, platforms that specialize in remote learning and education distribution, have become invaluable. UPchieve is such an open source platform. This platform connects a volunteering tutor and a student to connect and learn. The communication methods also include an interactive whiteboard and audio calling.

UPchieve also has iOS and Android apps, built on the React Native platform.

In a previous Technically Speaking installment, we outlined the steps needed to upload a Node.JS web app onto the Google Compute Engine. The NodeJS app used as the sample is the UPchieve/web app. The procedures for deploying the web app can be found in a previous Technically Speaking installment. In this article, we talk about how the server portion of the platform can be deployed on the Google Compute Engine.

This NodeJS stack comes with a twist – we will be using nginx as a reverse proxy server for NodeJS. This extra server setup has an advantage – NodeJS itself will run without requiring root permissions. Instead, nginx will handle http/s access for us and route requests locally to the NodeJS server.

As requirements, we are assuming that you have already created a new virtual instance on the Google Compute Engine.

Procedure

Install Applications

Initially, a VM would not have any software. We need to populate it with the software that fits our needs. For our scenario, we are going to need these software:

  • git
  • NodeJS
  • MongoDB
  • make
  • certbot (for SSL connectivity)
  • nginx

The commands to install these tools on a Debian 9 system are given below:

# Install git
sudo apt-get install git

# To install nodeJS, we need to install curl
# https://tecadmin.net/install-latest-nodejs-npm-on-debian/
sudo apt-get install curl software-properties-common
curl -sL https://deb.nodesource.com/setup_13.x | sudo bash -
sudo apt-get install nodejs

# Install mongodb
#Update 02Jul20 : Run `sudo apt-get install wget` if wget is not there
wget -qO - https://www.mongodb.org/static/pgp/server-4.2.asc | sudo apt-key add -
echo "deb http://repo.mongodb.org/apt/debian stretch/mongodb-org/4.2 main" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.2.list
sudo apt-get update
sudo apt-get install -y mongodb-org

# Install make
sudo apt-get update
sudo apt-get install build-essential

# Install nginx
sudo apt update
sudo apt install nginx
systemctl enable nginx # start the nginx service

To install some of the tools on macOS (maybe as a development environment):

brew tap mongodb/brew
brew install [email protected]

Clone the Repository

We need to pull the source from the remote GitHub server and store it on our server locally. The open source UPchieve server source can be pulled from its GitHub repository.

# Clones the repository onto a folder named 'server'
git clone https://github.com/UPchieve/server.git server
cd server

Starting Applications

Next, we will start the servers necessary for our UPchieve server to perform its first run/ setup. First, we will start the MongoDB background service.

For Linux systems:

sudo systemctl daemon-reload
sudo systemctl start mongod
systemctl status mongod # check if the service works

If the service has been set up successfully, you will see an output similar to this:

UPchieve requires MongoDB service to run properly

For macOS:

brew services stop [email protected]
brew services start [email protected]

mongod --config /usr/local/etc/mongod.conf --fork

# check if MongoDB is running
ps aux | grep -v grep | grep mongod

Next, we need to set up the UPchieve server and databases.

# setup database and install dependencies
cd server
bash bin/setup # if there is an error, run npm rebuild
node init
npm run dev # start upchieve server
# if you get a New Relic error, run
# cp node_modules/newrelic/newrelic.js newrelic.js
# if you get a bcrypt error, run `npm rebuild`
# if you still get the bcrypt error, run `npm install bcrypt`

You should be able to check if the server is working at this point. Open your browser and open the page at

http://<VM IP Address>:3000/eligibility/school/search?q=test

If it works, you might want to open a new shell (the current shell will be running the node server) to execute the other commands.

Production-ready!

We need a few more changes to make the application server ready for production. At the moment, we need to type in the IP address and the port number to access the server. Although this should also be fine since the consumers of the application would not have to access the server directly, this is not recommended. Besides, there is no SSL facilities on the server.

To set up free SSL using Let’s Encrypt SSL, you can refer to our previous article that outlines the procedure also for the Google Compute Engine.

Configure nginx

We will use nginx as a reverse proxy for our NodeJS server. Assuming that nginx is already installed, we need to configure it.

sudo nano /etc/nginx/nginx.conf

Add a server in this file to listen on port 80 (http port). We do this by adding an entry inside the http block (make sure to replace server.example.com by your sever name):

	server {
		listen 80;
		server_name server.example.com;

		location / {
			proxy_set_header   X-Forwarded-For $remote_addr;
			proxy_set_header   Host $http_host;
			proxy_pass         http://localhost:3000;
		}
	}

Now you can test the server using the domain name instead of the IP address and port number

http://<DOMAIN NAME>/eligibility/school/search?q=test

Configure SSL

Before starting configuration of SSL, we might have to stop both our NodeJS and nginx servers

sudo systemctl stop nginx
ps aux | grep -i node # to find our node processes and PIDs
kill -9 <PID> # here PID is the ID of the node process

Since we have nginx running as a proxy, the usage of slightly different and tailored for an nginx environment:

sudo apt-get install certbot python-certbot-nginx
sudo certbot --nginx # automates the editing of nginx configuration file

sudo systemctl start nginx # start nginx service
cd server
npm run dev # start our NodeJS server

Depending on the selections you made during the SSL configuration, you would be able to access the server on both http and https at this point.

http://<DOMAIN NAME>/eligibility/school/search?q=test
https://<DOMAIN NAME>/eligibility/school/search?q=test

Configure Socket.io

The socket.io protocol is used by the server to trigger request notifications on the volunteer dashboard and the session chat system. This should not be confused with WebSockets. These are two different protocols.We will be configuring WebSockets separately.

By default, the NodeJS server is listening to socket.io based requests port 3001. But we need to route it through our nginx server if we are to enable SSL for socket.io requests.

Our game plan to cover all these grounds is to:

  1. Add a destination server on port 3001 to our NodeJS server (which will be http://localhost:3001)
  2. Add a reverse proxy for the location /socket.io/ – this specific location is defined by the socket.io protocol. This proxy will take care of other socket.io requirements such as http upgrade
  3. Publicize a SSL supported port, 3002, that can be accessed externally by our web app

We cannot use 3001 or 3000 in place of the 3002 without changing any NodeJS config code as this is what the NodeJS server itself is listening on. Instead, we define an unused port 3002.

http {
  upstream upstream-nodejs { # NodeJS socket.io destination
    server  127.0.0.1:3001; # OR localhost:3001
  }
  # other stuff...
  # Add SSL support to port 3002 which will be publicized
  listen 443 ssl; listen 3002 ssl; # managed by Certbot
  # Other SSL properties...
  location /socket.io/ {
    # listen for this location on port 3002
    proxy_pass              http://upstream-nodejs;
    proxy_redirect off;
    
    proxy_http_version      1.1;
    
    proxy_set_header        Upgrade                 $http_upgrade;
    proxy_set_header        Connection              "upgrade";
    
    proxy_set_header        Host                    $host;
    proxy_set_header        X-Real-IP               $remote_addr;
    proxy_set_header        X-Forwarded-For         $proxy_add_x_forwarded_for;
  }
  
    

Configure Websockets

WebSockets also requires the http upgrade method. In contrast to the socket.io method we followed, we use a map symbol to describe this.

Next, we specify the URI that we should listen for WebSocket requests is /whiteboard/. Requests to this URI will be automatically referred to the back-end WebSockets server on NodeJS.

The specialty with this NodeJS server is that it is listening on port 3000. If you remember, this is also the port NodeJS processes normal HTTP requests. The difference is the URI. Only requests sent to /whiteboard/ will be upgraded to WebSocket protocol. This is quite sensible as WebSockets operates on the same ports as HTTP/S, 80 and 443.

http {
  map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
  }
  
  upstream websocket {
    server localhost:3000;
  }
  
  # other stuff...
  
  location /socket.io/ {
    # socket.io stuff
  }
  # add right after socket.io location definition
  location /whiteboard/ { # this URI is where WebSockets is used
      proxy_pass http://websocket;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "Upgrade";
      proxy_set_header Host $host;
  }
}

Configure Database Backups

A method to backup and restore is crucial when it comes to production environments. MongoDB has built-in functions do carry out these procedures.

To backup the entire database:

# Install `zip` utility to compress the exported folder for download
sudo apt install zip unzip

sudo mongodump -d upchieve -o home/backups/
# this will create a backup directory (upchieve) with JSON or BSON of the collections
cd /home/backups
sudo zip -r db_29Jul20.zip upchieve

The resultant zip file can be downloaded from the SSH terminal by clicking on ‘Download File’ in the cog icon.

To restore the entire database, the zip file should be extracted to show the upchieve folder. Then;

mongorestore -d upchieve upchieve

UPchieve Server Gotchas

Git ignores certain configuration files like config.js. Therefore if you need to update them, you have to edit the copy on the production server. Updating on the git repository will have no effect.

That has been all for today’s Technically Speaking discussion. Although we focused specifically on UPchieve, we hope this document will act as a summary of setting up any server based on a similar stack (nginx and NodeJS). Hope you will use your newly learnt knowledge for improving the accessibility of education in the current times. Don’t hesitate to leave any comments or suggestions below!

Categories
Technically Speaking

Using Google Compute Engine to run a Node.js app

In this new installment of Technically Speaking, we bring to you another article talking about the features of the Google Cloud platform. This article would walk you through the procedure for setting up a Node.js server app from a GitHub repository. Then we are going to deploy it on a Google Compute Engine instance.

As the GitHub repository, we are going to select UPchieve/web. This is the front-end web app portion of a homework help platform called UPchieve. We think this is a highly relevant platform to know about in the current day. Platforms that encourage remote learning are becoming increasingly important in lieu of the global pandemic. We are hoping to release an article detailing the deployment of the server portion. So, you or your educational institute can continue delivery of material.

Requirements

You will be expected to create the virtual machine instance on the Google Compute Engine dashboard which is a straightforward and very quick procedure. The walk through is using a VM instance running Debian 9 (OS can be selected during the creation process). But the idea should be the same regardless of the paltform.

Procedure

First, you need to access an SSH connection to the VM instance. There are several ways to do it but we will go the default way and select a browser-based SSH terminal. This can be accessed by going to the VM Instances page and clicking on the SSH button as shown in the screenshot.

Screenshot of the VM Instances page showing the location of the SSH button

If you have more than one instance, make sure you select the correct one!

Installing Git

This is the first thing we have to do if we are going to deal with GitHub repositories. The method would be quite familiar for the average Linux user:

sudo apt update
sudo apt install git
git --version

The last command is of course to verify that the installation was successful.

Installing Node.js

To install Node.js, we need to add the PPA to our system before installation. To do this, we need to first install curl.

sudo apt-get install curl software-properties-common
curl -sL https://deb.nodesource.com/setup_13.x | sudo bash -
sudo apt-get install nodejs

Make sure you replace the version number with the version number you want (preferably the latest version) before you execute the command.

Cloning the repository

We will clone the repo onto the instance. This is done using:

git clone https://github.com/UPchieve/web.git
cd web

If you plan on on cloning from a private repository, use the same command. The difference will be that you will be prompted for the GitHub login details.

Installing Dependencies

Next, we need to install the app’s dependencies. This simply means that we are going to create and populate the app’s node_modules folder. All of this is automated (thank God!) and can be executed by running;

npm install

Deployment

You are ready to run the Node.js app! To run a development build, simply run;

npm run dev

If you got to this point in the procedure, you should be able to at the least, run the app on the localhost.

Configuring the Firewall

To allow external access to our VM instance, so that anyone can view our awesome app, wee need to configure the firewall. By default, the Google Compute Engine restricts external access to the instance.

To do this, go to the Network interfaces detail of the VPC Network Compute section by clicking on the following menu option from the VM Instances page:

Click on the Firewall rules option in the right hand side drawer. Then click on Create Firewall rule button on the top.

Give any name for the rule you want to. And make sure the other options match the options as per the screenshot:

Google Compute Engine add firewall page
The options to select for our firewall

You should be able to access the Node.js app using <externalIp>:8080 where external IP mentioned on the VM Instances page. If it still does not work, make sure that both these options are unchecked on the VM Instances details page:

If they are not, click on the Edit button at the top to edit the details.

Configuring the App

Depending on the app you choose to deploy, there might be certain configuration options to modify. In the case of the UPchieve/web app, we need to point it to the server of our UPchieve server. To do this, we need to edit the Environment files which are used to provide the server location…etc.

vim .env.development

The command above can be run to edit the environment variables used when running in the development mode. For production mode, please use .env.production instead. If you are new to vim, you can watch this crash course in vim.

Edit the file so that it points to the server you want.

You can run the development build again and see the changes. My deployed app has an end-result that looks a little like…this

UPchieve deployed on the Google Compute Engine
UPchieve web app up and running!

Deploying a Production Build

The UPchieve web app is based on the VueJS platform. Therefore, we follow VueJS production build generation options. To create a production build, run;

npm run build

This would create a dist folder that can be served via HTTP. To do this, we need to run;

npm install -g serve
sudo serve -s -l 80 dist

This will serve the app over the default HTTP port (80) so you can simply enter the IP address without having to explicitly mention the port. To serve on this port, you need elevated access.

But in practice, we will have to serve our application on both HTTP and HTTPS (443) ports. We can specify multiple ports for the serve command;

sudo serve -s -l 80 -l 443 dist

Setting up SSL

At this point, if you tried accessing the HTTPS version of the app, you would immediately get an SSL protocol error. If you have a rough idea about how SSL works, this shouldn’t come as a surprise for you. We need to install an SSL certificate.

The SSL procedure is a bit of a lengthy affair. There are certificate authorities (CAs) that grant paid certificates as well as ones that are free. You can get a free certificate from Let’s Encrypt. The recommended way of setting up an SSL certificate from Let’s Encrypt is using Certbot. They have detailed instructions on their site. In brief, you just have to run two commands;

sudo apt-get install certbot
sudo certbot certonly --standalone

Certbot will ask some questions including prompting for an email address and domain/sub domain (if you are registering a sub domain, enter the sub domain, ex: app.example.com).

The last command will generate a public key and a private key. The paths to the both the keys will be listed at the end. To add SSL support to our server command, we need to add some extra parameters referencing the keys.

sudo serve -s -l 80 -l 443 --ssl-cert /etc/letsencrypt/live/<domain>/fullchain.pem --ssl-key /etc/letsencrypt/live/<domain>/privkey.pem dist

Both of these paths will be listed out for you. And <domain> will be replaced by the domain or sub domain you selected.

Renewing SSL certificates (Update 07 July 2020)

Due to the procedure used to install the SSL certificates, auto renewal feature of certbot would not work some times. However, the certificate can be manually renewed using the command below:

sudo certbot renew

This will renew all installed certificates on the instance. Before executing this command, it would be necessary to stop any applications using the ports 80 and 443.

Now, you would have a production ready instance of UPchieve on your Cloud VM. There is a few more problems however. How do you make sure the script runs continuously even when the SSH terminal is closed? How do you ensure that service starts with at startup – if a restart occurs how would the server start back up? These are questions on their own and deserve their own article!