Bring Push Notification to Diablo Clone with Netlify CMS, Gatsbyjs and Fauna Database(1)

I have always been a fans of Diablo and have been playing the Diablo II Expansion on and off for years. One thing in the Diablo II, Diablo Clone(DC), is definitely the most exciting event in the world of Diablo. I cannot stress enough that how frustrated I was whenever I missed a hunting event.

So.....what could I do for this? huh.....why not make an tiny application for DC notification? How hard could it be? Let's do it??

Let's see the result first :

and how to add the PWA website as an app-like onto mobile desktop:

In this article, I will demonstrate:

1. How to make a Windows Service with C# .Net Core
2. How to use Netlify Lambda function (Powered by AWS Lambda. Simplified by Netlify) as a server-less back-end
3. Convert a Gatsbyjs website to Progressive Web App(PWA)
4. How to integrate Push Notification into the PWA
5. Use Fauna Database to store push subscriptions.

Initially, I was thinking of flashing a LED light when the DC event is happening and playing a music for selling Stone of Jordan and a different music for Diablo walks the earth. Then setting up my old Arduino components exhausted all my interests on it plus I won't be able to get notified when I'm not at home, so I dropped the LED light idea.

Then I turned around and had a look at what I had: a Gatsby + Netlify CMS website. Interestingly, I even found that the Gatsbyjs Netlify CMS template that I'm using supports Netlify CMS Function~!?? Then this opens a door for me to use the lambda function as a server-less back-end so that I could do more interesting things.

So this is how the project works:

??when events of selling Stone of Jordan and Diablo walks the earth happen, a file located under a folder let's say: d:\games\diablo2\dc.txt will be updated with the room name, game server ip, my character's name and the date and time will be recorded inside this file.

??a Windows Service keeps track of the changes of this file and will notify the server-less Netlify Lambda function

??the Netlify Lambda function then push out notifications to all related subscribers

So how does this all fit together? Okay, let's have a look what the project look like:

Section 1: Windows Service

For the Windows Service project, its main task is monitoring the dc.txt file and notify the Netlify Function(NF) once the content of this file is changed.

Why makes it a Windows Service?

Because I would like to have a service running in background and continuously keeps tracking of the file and informs the NF once the events happened. In addition, I am lazy, I do not want to manually run this project every time myself, instead once my laptop started, run the service automatically.

Okay, enough talky talk, let's rock and roll:

Open Visual Studio 2019 and create a Worker Service project

Install these 2 packages:

Microsoft.Extensions.Hosting.WindowsServices

Microsoft.Extensions.Http

Then open the Program.cs, make it look like this:

public static IHostBuilder CreateHostBuilder(string[] args) =
            Host.CreateDefaultBuilder(args)
                .ConfigureServices((hostContext, services) =>
                {
                    services.AddConfiguration<DCMonitorConfiguration>(hostContext.Configuration, "DCMonitor");
                    services.AddHttpClient();
                    services.AddHostedService<Worker>();
                })
                .UseWindowsService();>        

services.AddConfiguration<DCMonitorConfiguration>(hostContext.Configuration, "DCMonitor"); is not necessary if you don't mind a little bit hard-coding stuff in a personal project.

Now open the Worker.cs, change it to something like this:

public class Worker : BackgroundServic
    {
        private readonly ILogger<Worker> _logger;
        private readonly string TargetFile = @"d:\games\Diablo2\dc.txt";
        private readonly string PushEndpoint = "https://yourwebsite.com/.netlify/functions/";
        private readonly string NotifyEmail = "[email protected]";
        private readonly HttpClient _httpClient;
        private FileSystemWatcher _fileWatcher;

        public Worker(ILogger<Worker> logger, IHttpClientFactory httpClientFactory)
        {
            _logger = logger;

            _httpClient = httpClientFactory.CreateClient();
            _httpClient.DefaultRequestHeaders.Add("User-Agent", "DCMonitorSvc");
            _httpClient.BaseAddress = new Uri(PushEndpoint);
        }

        protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            try
            {
                _logger.LogInformation("Worker running at: {time}", DateTimeOffset.Now);
                await Task.Delay(1000, stoppingToken);
            }
            catch (Exception ex)
            {
                _logger.LogError($"Worker error at {DateTime.Now}{Environment.NewLine}{ex.Message}");
            }
        }

        public override Task StartAsync(CancellationToken cancellationToken)
        {
            try
            {
                //System.Diagnostics.Debugger.Launch();

                _fileWatcher = SetupFileWatcher(TargetFile);
                return base.StartAsync(cancellationToken);
            }
            catch (Exception ex)
            {
                _logger.LogError($"Worker error at {DateTime.Now}{Environment.NewLine}{ex.Message}");
                return Task.FromResult(ex);
            }

        }

        public override Task StopAsync(CancellationToken cancellationToken)
        {
            _fileWatcher.EnableRaisingEvents = false;
            _fileWatcher.Dispose();
            return base.StopAsync(cancellationToken);
        }

        private FileSystemWatcher SetupFileWatcher(string targetFile)
        {
            var watcher = new FileSystemWatcher(targetFile, "*.txt");
            watcher.NotifyFilter = NotifyFilters.LastAccess
                                             | NotifyFilters.LastWrite
                                             | NotifyFilters.FileName
                                             | NotifyFilters.CreationTime
                                             | NotifyFilters.DirectoryName;

            // Add event handlers.
            watcher.Changed += OnChanged;

            // Begin watching.
            watcher.EnableRaisingEvents = true;
            return watcher;
        }

        private async void OnChanged(object source, FileSystemEventArgs e)
        {
            // Specify what is done when a file is changed, created, or deleted.
            Console.WriteLine($"File: {e.FullPath} {e.ChangeType} at {DateTime.Now}");

            if (e.ChangeType == WatcherChangeTypes.Changed)
            {
                try
                {
                    var fileContent = File.ReadAllText(e.FullPath);
                    var parts = fileContent.Split('|');

                    // 2020-08-08 16:59:00|AcA|aca-123|398120|112
                    if (!string.IsNullOrWhiteSpace(fileContent))
                    {
                        await Alert(fileContent);
                    }
                }
                catch (Exception ex)
                {
                    Console.WriteLine($"File is in use.{Environment.NewLine}{ex.Message}");
                }

            }
        }

        private async Task Alert(string notificationContent)
        {
            string url = $"alert?dc={notificationContent}&email={NotifyEmail}";
            var response = await _httpClient.GetAsync(url);
            response.EnsureSuccessStatusCode();
        }
    }
        

So what I have done here is:

Defining the StartAsync(), StopAsync() and Execute() functions as they map to start/stop the Windows Service we are developing. Most importantly, we create a FileWatcher which keeps tracking of the file(dc.txt) under the game folder and sends out notifications by calling an endpoint which I will cover in the next section.

Once the project is built, register this Windows Service in a command line like this:

# Register your Windows Servic
> sc.exe create DCMonitor binPath="to-your-windows-service.exe"

# Delete your Windows Service
> sc.exe delete DCMonitore        

Then open press Win button and type Service to open the Services panel and start your very own Windows Service:

We can make it automatic so that it starts once PC starts.Go to Task Manager, you should be able to find your Windows Service under background processes:

Section 2: Progressive Web App + Push Notification

In this section, I will provide references on how to set up a Gatsby.js + Netlify CMS website and add support to Service Worker so that we can turn the website into a Progressive Web App(PWA). Then I will set up Push Notification so that the PWA will be able to notify users when the delicious Diablo Clone events happens.

I use this gatsby-starter-netlify-cms template to start building my personal blog website. The reason is very simple, with a few steps of configuration, I am able to have a fully functional blog website. It also has the real-time preview functionality which enables me to see how my blog looks like when I am still in writing.

What is more interesting is, I can even use the Blog editor to add html components on a page on the fly! This means for simple UI changes, I do not even need any IDEs, simply just open my website's admin panel and write a few lines of code, then clicks Publish, the new html components will be immediately added onto my website. How cool it is~! ??

To give you a bit of idea about how it looks like:

PWA

Adding Service Worker and convert our normal website to PWA has been made incredibly easy. Digital Ocean has written a great article which helps me a lot on converting my Gatsby website to PWA, I will not repeat what has been written there, so follow the tutorial here:

Making Gatsby a PWA: Service Worker and Web App Manifest.

One thing to notice here, in order for the web app manifest to be cached, we’ll need to list gatsby-plugin-manifestBEFORE gatsby-plugin-offline

Push Notification

Google has written its fantastic tutorials which I found particular useful and easy to follow:

Developing Progressive Web Apps 08.0: Integrating web push

Notes: on section 4, the old approach doesn't work anymore, so don't stop there, go ahead use the VAPID approach, everything will be fine ??

When talking about push notification, there are 2 actions: Push and Notifying. Once I speak it out, it seems obvious, but back then I was not aware of these. So here is my learning notes:

The Push events happen at server side, the Notifying part happens at client side.

So let's do the client side part. In the project under the staticfolder add a file helpers.js:

async function askForPermission() 
  console.log("ask for permission");
  if (!("Notification" in window)) {
    console.log("This browser does not support notifications!");
    throw new Error("This browser does not support notifications!");
  }

  const status = await Notification.requestPermission();
  console.log("Notification permission status:", status);
  return status === "granted";
}

function urlB64ToUint8Array() {
  const applicationServerPublicKey = "";

  const padding = "=".repeat((4 - (applicationServerPublicKey.length % 4)) % 4);
  const base64 = (applicationServerPublicKey + padding)
    .replace(/\-/g, "+")
    .replace(/_/g, "/");

  const rawData = window.atob(base64);
  const outputArray = new Uint8Array(rawData.length);

  for (let i = 0; i < rawData.length; ++i) {
    outputArray[i] = rawData.charCodeAt(i);
  }
  return outputArray;
}

// Subscribe user is actually subscribing push registration
async function subscribeUser(swRegistration) {
  const applicationServerKey = urlB64ToUint8Array();

  try {
    const subscription = await swRegistration.pushManager.subscribe({
      userVisibleOnly: true,
      applicationServerKey: applicationServerKey,
    });

    console.log("User has subscribed successfully");
    return subscription;

  } catch (subscribeError) {

    if (Notification.permission === "denied") {
      console.warn("Permission for notifications was denied");
    } else {
      console.error("Failed to subscribe the user: ", subscribeError);
    }

    return null;
  }
}

async function getSubscription() {
  try {
    const swRegistration = await navigator.serviceWorker.ready;
    let pushSubscription = await swRegistration.pushManager.getSubscription();

    // if not found from pushManager, then we subscribe the user right now
    if(!pushSubscription) { 
      pushSubscription = await subscribeUser(swRegistration)
    }

    pushSubscription = pushSubscription.toJSON();
    document.getElementById("subkey").textContent = pushSubscription.keys.auth;
    return pushSubscription;
  } catch (error) {
    console.log("getSubscription() error: ", error);
    return null;
  }
}

function validateEmail(email) {
  const re = /^(([^<>()[\]\\.,;:\s@\"]+(\.[^<>()[\]\\.,;:\s@\"]+)*)|(\".+\"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;
  const valid = re.test(email);
  return valid;
}

async function updateSubscriptionOnServer(pushSubscriptionObject) {
  let url = "";
  try {
    url = "https://yourwebsite.netlify.app/.netlify/functions/updateSubscription";

    await fetch(url, {
      method: "POST", // *GET, POST, PUT, DELETE, etc.
      mode: "no-cors", // no-cors, *cors, cors, same-origin
      cache: "no-cache", // *default, no-cache, reload, force-cache, only-if-cached
      credentials: "same-origin", // include, *same-origin, omit
      headers: {
        "Content-Type": "application/json;charset=utf-8",
      },
      redirect: "follow", // manual, *follow, error
      referrerPolicy: "no-referrer", // no-referrer, *no-referrer-when-downgrade, origin, origin-when-cross-origin, same-origin, strict-origin, strict-origin-when-cross-origin, unsafe-url
      body: JSON.stringify(pushSubscriptionObject), // body data type must match "Content-Type" header
    });

    return true;

  } catch (ex) {
    console.log("saveReg() catch block:", ex);
    return false;
  }
}

async function updateSubscription() {
  try {
    const allowed = await askForPermission();
    if (!allowed) return;

    let subscription = await getSubscription();
    if (!subscription) return;

    // email
    const email = getEmail();
    if (!email || !validateEmail(email)) {
      alert("huh...so how are you going to receive notifications?");
      return;
    }

    let extra = {
        email: email
    };

    subscription.extra = extra;
    const successful = await updateSubscriptionOnServer(subscription);

    if (successful) alert("you have successfully subscribed to the DC monitor");
    else alert("shit happens, try it later");
  } catch (err) {
    console.log("updateSubscription() failed: ", err);
  }
};

function getEmail() {
  console.log("getEmail()");
  let email = document.getElementById("email").value;

  const pathname = window.location.pathname;
  console.log("the current location is: ", pathname);
  if (pathname.indexOf("/about") >= 0) {
    email = document.getElementById("email").value;
  }

  if (localStorage) {
    if(!email) {
      email = localStorage.getItem("dc_email");
    } else {
      localStorage.setItem("dc_email", email);
    }
  }

  if (email) {
    document.getElementById("email").value = email;
  }

  console.log("getEmail(): ", email);
  return email;
};
{        

So what we do here is:

1. we need to ask user's permission to do push notification.
2. get push subscription for this user from pushManager, if this user hasn't subscribed, then subscribe the user
3. patch the email onto the push subscription object, so that later we can target specific users for different notifications.
4. finally we save the updated push notification object into Fauna Database

In order to make the Javascript code available to the html components I added from the Blog editor, I have to put the script under static folder as all files under this folder will be kept as is. If I put the helpers.js under src folder, then the functions will be uglified then the function names will be simplified and I won't be able to reference them from the html code. Ideally, I would like to hide the public and private keys from the code and use Environment variables, but I have not figured it out. So please let me know if you know how to import the env variables for files under static folder. I have tried the dotenv, but had no luck.

Section 3: Netlify Functions

The features of Netlify Functions(NF) fits well as our server-less back-end. The idea is: our Windows Service calls the serverless functions and the functions will then notify the subscribed users.

Let's open lambda folder and add a file called: dcalert.js

const webPush = require("web-push")
const { getSubscriptions, removeSubscription, getResponse, getSubscriptionsByEmail } = require("./utils/utils.js");

const vapidPublicKey = process.env.VAPID_PUBLIC_KEY;
const vapidPrivateKey = process.env.VAPID_PRIVATE_KEY;
const notificationOptions = {
  TTL: 60,
  // TODO 4.3b - add VAPID details
  vapidDetails: {
    subject: `mailto:${process.env.VAPID_EMAIL}`,
    publicKey: vapidPublicKey,
    privateKey: vapidPrivateKey,
  },
};

async function notifySubscriber(pushSubscription, dc) {
  try {
    const response = await webPush.sendNotification(pushSubscription, dc, notificationOptions);
    if (response.statusCode % 200 <= 1) {
      console.log(`notification successfully sent at ${(new Date()).toString()}`, response);
      return true;
    }
    else {
      console.log(`notification sent at ${(new Date()).toString()} with status code ${response.statusCode}`, response);
      console.log("error: ", err);
      return  false;
    }
  }
  catch (ex) {

    if (ex.statusCode === 410) { // the subscription has expired or gone, then we remove the subscription from our database
      const auth = pushSubscription.keys.auth;
      const deleted = await removeSubscription(auth);
      if (deleted) {
        console.info(`subscription ${auth} has expired or gone, removed from database.`);
        return false;
      }
      else {
        console.error(`failed to remove expired subscription ${auth}`);
        return false;
      }
    }
    else {
      console.log("error: ", ex);
      return false;
    }
  }
}

function validateRequest(event) {
  const userAgent = event.headers["user-agent"];
  const dc = event.queryStringParameters.dc;

  if (!userAgent || !dc)
    return false;

  return true;
}

module.exports.handler = async function(event, context) {
  if (!validateRequest(event))
    return getResponse(400, "bad request");

  const userAgent = event.headers["user-agent"];
  const dc = event.queryStringParameters.dc;
  const email = event.queryStringParameters.email;

  if(!dc) {
    return getResponse(400, 'notification content cannot be empty');
  }

  let subscriptions = [];
  if (!email) {
    subscriptions = await getSubscriptions();
  } else {
    subscriptions = await getSubscriptionsByEmail(email);
  }

  if (!subscriptions) {
    return getResponse(200, 'No subscriber to notify');
  }

  let successfulCounter = 0;

  for (let pushSubscription of subscriptions) {
    const successful = await notifySubscriber(pushSubscription, dc);
    successfulCounter += successful ? 1 : 0;
  }

  if (successfulCounter === subscriptions.length) {
    return getResponse(200, `notification has been sent to all ${subscriptions.length} clients`);
  } else if (successfulCounter === 0) {
    return getResponse(500, `sending notification has failed`);
  } else {
    return getResponse(200, `notification has been sent to ${successfulCounter} out of ${subscriptions.length} clients`);
  }
};
;        

What happens here is, once a Http request hits this api endpoint, it does

1. validation against the request.
2. get subscriptions according to whether we passed in an email or not, via setting the email, we will be able to control sending notifications to all our subscribers or just a particular user.
3. iterate through the subscriptions and send out notifications.

One thing I'd like to point out is, push subscriptions could expire at some time, so we need to clean up those expired subscriptions. How do I handle this is when sending out notification, it will error out if the subscription has gone. I then use it as a chance to clean our the expired subscription. So take a look at catchsection of notifySubscriber().

This pushed me to the final solution: Fauna Database. It has free tier and recommended by Netlify, plus I always wanted to try this kind of NoSql like, non-structural database.

Look back what I have done so far, I have basically tried the JAM Stack with all these fantastic tools, it doesn't harm to add in one more fancy component into my project????. I will continue the article in series 2 as every time adding a new paragram into this article takes too long to response...