Project Day 6: The Heartbeat of an Emerging System

Note: This isn't a guide for people that have never touched AWS, serverless, etc. There are better guides than mine for understanding those technologies. It's also not that hard, and my project at this point puts you in a really good spot to just focus on the code.

Today is when the system starts to come to life for the first time. Last time we accepted a Follower request. While important, what we want is to follow other accounts, because this is how we get data into our system. The best way to get a lot of data is to follow a bot, and the best type of bot to follow would be ones that are reporting news stories, because news feeds have tons of data (All of it sad).

Ok, knowing that we can get a lot of incoming data from news bots, we'll head over to https://press.coop/directory which is a Mastodon server will lots of news bots. It looks like the following accounts generate a lot of posts per week:

• https://press.coop/@BBCNews
• https://press.coop/@business
• https://press.coop/@chicagotribune
• https://press.coop/@guardian

Before we start to follow people let's talk about the new code.

The Code

git clone https://github.com/alexayers/hellofriend.git
git checkout tags/day6 -b day9progress        

Inbound and Outbound Queues

Generally speaking we want to keep the user experience in the eventual WebApp as responsive as possible, so anything that's going to take more than a second we probably want to do asynchronously. In order to do that we'll work with two different queues, maybe we'll need more later, but two seems good enough to start.

Inbound Queue

file: ./backend... wait a minute this is all on Github... I can just link to this directly... what a fool I've been...

Inbound Queue Code

Anything someone sends to our instance on the Fediverse will go into our inbound queue following validation in a public or private inbox.

Outbound Queue

Outbound Queue Code

Anything we want to send to someone out on The Fediverse will be placed onto our Outbound Queue.

What's particularly handy about this design is it's pretty simple to just craft a JSON payload and queue it directly with SQS via AWS' UI. In the longer term our public facing API will be the one queuing data. For now, we'll want to craft and queue some Follow Activities for the news bots above so that we start to get data. Let's craft some JSON. We'll drop each one of these artisan crafted JSON objects into our SQS Outbound queue

Follow Format

• context will always be https://www.w3.org/ns/activitystreams
• id is a unique identifier for this request. This could be used to track the accept or pending status of a request
• type will always be Follow
• actor is a URI to your user
• object is a URI to the person you want to follow

BBC News

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "id": "https://www.hellofriend.social/follow/176cf1b1-064b-4d4a-8beb-31cbb965928a",
  "type": "Follow",
  "actor": "https://www.hellofriend.social/users/alex",
  "object": "https://press.coop/users/BBCNews"
}        

Bloomberg

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "id": "https://www.hellofriend.social/follow/276cf1b1-064b-4d4a-8beb-31cbb965928a",
  "type": "Follow",
  "actor": "https://www.hellofriend.social/users/alex",
  "object": "https://press.coop/users/business"
}        

Chicago Tribune

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "id": "https://www.hellofriend.social/follow/376cf1b1-064b-4d4a-8beb-31cbb965928a",
  "type": "Follow",
  "actor": "https://www.hellofriend.social/users/alex",
  "object": "https://press.coop/users/chicagotribune"
}        

The Guardian

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "id": "https://www.hellofriend.social/follow/476cf1b1-064b-4d4a-8beb-31cbb965928a",
  "type": "Follow",
  "actor": "https://www.hellofriend.social/users/alex",
  "object": "https://press.coop/users/guardian"
}        

Then in a few seconds we should expect to get four Accept Activities in our Inbound Queue. We haven't written the code to process an Accept Activity yet, and we don't need to do that in order to start getting data. You will want to track this though because in our UI you'll want to be able to show:

1. Who you aren't following.
2. Who you are following.
3. Who you asked to follow and are waiting for approval.

Now We Wait

Ok, now that we've told four different news bots to drown us in news, we should start getting Create Activities and feeling depressed about society in no time! Every time you want to post data to Fediverse you are creating a Create Activity (Update Activity to update something and Delete Activity to delete something). Oh! This reminds me, there are a few different ways to publish your push to Fediverse:

1. I want everyone to know
2. I want only my followers to know
3. I want you specifically to know

Most people want everyone to know everything they are thinking, which works well for us when we want a lot of data from the Fediverse, and less well when we are waiting for the bus. Anyhow, back to the queue!

The other cool thing about our approach is we can just capture all the different Activity types, and implement the logic behind each one we get until we have implemented them all. Could we read the spec? Sure, but why read the spec which tells us what systems should be doing, when we can look at network activity to see what systems are actually doing?

Create Activity

Create Service

Within a few minutes we will start to get incoming statuses, and here's one now, and it's about death. Wonderful!

{
  "@context": [
    "https://www.w3.org/ns/activitystreams",
    {
      "ostatus": "https://ostatus.org#",
      "atomUri": "ostatus:atomUri",
      "inReplyToAtomUri": "ostatus:inReplyToAtomUri",
      "conversation": "ostatus:conversation",
      "sensitive": "as:sensitive",
      "toot": "https://joinmastodon.org/ns#",
      "votersCount": "toot:votersCount",
      "Hashtag": "as:Hashtag"
    }
  ],
  "id": "https://press.coop/users/BBCNews/statuses/111612537869751036/activity",
  "type": "Create",
  "actor": "https://press.coop/users/BBCNews",
  "published": "2023-12-20T11:36:12Z",
  "to": ["https://www.w3.org/ns/activitystreams#Public"],
  "cc": ["https://press.coop/users/BBCNews/followers"],
  "object": {
    "id": "https://press.coop/users/BBCNews/statuses/111612537869751036",
    "type": "Note",
    "summary": null,
    "inReplyTo": null,
    "published": "2023-12-20T11:36:12Z",
    "url": "https://press.coop/@BBCNews/111612537869751036",
    "attributedTo": "https://press.coop/users/BBCNews",
    "to": ["https://www.w3.org/ns/activitystreams#Public"],
    "cc": ["https://press.coop/users/BBCNews/followers"],
    "sensitive": false,
    "atomUri": "https://press.coop/users/BBCNews/statuses/111612537869751036",
    "inReplyToAtomUri": null,
    "conversation": "tag:press.coop,2023-12-20:objectId=2434829:objectType=Conversation",
    "content": "<p>Esther Rantzen: Minister says he is 'not averse' to new assisted dying vote</p><p>Mel Stride says MPs may wish to revisit topic after Esther Rantzen announces she is joining Dignitas. <a href=\"https://press.coop/tags/press\" class=\"mention hashtag\" rel=\"tag\">#<span>press</span></a></p><p><a href=\"https://www.bbc.co.uk/news/uk-67770958?at_medium=RSS&at_campaign=KARANGA&utm_source=press.coop\" target=\"_blank\" rel=\"nofollow noopener noreferrer\"><span class=\"invisible\">https://www.</span><span class=\"ellipsis\">bbc.co.uk/news/uk-67770958?at_</span><span class=\"invisible\">medium=RSS&at_campaign=KARANGA&utm_source=press.coop</span></a></p>",
    "contentMap": {
      "en": "<p>Esther Rantzen: Minister says he is 'not averse' to new assisted dying vote</p><p>Mel Stride says MPs may wish to revisit topic after Esther Rantzen announces she is joining Dignitas. <a href=\"https://press.coop/tags/press\" class=\"mention hashtag\" rel=\"tag\">#<span>press</span></a></p><p><a href=\"https://www.bbc.co.uk/news/uk-67770958?at_medium=RSS&at_campaign=KARANGA&utm_source=press.coop\" target=\"_blank\" rel=\"nofollow noopener noreferrer\"><span class=\"invisible\">https://www.</span><span class=\"ellipsis\">bbc.co.uk/news/uk-67770958?at_</span><span class=\"invisible\">medium=RSS&at_campaign=KARANGA&utm_source=press.coop</span></a></p>"
    },
    "attachment": [],
    "tag": [
      {}
    ],
    "replies": {
      "id": "https://press.coop/users/BBCNews/statuses/111612537869751036/replies",
      "type": "Collection",
      "first": {}
    }
  },
  "signature": {
    "type": "RsaSignature2017",
    "creator": "https://press.coop/users/BBCNews#main-key",
    "created": "2023-12-20T11:36:12Z",
    "signatureValue": "u5v0p1AYNoB3Mn6BieEu3szHqnF4nMk5z6bO8FKIYyNq2dn6Y+I8WnyaFzQEbB4rL0VQilRxdCFBp5WR7K+IeKi3fc44sKrtvF9U/Oacb/1JAzXtRduZAo+DMOWYrJXJ47e6Bp3COtJ9ku7SfZzPf9qCkmoCllt8JuLI365floT8wU2qYSSDaALf5VQwgN8zABYoq7GdxNDBr7w3HzuPw9URVL/guTmXhm5YN8hUTfN7X2IjcguMqD3uvIaoaMjQwmOCmvVzZ6h8Llm/Axh6oIreJxwxtvK0qR9lCje+0NXt4QZdXbE2Hr7X7ZBEb5WypyHNwd4JV6vvyyV5rgCeBw=="
  }
}
        

For now at least, we only care about a few elements of this huge (for you) payload.

Fields we'll store

• attributedTo: Tells us who wrote this message
• content: Tells us the text of the message
• conversation: Acts a key tying this status to any replies to this status which will become important if we want mute anything related to a given status
• inReplyToAccountId: Is this status a reply to a particular person?
• inReplyToId: Is this status a reply to a particular status?
• language: The language of the status
• published: The date of status creation
• sensitive: If we should treat this message as sensitive
• summary: Spoiler text
• updated: The date of status update
• uri: The URI to the status on the sending server
• url: The URL to the status on the sending server
• tag: If the status has any tags we'll want to process and store those too

Fields of interest, but we won't store them

• to: tells us the audience of this status
• cc: tells us any additional people intended to receive the message

 "to": ["https://www.w3.org/ns/activitystreams#Public"],
 "cc": ["https://press.coop/users/BBCNews/followers"],        

Basically, the BBC asked for their status to be delivered to the public timeline, and also send it to all followers.

Tags

Tag Service

If you've used a cool social network like LinkedIN, you'll know you can tag your statuses. Well, ActivityPub supports tagging on both statuses and accounts. We will process both, and store this information in our database

Unfollow

We also support the reverse, and we can now send an Undo Activity with a Follow Activity attached. This will cause the originating servers to stop sending us data. Simply drop these payloads into your SQS queue, and the data will stop flowing to your instance.

BBC News

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "id": "https://www.hellofriend.social/users/alex#follows/1/undo",
  "type": "Undo",
  "actor": "https://www.hellofriend.social/users/alex",
  "object" : {
    "@context": "https://www.w3.org/ns/activitystreams",
    "id": "https://social.alexayers.com/follow/176cf1b1-064b-4d4a-8beb-31cbb965928a",
    "type": "Follow",
    "actor": "https://www.hellofriend.social/users/alex",
    "object": "https://press.coop/users/BBCNews"
  }
}        

Bloomberg

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "id": "https://www.hellofriend.social/users/alex#follows/1/undo",
  "type": "Undo",
  "actor": "https://www.hellofriend.social/users/alex",
  "object": {
    "@context": "https://www.w3.org/ns/activitystreams",
    "id": "https://social.alexayers.com/follow/176cf1b1-064b-4d4a-8beb-31cbb965928a",
    "type": "Follow",
    "actor": "https://www.hellofriend.social/users/alex",
    "object": "https://press.coop/users/business"
  }
}        

Chicago Tribune

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "id": "https://www.hellofriend.social/users/alex#follows/1/undo",
  "type": "Undo",
  "actor": "https://www.hellofriend.social/users/alex",
  "object": {
    "@context": "https://www.w3.org/ns/activitystreams",
    "id": "https://social.alexayers.com/follow/176cf1b1-064b-4d4a-8beb-31cbb965928a",
    "type": "Follow",
    "actor": "https://www.hellofriend.social/users/alex",
    "object": "https://press.coop/users/chicagotribune"
  }
}        

The Guardian

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "id": "https://www.hellofriend.social/users/alex#follows/1/undo",
  "type": "Undo",
  "actor": "https://www.hellofriend.social/users/alex",
  "object": {
    "@context": "https://www.w3.org/ns/activitystreams",
    "id": "https://social.alexayers.com/follow/176cf1b1-064b-4d4a-8beb-31cbb965928a",
    "type": "Follow",
    "actor": "https://www.hellofriend.social/users/alex",
    "object": "https://press.coop/users/guardian"
  }
}        

Totally Based Database Design

Personally, I love thinking about the structure and relationships between data. When I work on a project or join a new project, I'll typically start with the database to understand the system. I'll read the schema, and try to commit the entire thing to memory. If I can recall the layout and relationships of the data, the code is just details at that point, and I can just simulate the whole system in my imagination.

What our database so far looks like this:

• hello-accounts: Holds the accounts and the account tags. We'll be doing most of our look up by the normalized account + domain index (<user>:<domain>), but our pkey is the ID which will be used with statuses.
• hello-follows: Will hold the people following us, and the people we are following.
• hello-statuses: Holds the statuses and status tags. We'll be doing most look ups by the skey which contains the Account ID of the status author
• hello-tags: Holds the tags. Our pkey is the name of the tag and it's normalized so ("#CoOl" will be stored as "cool")

Where are we currently?

What can our ActivityPub implementation currently do...

1. We can follow
2. We can be followed
3. We can reply to a Web Finger request
4. We can make a Web Finger request
5. We can unfollow
6. We can get statuses

What's Next?

It feels like a good next step is finally store the follow data. Then we'll follow some real people which will give us the data we need to implement the Boost, Update, and Delete activities. We're nearly done with the core functionality.

Other Articles in this Series

Project Day 0: Let's Get Cereal and Build Something Cool.

Project Day 1: ActivityPub or oh great, another social media thing... next article please.

Project Day 2: There are a million ways to design a system, and eventually any option will make me look dumb.

Project Day 3: Project Scaffolding and other OSHA violations

Project Day 4: Using your Web Finger

Project Day 5: The Inbox or the Hardest Part of an Integration is the Integration Part