Project Day 5: The Inbox or the Hardest Part of an Integration is the Integration Part
Note: This isn't a guide for people that have never touched AWS, serverless, etc. There are better guides than mine for understanding those technologies. It's also not that hard, and my project at this point puts you in a really good spot to just focus on the code.
If you've ever integrated with a system, you've probably made some assumptions about how that other system works. If you've been around the block, you know more than likely you're going to be wrong somewhere along the way. Thankfully the hard part is sending out data with a valid signature which we did last time, and next hardest problem is validating the incoming signature. Once you solve both of those problems, you're back to a normal run of the mill CRUD backend.
Today we'll solve the incoming signature problem.
One Inbox, Two Inboxen
We have two inboxes: a public inbox and your personal inbox. A public inbox gets data intended to be read and acted on by anyone on the server. Your inbox is intended to be read and acted on by just you. In order to get data on our server you need to follow people, just like how you follow people on Facebook, or LinkedIn. That's a complicated problem for us right now, let's solve something easier; people following us.
Now, we have no way to currently contribute anything to the Fediverse, so accepting followers at this point might feel weird. However, this is a great way to prove we can receive Fediverse data and validate it because it's the easiest type of Activity to implement.
I've got 99 Problems and Capturing Network Traffic is All of Them
Getting WebFinger to work was trivial. Trying to figure why the communication between Mastodon wasn't working proved more challenging. I finally dug into Mastodon's source code, modified it to output the signature prior to signing and found out the trickiest part of my code worked without a problem, but the dumbest part of my code had the bug.
Always follow your heart, unless you have network traffic to suggest otherwise, and then follow the network traffic.
Hundreds of lines of perfectly written code, all defeated by forgetting my own path was /activitypub/users/{user}/inbox and not /users/{user}/inbox. Well, that was a fun day.
Checkout the Code
git clone https://github.com/alexayers/hellofriend.git
git checkout tags/day5 -b main
Personal Inbox
file: ./backend/activity-pub/user/handler.ts
We'll only focus on the personal inbox today, and we'll do so in chunks.
let account : Account = await accountService.getByNormalizedUsernameDomain(event.pathParameters.user);
if (!account) {
return notFoundResponse( "Requested user's inbox is not found");
}
First we'll validate that the incoming request is for a user on our server, since users that aren't on our server don't have their own inbox.
let validationStatus : ValidationStatus = await inboxSerivce.validateRequest(`/activitypub/users/${account.username}/inbox`,event.headers);
if (validationStatus != ValidationStatus.VALID) {
return notAuthenticatedResponse("Unable to validate request")
}
Next we are going to validate that the incoming request is considered valid by our server. We'll get into what we mean by valid in a minute.
let activity : Activity = event.body as unknown as Activity;
switch (activity.type) {
case ActivityType.Follow:
await followService.acceptRequest(event.body as unknown as FollowActivity)
break;
default:
console.warn(`I don't know what to do with Activity Type: ${activity.type}`)
}
return successResponse({"Success": true});
We are going to cast the event body to an Activity and look at the type of Activity. Today we only care about Follow Activities, so if we get one of those we'll process it, otherwise we'll just log the fact we don't know what to do with the Activity. Your server will get these any time someone on the Fediverse wants to Follow you.
{
"Success": true
}
Lastly we'll return a nice success response because we're just positive people around here, and ultimately this response doesn't matter.
Inbox Service
Our inbox service is going to ensure the incoming data is valid.
async validateRequest(inboxUrl: string, headers: APIGatewayProxyEventHeaders): Promise<ValidationStatus> {
let signature: string | undefined = headers["Signature"];
if (!signature) {
console.error("Header missing Signature");
return ValidationStatus.NO_RETRY;
}
let signatureMap: Map<string, string> = this.extractSignature(signature);
We're going to check for the presence of a Signature in the request header which is the bare minimum required to be a valid message.
let date: string = headers["Date"];
if (!date) {
console.error("Header missing Date");
return ValidationStatus.NO_RETRY;
}
let incomingDate : Date = new Date(date);
const oneMinuteAgo: Date = new Date(incomingDate.getTime() - 60000);
if (incomingDate < oneMinuteAgo) {
console.error(`Date ${incomingDate} is older than 60 seconds ${oneMinuteAgo}.`);
return ValidationStatus.NO_RETRY;
}
Next we want to validate that the incoming request isn't too old to guard against someone tampering with the message during it's long journey across the Internet. Now that the payload has passed our basic tests, we are going to extract the components of the signature.
Incoming Signature
Here's an example Signature.
Signature: 'keyId="https://mastodon.social/users/test#main-key",algorithm="rsa-sha256",headers="(request-target) host date digest content-type",signature="D09HcFnq0Q2uAnzgqGijHt7FleCWo6mxyaQU5tHqYYlQx3EHbD5Di/9VbssP2nTgpX3FHC5+LDbsJ48hG5d7lDH6z6cPosgW8dYJvhnPVO31FP1duoudT3upBNsC7ft8wEXfWGOVw4CSdq8+smQLoBDL4Vlpp31iHfz1ZXh663eeCbERkitQCM0rKb9bFlW88yYpG9BfcgaFqMOXcd294OwiVWWSfn9jc1xA+zsbO/YtauxwRq5zT6mH8kflqNjC9c4VsfxzU/YRcRF5scKP4nLPgV4SmHtGFNGfKR6wrNhwj1xpiIQnwOiCoc0GP9cP82ThZZRrJhTbCIdbUfrbaw=="'
We are going to use this function break that out into key / value pairs.
private extractSignature(signature: string): Map<string, string> {
let parts: string[] = signature.split(",");
let signatureMap: Map<string, string> = new Map();
for (const part of parts) {
let pair: string [] = part.split("=");
let key: string = pair[0];
let value: string = pair[1].replaceAll("\"", "");
signatureMap.set(key, value);
}
return signatureMap;
}
This means our signature map becomes:
keyId="https://mastodon.social/users/test#main-key"
algorithm="rsa-sha256"
headers="(request-target) host date digest content-type"
signature="D09HcFnq0Q2uAnzgqGijHt7FleCWo6mxyaQU5tHqYYlQx3EHbD5Di/9VbssP2nTgpX3FHC5+LDbsJ48hG5d7lDH6z6cPosgW8dYJvhnPVO31FP1duoudT3upBNsC7ft8wEXfWGOVw4CSdq8+smQLoBDL4Vlpp31iHfz1ZXh663eeCbERkitQCM0rKb9bFlW88yYpG9BfcgaFqMOXcd294OwiVWWSfn9jc1xA+zsbO/YtauxwRq5zT6mH8kflqNjC9c4VsfxzU/YRcRF5scKP4nLPgV4SmHtGFNGfKR6wrNhwj1xpiIQnwOiCoc0GP9cP82ThZZRrJhTbCIdbUfrbaw=="
We're going to break things down further in our next function.
let expectedHeaders: string = this.extractExpectedHeaders(signatureMap, inboxUrl, headers);
We pass in our new signature map, the URL of our personal inbox (which very much is /activitypub/users/{user}/inbox), and the request headers.
private extractExpectedHeaders(signatureMap: Map<string, string>, inboxUrl: string, headers: APIGatewayProxyEventHeaders): string {
let expectedHeaders: string = "";
// Normalize header keys
for (let key in headers) {
if (headers.hasOwnProperty(key)) {
headers[key.toLowerCase()] = headers[key];
}
}
let headerList: string[] = signatureMap.get("headers").split(" ");
for (const header of headerList) {
if (header == "(request-target)") {
expectedHeaders += `(request-target): post ${inboxUrl}\n`;
} else {
expectedHeaders += `${header}: ${headers[header]}\n`;
}
}
return expectedHeaders.substring(0, expectedHeaders.length - 1);
}
The first thing we're going to do is normalize our keys. This is because the request header is a map<string, string> where the request header keys are capitalized, but our signature header keys are all lowercase. We'll end up with duplicate keys in our map, but we don't care, because this is all throw away work.
Next we're going take the signature headers, look up the corresponding values in the request header, and construct our expected header (i.e. the header we are expecting to be signed by the sending server). This means we take our signature header which looks like this:
headers="(request-target) host date digest content-type"
And we're transforming it into this
(request-target): post /activitypub/users/alex/inbox
host: api.hellofriend.social
date: Tue, 19 Dec 2023 01:23:28 GMT
digest: SHA-256=RWTWWgJmIqU0y1r/2AzMNYnM16Tv69MBWcxy6NrhRJg=
content-type: application/activity+json
We return from extractExpectedHeaders and continue in our function
let keyId: string = signatureMap.get("keyId");
let actor : {username: string, domain: string} = actorFromUrl(keyId);
let account : Account = await accountService.getByNormalizedUsernameDomain(actor.username, actor.domain);
Now fetch out keyId from the map "https://mastodon.social/users/test#main-key" and we call a simple util function to extract the username "test" and domain "mastodon.social." We then look up that user in our database to see if it's cached (spoiler: we haven't done caching yet).
if (account) {
if (!account.publicKey) {
return ValidationStatus.NO_RETRY;
}
return this.validateKey(account.publicKey, signatureMap.get("signature"), expectedHeaders);
}
If it were cached, we'd confirm we have a public key for the user. If we don't, return in an error state. If we do, we'll validate the signature using the key.
} else {
let body = await fediverseService.signedRequest("get", keyId);
let personActor: PersonActor = body as PersonActor;
if (!personActor || !personActor.publicKey) {
return ValidationStatus.NO_RETRY;
}
return this.validateKey(personActor.publicKey.publicKeyPem, signatureMap.get("signature"), expectedHeaders);
}
Else (or in our case at the moment "always") We are going to use our Fediverse Service from the last article, but instead of performing a Web Finger we're request the actor object which is stored in the keyId. We'll confirm that we got a Person Actor and that the Person Actor has a public key, otherwise we'll return in error. Lastly we'll validate the key.
private validateKey(publicKey: string, signature: string,expectedHeaders: string ) : ValidationStatus {
const key : crypto.KeyObject = crypto.createPublicKey(publicKey);
let isValid: boolean = crypto.verify(
"sha256",
Buffer.from(expectedHeaders, 'utf8'),
{
key: key,
padding: crypto.constants.RSA_PKCS1_PADDING,
},
Buffer.from(signature, 'base64')
);
return isValid ? ValidationStatus.VALID : ValidationStatus.REFRESH;
}
This is more or less boilerplate, but let's review it anyhow. First we create a public key object using the public key of the person who sent us this message. Then we're going to compare the plaintext expected headers, against the hashed and signed value. This will only be equal if the person who sent us the message controls the private key, so if this passes, we know this message can be trusted.
The sending and receiving of Fediverse data is really the most complicated part of the project, and we've finished it. Now we can move on to the dumb plumbing.
Follow Activity
Alright, so we have two approaches with Follow requests: we can immediately agree to the request, or we can do a bit more work and have an approval workflow. We're friendly, and on a schedule, so we'll just automatically agree to all requests for now. Most people automatically accept invitations, because it's an easy way to maximize the amount of Fediverse you can see on a given server.
{
'@context': 'https://www.w3.org/ns/activitystreams',
id: 'https://mastodon.social/1db0df88-50c2-4d5a-a87b-26459ea84e56',
type: 'Follow',
actor: 'https://mastodon.social/users/test',
object: 'https://www.hellofriend.social/users/alex'
}
Here's an ActivityPub event for Follow. Let's break it down.
Accept Activity
We must reply now or eventually with an Accept Activity to let the sending server know we agreed to the Follow request.
{
"@context": "https://www.w3.org/ns/activitystreams",
id: `https://${this.domain}/${uuidv4()}`,
actor: followActivity.object,
type: ActivityType.Accept
object: {
"@context": "https://www.w3.org/ns/activitystreams",
actor: followActivity.actor,
id: followActivity.id,
object: followActivity.object,
type: ActivityType.Follow
},
}
Right now our Follow Service is going to handle replying to all Follow requests with an Accept. I don't think that's terribly interesting so we aren't going to dive into it. We'll basically send the Accept Activity and sign it (just like we signed the WebFinger) so the receiving server knows it's from us. We aren't even storing the Follow request at this point, but we will do that later.
Shouldn't this be on a Queue?
I only want to validate the requests when they come into my inbox. Currently I'm also processing them. It's a lot of network traffic, and it's not even all the potential traffic, before replying to the sending server. Imagine we got a request from someone we never saw before, we might want to do the following:
That's five separate network requests which are going to add up especially steps 3 and 4. Now the following and validation is working, let's instead stick this request onto a queue for processing provided the message is valid.
The other reason you want a queue is Fediverse can be unstable. These servers aren't run by a team of professionals, so you have to design around servers blowing up, and going down. By processing data in a queue, we can leverage retry logic, so that we can retry network requests a few times before we give up.
Next Time
Next time we'll take some of our existing logic, and have that work off a queue. Until next time!
Other Articles in this Series