Firebase Schema Evolution

Schema evolution is a natural part of your application lifecycle. Firebase is my go-to backend for web and mobile projects and I rely on two technologies when refactoring the data model of an app:

• Bolt as the single source of truth for both the backend (database rules, cloud functions) and the client (React Native). This prevents the backend and the client from evolving separately: Firebase cannot have a type definition that is different from the client and vice versa.
• JSONiq to quickly develop migration queries that can be executed on large datasets. Using JSONiq, it is possible to express complex data transformations with only few lines of code.

Single Source of Truth with Bolt

Bolt is modeling language for Firebase. By default, the Bolt compiler outputs Firebase rules that can be automatically deployed to the database. This feature alone is already a life saver since Firebase rules can be incredibly cumbersome to write manually. On top of that, there is also a compiler to Typescript available that allows you keep your data model in sync on the client. I haven’t tried yet to output type definitions for flow but it should be fairly easy to do as well. Other asset types such as documentation or customized code artifacts could be generated as well.

Let’s look at the following Bolt definition.

path /users/{uid} is User {
  read() { isAuthenticated() }
  write() { isCurrentUser(uid) }
}

type User {
  name: String,
  email: String
}

isAuthenticated() { auth != null }
isCurrentUser(uid) { auth != null && auth.uid == uid }

This file can be used to deploy the database security rules to firebase and to generate the data model on the client. To execute the first task, you can use the firebase-bolt command-line tool and curl.

# Generate firebase rules and deploy them
firebase-bolt < rules.bolt | curl -v -X PUT -d \
@- "https://$FIREBASE_APPNAME.firebaseio.com/.settings/rules.json?auth=$FIREBASE_SECRET"

The firebase-bolt-transpiler command can be used to generate typescript command. Since src/Model.ts is a generated file, you can add it to your .gitignore.

#Generate Typescript data model
firebase-bolt-transpiler < rules.bolt > src/Model.ts

The generated file can be used when fetching data as below.

import {User} from "./Model";

type Snapshot = firebase.database.DataSnapshot;
export default class UserStore {
  constructor() {
    FirebaseProvider.get().currentUserDBRef.on("value", (snapshot: Snapshot) => {
      this.user = snapshot.val() as User;
    });
  }
}

Now if you rename a field in the bolt file without refactoring the code properly, it won’t compile.

Now that you are certain to have the Firebase rules and the client in sync, the last step is to migrate the legacy data to the new schema if necessary.

Data Migration with JSONiq

The JSONiq query language enables you to easily express data transformations that otherwise would be a nightmare to write in Javascript. Personally, I like use the Zorba docker image to execute JSONiq code.

The following JSONiq query fetches a document from firebase and applies a simple transformation to it before sending it back to the server.

import module namespace fb = "https://wcandillon.io/modules/firebase" at "firebase.jq";

declare variable $token as string external;

(: 1. Get firebase documents :)
variable $old-users := fb:get("users.json", $token);

(: 2. Add a new lastModified field for each users :)
for $user in keys($users) ! $users($$)
return
    insert json { lastModified: $user.createdAt } into $user;

(: 3. Send back the updated document :)
fb:put("users.json", $users, $token);

To execute this query, you need to set a proper Firebase token, which you can do with the following command.

docker run — rm -v $(pwd)/admin:/queries fcavalieri/zorba -q migrate.jq -f -i -e token:=$FIREBASE_SECRET

JSONiq provides capabilities to interact with the Firebase REST API easily. The following is the implementation of fb:get(), fb:put(), and fb:delete().

module namespace fb = "https://wcandillon.io/modules/firebase";

import module namespace http = "https://zorba.io/modules/http-client";

declare namespace an = "https://zorba.io/annotations";
declare variable $admin:firebase-endpoint := "https://myapp.firebaseio.com/";

declare %an:nondeterministic function fb:get($document-uri as string, $token as string) {
  http:get(
    $admin:firebase-endpoint || $document-uri || "?auth=" || $token
    ).body.content ! parse-json($$)
};

declare %an:sequential function fb:delete($document-uri as string, $token as string) {
  http:delete($admin:firebase-endpoint || $document-uri || "?auth=" || $token)
};

declare %an:sequential function fb:put($document-uri as string, $document as object, $token as string) {
  http:put(
    $admin:firebase-endpoint || $document-uri || "?auth=" || $token,
    serialize($document),
    "application/json"
  )
};

In this last example, we remove data that has been duplicated from multiple locations since it is a common pattern in Firebase.

import module namespace fb = "https://wcandillon.io/modules/firebase" at "firebase.jq";

declare variable $token as string external;

for $uid in fb:get("users.json", $token) ! keys($$)
for $eid in fb:get("users/" || $uid || "/feed.json", $token) ! keys($$)
for $uri in (
  "feed/" || $eid,
  "users/" || $uid || "/feed/" || $eid,
  "comments/" || $eid
)
return fb:delete($uri);

Et voilà

I hope this article was helpful to you, and I look forward to receiving your feedback.