#1 About the difficulty of finding documentation, and a proposal

This is the second issue of Arranged ROMs (Rational Opinions, Mostly) . After sharing my thoughts on software engineering last month, I cover below the miscellaneous places to write documentation relating to software service code, and dependent configurations. I highlight the difficulty of finding it, and I share a related anecdote. This serves as an introduction to a candidate solution to mitigate the problem when there's no way to link to the documentation from where it is needed.

Documentation

You're part of a team owning multiple, different, and complex services. These are outnumbered by the count of engineers contributing to them. You have rookie and senior engineers, and you believe everyone can address any issue. They're trained, and have exhaustive knowledge about everything. Great. Just know that most of your engineers haven't been hired yet, so it's your documentation which will survive departures, not the institutional knowledge your team holds. Good point if the team knowledge is available as (preferably written) documentation. Reading documentation will speed up the onboarding of new joiners. Engineers can later unload such knowledge from their busy brain, once they are confident in being able to find the right documentation, quickly — like a pointer to the documentation instead of remembering its full content. As a team member or leader, you certainly prefer making room for current problems to work on, rather than keeping an every-growing amount of details. On the other side, if the documentation is difficult to find, it pushes engineers to keep it loaded in memory, or to waste time locating it when necessary. Here's a good test: how hard is it for an engineer with little to zero knowledge of a service to find its documentation, and investigate an incident at 3:00 AM local time, on Sunday? Documentation can be difficult to find, depending how far it is stored away from the place it is actually needed. Let's review some options below.

Documentation in code

As far as I know, all programming languages allow to write text which is ignored: comments. The dreaded comments. When I started my career in tech, I advocated for writing plenty of comments in the code. This seemed a great idea, because one could understand what the intent was along with the code. The pinnacle of this erroneous idea was a time I stated "source code should have two columns: code on the left, comments on the right, with an ideal 1:1 match". I later changed my mind, to write code which was self-explanatory, and documented the rationale for the change, not the technical change itself, in the application modeling the request (and in the commit to some extent).

Still, there are occasions where a comment is useful. Even if a constant bears a name clearly explaining its purpose (like MAX_NUMBER_OF_PARALLEL_CONNECTIONS), it might be set to a "magic value", within a range of accepted values. A comment mentioning the said range, or a link to a documentation page explaining why this is the chosen value, can help both the engineer modifying the value, the code reviewer, and the person investigating an incident after this value was changed.

Comments in the code are barely miss-able, especially if there are few of them. They can be useful pointers to a comprehensive documentation, to inform about the why and the how of the code, if necessary. Their text can invite to be extra cautious when modifying parts of the code.

Documentation alongside the code

For what doesn't change after code is initialized, like constants as mentioned in the previous section, executable specifications (also called "tests"), can serve as a pointer to the documentation. They can check for values to be within certain ranges, and fail with comprehensive messages including links to the documentation.

You can write documentation and place it in the same folder as the code source files. Should you have additional details you want to keep close to the source code, write it here; naming it after the same source code file will also help connect the dots. Doing so, you have higher chances of updating the documentation in the same commit as the code itself, resulting in both being versioned at the same time. You can write in any human-readable format, so the code review tool shows the diff properly. I confess I have a preference for writing such documentation using Markdown syntax. I like how it reads nicely both in plain-text and when it is rendered in Github, or other Web interfaces to explore code repositories — yes, this doesn't limit to the root README.md file.

Documentation outside of the code

Documentation outside of the code itself can be Word or PDF documents stored on a shared folder (so 90s!), entries in an internal Wiki, Quip or Notion documents (better options). The documentation then exists, however it needs to be linked to from an actual place. Unless it is mentioned in the code itself as suggested earlier, engineers will have a hard time finding it. When there are no direct references to the documentation, its discoverability becomes limited to a set of keywords engineers can think about to search for it, and to the performance of the search engine they use. In this game, files on a shared folder have a penalty in that the engineer's computer has certainly no option to index fast the remote file names (not to mention their content).

Git commits are good for documentation purpose, too. I often run git blame to identify which commits lead to the state of the code I'm reading, then jump to the commit messages — hoping they're well-written. It's expected a source code file is created, then updated multiple times. This requires a bit more work to parse the successive git log entries, however they amount to valuable documentation — again, provided they're well-written. In my past experiences, my team was adding references to the bottom of commit messages, so the reader could jump to the feature request or incident ticket in another application. I don't worry if the documentation is spread across several places, as long each consistently holds a given type of information: business request, system design, technical implementation, ...

What about configuration, deployment, and other services?

Your services are powered by your code, dependency code, and other services. These other services can be runtime dependencies to yours, or "utility" services. The latter usually proceed with the build of your packages, handle the deployment of your services across fleets of hosts, manage the configuration of load balancers, trigger alarms based on setup and monitoring probes, and so on. If you also own these services, you're likely a small company. Large companies have dedicated teams supporting and enhancing the "builder experience". In such case, tens, hundreds, or thousands of teams rely on these services. Each team has little to no control on what the Web interface shows to its members, because the UI is generic to fulfill the majority of the needs, and doesn't allow for customization.

This can be a problem if some engineers in your team still don't know the configuration of environment variables has been migrated to a file tracked in one of your code repositories. In such case, changing the values on the Web UI, then re-deploying the service has no effect (actual story). It's also a problem if one of the load balancers has been migrated from HTTP to TCP, and shows a crazy value for concurrent connections, 60 times the one of others (actual story number 2).

When you have no visible, discoverable tie between a configuration page and the related documentation your team needs to know about, you're at risk of wasting time finding it, of wasting brain memory to remember it, or over-relying on team members who wrote it. And if you suffer from the lack of linking Web pages to your documentation, please realize your very own users experience the same frustration.

You may think that, even if the link between the Web page and the documentation isn't visible, team members know where to search for documentation. Nobody would not think about searching for documentation, in one of the many places it is stored, right? Before jumping to a proposal to mitigate the risks mentioned earlier, let me share an anecdote about a similar issue, which impacted a major Internet player in France.

An anecdote

I had a difficult time finding additional details about this specific anecdote. Should you have more context, please leave a comment, I would be more than happy to update it with references, even if they're in French.

In the early 2000s, French Internet service provider (ISP) France Telecom was chasing after the new market offered by the deployment of the xDSL technology in Europe.

One of the first models of their DSL box, the "Livebox", could be connected to PCs with a USB cable. The vast majority of homes owned only one computer. Remember it was before the iPhone and the iPad existed; a time laptops were expensive enough that most people bought bulky desktop computers. So, connecting the Livebox with the USB cable provided in the bundle was a simpler option compared to setting up the computer to connect to the LAN or WAN of the Livebox. The device was great (models manufactured by Inventel were known to be better than Sagem ones for configuration nerds, like I was), however there was one caveat: it was mandatory to install the device driver from the CD-ROM packed, before connecting the Livebox to the USB port of a Microsoft Windows PC. It was written in the leaflet, the one nobody read before they were stuck, unable to access the Internet If you think it was straight-forward to fix the issue by manually removing a peripheral from MS Windows, before loading the CD-ROM containing the driver installation program, you're tech-savvy. Since it was leading the frustrated customers to call the ISP hotline, the information was then properly displayed on the packaging itself, with a red warning label bearing white text. Unmissable, right? Still, customers continued to call. I don't know how many steps it took, however the warning label was eventually tied to the USB cable, at the last time customers could stop from doing something the documentation in the leaflet forbid.

Lesson learned: it's not because the documentation, or instruction, is available somewhere that people will read it. It has more chances to be read if the information is pushed to the intended audience the closer to where and when it needs it.

A Proposal

Since the best approach is to push the documentation to the engineers at the time it is needed, important annotations in the code under the form of comments are a good approach, provided their use is limited to important places. As I wrote earlier, comments are unlikely to be missed if they appear on the line the engineer changes, or within the surrounding lines that the code review tool displays.

For external Web user interfaces engineers use to configure, deploy, monitor, and control their services, I'd like to see something like an information banner on Web sites. Information banners are user interface components which the Web browser loads from a separate server, displaying a message and the status of a service, usually the one behind the Website visited. This works by having the said service publish its status to the third-party monitoring endpoint. As long as the visited Web application visited returns a page, the component loads and informs of service issues, from simple warnings to operations going down. Such components are hidden most of the time, as (fortunately) there are nothing to report about.

My proposal is to build something similar to banners, where a user script takes care of fetching the content to display, and creates the UI component on the Web page. This removes the need for each service to update its Web page code to load the required third-party script. Such user scripts are common in large organization, to enhance the experience by adding convenient features to Web interfaces. Although the idea comes from the need to inform engineers about important information, it can be grown as a "generic annotation tool", to allow any user to push important information and documentation to their team at the time and place they need it.

In the diagram above, the flow for a user visiting an annotated Web page is the following:

1. User requests a Web page from the configuration / deployment / ... service's Web server.
2. The configuration / deployment / ... returns a Web page - hopefully.
3. With a userscript manager (GreaseMonkey , or other) add-on, the Web browser runs the user script, which requests the annotation service's Web server for messages to display for the current page URL.
4. The annotation service requests the company identity and groups server for the list of groups the user is a member of; this is usually a request sent to the domain Active Directory, or to AWS IAM or Cognito.
5. The server returns the list of groups the user is a member of.
6. The annotation service queries its database to select all the messages which are of importance for the given groups and the specific URL. Without getting too much into details, it needs to evaluate the exact URL visited against regular expressions stored in the database; this can be done at this step or in the next one, depending of the type of DB used.
7. The DB returns the list of messages.
8. The annotation service performs post-processing, for example to inject colors based on the importance of the annotations, then packs them into a formatted response, which it returns to the user script.
9. The user script creates all the necessary DOM elements to display the messages, in a smart way so their existence is visible to the user, without impairing the browsing experience.

The idea can be extended to allow annotations to target a specific DOM element, so a form input would be grayed with a hint explaining why, and providing a link to the related documentation. The caveat of this option is that a DOM change can break the annotation. To mitigate the issue, the service can periodically load annotated URLs to validate they're still relevant. How would it know the URLs from regular expressions? From previously requested URLs. How would this scale for a large company? I can continue this Q&A for long... pardon my inner engineer, and let's call these unnecessary implementation details at such an early stage of the proposal.

Although I did not picture the annotation creation flow, it can be handled by another script where users click on a floating "Annotate this page" to enter a special mode. Here, users select teams to share the annotation with, a text, and a link to the documentation. They modify the URL captured from the Web browser, to replace parts, and turn it into a validated regular expression. Finally, the annotation service needs a Web interface to review, update, and delete the annotations stored.

Thanks for reading this far

These are my thoughts on documentation, the difficulty of finding it, and where it is most needed. I would love to know if you use other approaches to ensure existing documentation is quick and easy to find — please comment below.

Stay tuned for another article on Arranged ROMs next month. Subscribe to the newsletter to be notified when future articles are published.