Best Practices in Documentation: A guide to engineers

This document provides an overview of the best practices in documentation: structure, format, content, style, and grammar. I originally wrote in 2002 for the engineers of Indian Origin. It provides guidelines on documentation, with links to several authoritative sources. After reading this document, you will know the basics of writing, which you can use in writing documents.

Introduction

The purpose of any documentation is to meet the expectations of the users. There are several components to these expectations: format, content, organization, supporting evidence, and relevant resources. Learning to document is an absolute must for anyone desiring to become a professional developer. All things being equal, we prefer open formats, especially the ones that can be converted to other formats. 

Writing well is difficult, but it can be learned. I am going to describe the process of writing that I use. The best way to gain from this article is to read it from the beginning to end once and then follow the links to understand the specifics of each topic. Several of the links are reference materials that can help you while you write your documents.

Establishing the purpose of a document

The purpose of any document is to impart a new ability to the reader. This can be a new understanding, a new technique, or simply new facts. Obviously, documents may have different target audiences and consequently may not be understandable by everyone. To understand what should go into a document, we must establish the specific purpose of the document. The purpose of a document can be established by answering the following questions:Who is the audience for this document? Who benefits from this document?

1. Note, for a marketing document the audience may be non-technical people; for an installation manual, the readers may be technically novice developers; For code documentation, it could be motivated coders.
2. What is the new ability that the reader gains at the end of the reading?
3. What does the document cover? What does it not cover?
4. Who is writing this? Where is the authoritative source for the document? How is this document modified? Who controls it? Who verifies it?

Once these questions are satisfactorily answered, the purpose for the documentation emerges. In fact, this information should be part of the document. If the document is small enough that most of the information can be inferred, a simple preamble may suffice. For example, for a transient document, you do not have to determine the life cycle of the document. You can omit what the document does not cover, if it is implicit. You only need to elucidate those portions of information where there is a possibility of ambiguity.

Creating the outline

Once the purpose of the document, we are ready to create the outline. The outline provides a framework, and guides the flow of the document. It is often best done without the confines of the final format. I prefer to use the text editor of my choice for this purpose.

Here are the steps to create an outline:

1. Write the abstract. This abstract is simply the answers for the four questions listed earlier.
2. Write the headers for the document. If you read the headers, it should tell the complete "story". Sometimes, we may need to add headers to support parallelism and consistency.
3. Expand the headers to include second level and third level headers. Write a single line explaining what is going in each of the headers. Use full sentences when outlining so that they can be used as central ideas for paragraphs.

One note about parallelism: Parallelism is needed both in structure as well as syntax. That is, if there are two items to talk about, the structure should show both of them at the same level. That is, both of them under the same header, or as sub-headers is a possible choice. Or, both of them as bullet items is another choice.

Syntactic parallelism dictates that we use similar sentence or fragment construction for both the items. There are several sources to learn about parallelism: You may google for many links, for example: How to use parallelism in various places.

Once you establish the flow of the document, you can work on the content of the document. If you have written your outline well, developing the content should be easy. As a first step in developing the content, you should define the format of the document.

Choosing a format

Format deals with machine representation and presentation to the reader. Examples are: plain text, Structured Text, XML, MS Word, and so on. Format has a significant role in content development and layout. For example, hypertext lends itself to different kinds of writing than linear writing. In hypertext, related but unimportant information can be relegated to hyperlinks. In linear writing interesting but unnecessary details are ruthlessly eliminated. In plain text, color or font variation cannot be used to outline a structure. Therefore, it is important to decide on a format before developing the content fully.

Here are some guidelines for choosing the formats:

1. All things being equal, an open format such as markdown or html is better than closed format such as MS Word.
2. All things being equal, a format amenable to cross platform tools is better.
3. Documentation that takes advantage of hyperlinks is better.
4. Documents that can be parsed by programs are better.
5. Being able to produce multiple formats is a good thing.
6. However, user familiarity triumphs over other considerations.

Here is my interpretation of the guidelines.

1. Use markdown if you can. Especially, if you intend as a developer documentation.
2. Use Google docs, if you like to collaborate in real time, with comments and resolutions. 
3. Use MS Word, if there is a specific requirement.

At this point, you have the outline of the document and are ready to organize the content. The organization must validate the outline and put the document in a form suitable for filling in the details.

Organizing the material

As the preacher said, "First tell the readers what you are going to say, then say it and in the end, tell them what you said". On the whole, the document organization is similar. Usually, it does the following:

1. It starts by providing the purpose and scope of the document.
2. It provides enough information, including the conventions to understand the document.
3. It is built on a well-structured table of contents.
4. It completes the document using a lot of examples.

By default, I use the following organization, making modifications as needed.

1. Name of the document: Giving proper name is half the game. It lets the document be referred by name in conversations and emails. It has to be descriptive and concise. Most importantly, it has to be searchable. So, “architecture description” is not a good name. “Architecture description for an event system for continuous updates of drug databases” is a good name. You can make it concise, depending on local conventions. Try to get a unique word in, so that searching becomes easier. 
2. Author and Date: Suppose the document gets forwarded to me by somebody else. How do I know who wrote the document? How do I know if this document is recent? Obviously, you need to have the name and date so that the document is better understood. 
3. Document Meta Details: Remember the questions that describe the purpose of the document. The answers go here.
4. Example 1: "This document is written for all the engineers who are starting to code using Python library XXX. It is an introductory document and is maintained at <gitlab location> or Google docs url." By looking at such information, I know whether this document is for me. I know where I can find the latest version, if I need it. I know who is responsible for this document.
5. Example 2: "This document is written for all the project managers to elicit feedback on the process. This is a transient document, and will result in a new document outlining the PM processes. Please send mail to [email protected] for discussions. The discussion period will be over by Nov 10, 2000, and the final document will be placed at: XXX"
6. Introduction: At this point a reader knows the purpose of the document. Next comes the introduction to lay the background and explain the context. It provides a roadmap to the document.
7. Main document: Here you say what you want to say. Be concise. I would like to quote a favorite author of mine who said "If I had more time, I could have written a shorter letter".
8. Conclusion: Tell the readers what you expect them to have understood. Essentially, restate part of the introduction. For example:
9. "In this document you have seen how we solve the problem of periodic rollbacks in the database. For further implementation details, please check XXX document."
10. References: If you are writing for the web, you probably used hyperlinks throughout the document. If not, please use the references section to explain the references used.

When writing for the web the following organization makes sense.

1. Use the inverted pyramid style of writing, just as journalists do: Conclusion first and then the explanation.
2. Write at multiple levels. Each level describes the idea progressively; therefore, the reader is free to stop reading where he wants to, and yet leave with a good understanding of the document.
3. Use bulleted and numbered lists liberally. They draw attention to the points you are making.
4. Use appropriate presentation techniques (italics and bold). Do not use underline, as it is used for hyperlinks. Also, underline is the relic of the days of the typewriters without italics.

Developing the Content

While there is no one way to develop the content, there are several points to consider. 

1. Always provide a context. Do not assume the context. Provide a gentle introduction to what you are going to discuss. If the space is short, dispense with the introduction quickly. Refer to other documents that talk about the history. In well-edited movies, they always use this trick: aerial shot, followed by a zoom. (Unless they want to establish suspense. Let us leave it to creative fiction).
2. Leave the reader with a new capability. Always make sure that the reader leaves with a new capability or a new understanding. Focus on this new capability to make the document succinct.
3. Add enough Examples and evidence. Provide a single example for illustrating any abstract ideas. Build on the same example throughout the document. Imre Lakatos wrote a famous book "Proofs and Refutations" taking a single example and enhancing it throughout the book. (Unfortunately, this suggestion may not work for all the documents, for example, the present one.)
4. Be brief, but only as brief as you can be. There is nothing worse than too little information except too much information. You can leverage hyperlinks to provide additional details in a different document.

Now let us write some specific guidelines on various parts of a document.

Here it is appropriate to give references to help understand the style and grammar issues in writing. For style, you can read the following.

1. Elements of style is a freely available classic. It helps you to understand the right rules. It lets you avoid the traps of too much learning.
2. Style : Toward Clarity and Grace is a wonderful book on style. "Telling me to 'Be clear,' " writes the author, Joseph M. Williams, "is like telling me to 'Hit the ball squarely.' I know that. What I don't know is how to do it." If you are in the same boat, this book gives you examples and explicit instructions to repair turgid prose.
3. You may want to consult the ultimate authority on Web usability: Jakob Nielsen. His website contains such information as how to structure the Web documents.

For grammar, the following on-line resources will help:

1. Fowler's: The King's English: All about usage of words and grammar.
2. Web Grammar: Site links to several available resources on the Web.
3. Line by Line: How to improve your own writing: This book can help you edit your own writing.

Writing a good introduction

Good introduction can establish context, set expectations, and interest the reader. There are several ways to write a good introduction. You could use the introduction to describe the general problem, history of the problem, what your solution is, and how it helps advance the state of the art. You could start with a question that the reader can relate to and introduce the topic in relation to the question. Several of these techniques can be looked up with examples on the Internet.

Do not assume that the reader knows anything about the topic. On the other hand, do not spend all the time teaching the reader basics of the topic.

Developing powerful paragraphs

Since you already have an outline with full sentences that can serve as central ideas, the rest of the work is to expand on these ideas and connect them through good transitional paragraphs. Here are some suggestions while writing the paragraphs:

1. Unity: Always have a central idea for each paragraph. Let each sentence establish or support the idea.
2. Coherence: Each sentence somehow should benefit from the other sentences. For example, they can refer to the established facts logically. Or, they can refer to the established nouns using pronouns.
3. Topic Sentence: While it is not true that every paragraph has an explicit topic sentence, you can make your idea clear by placing it in the beginning of the paragraph and supporting throughout the rest. Or, you can use the topic sentence as a climactic conclusion.

How long should a paragraph be? First rule is to use balanced paragraphs in a document. Try three to four paragraphs per page. Err on the side of having shorter paragraphs. However, do not start a new paragraph unless you have a new topic sentence.

Writing Effective Sentences

Sentences are the building blocks for paragraphs. Here are some principles of writing effective sentences:

1. Strive for variety in sentences. Try varying the sentence length, form, and style, while maintaining logical and linguistic bridges.
2. Make sentences flow from one to another logically. You can get that effect by using the right transitional devices.
3. Use techniques such as coordination and subordination to place the ideas and conclusions in the proper context.

Common Mistakes by people with English as Second Language

Resources:

1. Almost all of these topics are discussed in the classic essay by George Orwell.
2. Plain language in government is a good idea! (Look at the guidelines and examples).
3. If you are writing for the web, take a look at the mailchimp style guide. 
4. For data journalism, please use https://datajournalism.com/read/handbook/one/. 
5. Try the free guides from the plain English initiative. 

People whose primary language is not English, like the author, make several common mistakes. I am going to list some common mistakes along with resources.

1. Avoid wordiness. If you can explain in fewer words, do so.
2. You do not have to sound formal to be taken seriously. Please use the words the readers are likely to understand. 
3. The main purpose of writing is to be read and understood by the reader; not to convince that you know more than the reader. 
4. Use active verbs instead of passive verbs.
5. Avoid cliches like plague :-).

Special guidelines for Indians

Here are some of the mistakes that Indians frequently commit:

1. Do not use adjectives when writing technical documents unless it is objective. For example, do not refer to "excellent product".
2. Do not use direct translations of adjectives from vernacular languages. For example, there are no such things as "small small things".
3. Do not use "myself" as a synonym for "I".
4. Do not use "present continuous verb" unless you need it. For example, "I am having excellent skills" is wrong.
5. Do not leave space before punctuation. For example, "This punctuation is wrong ." -- notice the space before the period.
6. When using bullet items, read the sentence as though there are no bullet items and see what punctuation you need.
7. Understand the tenses well. In particular, the present perfect is not a substitute for past tense.

Conclusion

In this document, I articulated some of the best practices in writing documentation. I also provided links to other on-line resources. Taken together, they provide a comprehensive set of information for creating well written documents. However, to really write good documents, you must also read good prose, and mimic it in your writing.

Appendix-I: File Naming Conventions

Since different systems treat upper and lower cases differently, we need to be careful with naming. For example, Win32 systems can differentiate between upper and lower case names, but several subsystems do not due to legacy issues. On the Internet, the domain name can be in either case (e.g.: https://WwW.AganiTha.Ai/ will resolve well), but the URLs can be case sensitive depending on the Web Server.

I get good results following these guidelines:

1. Use only all lower case names, except when you are using an acronym.
2. Do not use spaces in the names, since Web URL naming and sharing with Unix get messed up. I prefer "-" for spaces. For example, "Marketing Plan.doc" becomes "marketing-plan.doc".
3. Do not use &s, %s, and ?s, since they cannot be a part of a URL without special encoding. They mean different things on the Web and need to be URL-escaped in programming.
4. Use the same conventions even for directories (folders).
5. Use the phone test: trying giving the URL or the filename over the phone. If the other party can access it without looking around, the name passes the phone test.

Appendix-II: Document Formatting Techniques

Writing in Plain Text

Plain text is the greatest common denominator; it is useful for several purposes. It is ubiquitous; it can be read on any machine; it does not need additional software to read; it is simple to edit.

For all these reasons, I generally prefer to use markdown. By using a converter, you can convert a markdown document into plain text, word doc, html, pdf, and other formats. You can even use online converters. These days, many publishers accept books written in markdown.

Writing in DOC format

Doc format is produced by MS Word. Here are a few tips to get a good document going:

1. Use white space sparingly. On the other hand, use white space generously. The idea is to use the white space where necessary. Although there is a school of thought that encourages white space as much as possible, developers typically prefer to use white space to group related concepts together.
2. When making a set of related points, use bulleted items. If you are enumerating them, then use numbered items.
3. Avoid complicated tables, unless you are using them for comparison. Flowing text is as effective.
4. Do not change the font manually for the topics. Choose the "style" from the menu and the font is automatically set.
5. Set the document properties appropriately. Since each document carries the title and author in properties, it can be embarrassing to let the world know whom you borrowed the original document from.
6. You can add other fields such as "reviewed by" as well, so that the complete history is preserved.
7. Use comments feature if you want to send the comments to the author. If you want to edit the document, use the "Track Changes" feature in Word which provides you complete version control over the document.
8. Use Styles effectively. If you want to check if you used style properly, change the style a couple of times, and come back to the original style. If the document looks like the original, that means you did a good job.
9. Start with the right template. There are several shipped with MS Office..
10. Always address those squiggly lines, at least for spelling errors. You can correct the spelling, or make Word "learn" correct spelling. Do the same with the grammar mistakes as well.
11. You are permitted to start the paragraph in the first column. You can use a blank line as a separator, although I recommend using "Paragraph spacing".

For good examples of MS Word documents, look for samples from Microsoft. They are well typeset and visually appealing.

Writing in HTML

Earlier, I used to advocate writing in HTML, for beautiful static website creation. These days, with markdown and tools such as hugo, 11ty, or even github pages, you do not need to use HTML for your documentation.