Dynamics 365 Architecture: Examining Record Status and Flagging Strategies

The status subsystem is a feature in the Dynamics platform that is often overlooked or at least often thought of as insignificant functionality that comes out of the box on every entity. Customizers have varying perspectives on how they ultimately use this feature. In the past, I’ve contemplated the best ways to maximize the value of record status as an information mechanism for users and business automation. Over the years I’ve seen various good and bad use cases. Since then I’ve come to some conclusions on patterns and practices that tend to provide the best overall outcomes.

The first thing you may find interesting about the entity status fields (state code, status code) is that they exist on both system and custom entities. You can’t create an entity without them. Depending on the use case of the entity they may not make sense, and you can leave them along. In other cases, they can prove to be valuable in the context of how an entity will be used in the future. The platform thinks record status is essential enough that it has its own platform SDK message out of the box that business processes can respond to when it changes. Even when creating workflows, you use a separate set state step to change the status. You might wonder if perhaps the status fields are meant to be more significant than what how we typically look at them. Let’s look at what I think about when it comes to record status.

Records Status Strategy

When I consider the status fields I see them as helping users understand the contextual state of a record. I’ve found it best when the record status reasons are something that remains steady and progressive. In consideration of the progressiveness concept, it helps when the status reasons represent a linear stage as opposed to something variable. The way I look at a record status is that it should be relatively static and slow moving. I see it as a journey a record takes from birth until death as it provides its business value to the system (excluding lookup and reference data). It can be very tedious to have a multitude of records where the status is constantly changing back and forth. I’ve noticed in the past that operations requiring a status change tend to take slightly longer than a routine update operation for some reason. In any case, I prefer record status to flow like a progress bar over the lifetime of the record. Record progression helps when I create dashboards, charts or reports that show users their current overall state of affairs.

The status of a record in my mind has a special designation. Even Dynamics chooses to impose its opinion on us based on some of the status fields of system entities such as lead and opportunity. When a lead is qualified for example, the system makes the record read-only. On system entities, we don’t have as much leeway to play with the status which has on occasion been a little frustrating, but sometimes you must make do. We should treat the status field with a degree of reverence.

One of my design paradigms is that active business data within a system should have the notion of a relevant lifespan and the status should give us an indication of where each record is in that business context. When a record reaches its end of life, it should either be deleted or archived. The concept of data finality is a natural part of an archive strategy. Many projects that go live, unfortunately, lack an operational archive strategy deferring it to some future project phase which often never happens. In addition to the typical inactive status reason, I like to add things like a "Ready To Archive" or "To Be Deleted" status reasons. In my mind, the default inactive status means that the record while not active still retains operating value within the current business context.

If we consider that typical chaos when dealing with leads, we'll end up with many duplicates or leads that didn't pan out. Depending on the business rules this data can be set to an appropriate status and dealt with at a later point in time outside of regular business hours. In this scenario, records will ultimately reach their end of life in either "Ready To Archive" or "To Be Deleted" state. I like to think of these additional status indicators as a kind of recycle bin that allows for some action to take place in case of error.

Most system designs only maintain a single concept of inactive which creates the necessity for archive logic to be dependent on an out of the box or custom date field. In more complex scenarios the archival logic may require slightly more sophisticated criteria in which to act. As a matter of functional separation, I don't necessarily prefer to keep the archival logic in the same application that is in charge of actually performing the archival operation itself. Separation of archive logic from the process helps to prevent the spread of duplicate business logic into other areas of the system. It just depends on the situation and context of the application being built.

If you find that your record status needs to fluctuate often or can require multiple states at the same time it is typically a sign that you may need to segment out separate flag field representing an independent status that can change more frequently. It could also be the case that if a status has no linear characteristic, it may be a good candidate as a flag field as opposed to a status option.

Flag Field Status Strategy

The flag field strategy isn't a new concept. Flag fields are used in many cases to indicate a state of some kind. Where there seems to be some confusion at times is what should be a status versus what should be a flag.

The typical properties that I look for to determine if something should be a flag are as follows:

• The status can change throughout the record lifespan
• The status isn't mutually exclusive of the entity status
• The status has no relevance to the staged progression of the record

What you may not have thought too deeply about are the various kind of flags you can create and when it makes sense to use one versus another. Let's take a look at what those options are and what we can do with them.

Flags Types

Boolean flags

On/off flags are the traditional staple of flag fields. In the simplest of flags, it's either on or off like a light switch. Nothing special here and we can use a Boolean (two option) field to accomplish this with ease. Just make sure you don't find out later there's a third option to represent.

Date Flags

Date fields make for beneficial flag fields since they can convey two pieces of information at the same time. The first piece of information is the fact that it has a value or not giving an on/off flag attribute and the second is the date value itself. For flags that have a temporal significance, this is the way to go. For example, if you want to indicate that a new lead in the system needs attention because it's been sitting untouched for too long, then a date flag works well. Flag fields are often used in conjunction with process timeout strategy workflows (which should be used with great care).

Option Set Flags

Similar to date flags, option set flags can convey two pieces of information at the same time. The difference with options sets in more recent time is with the introduction of the multi-select option set fields this field type can convey much more than two pieces of information. We won't go so far as to say infinite possibilities, but a fair number for sure. While this can be used very effectively in more complex scenarios, I'd be careful not to take this too far. If I'm not mistaken the multi-select option set is stored as a string, and this concerns me from a performance perspective so be careful in enterprise scenarios.

Compound Flags

A compound flag is when two or more flag fields are used together to indicate a state. A typical example of this is a start date and end date to indicate whether a record should be considered in the context of some operation. Another compound design pattern is an on/off field to indicate if other flag fields are relevant. In any case, I try not to go too crazy with more than two fields. I’ve never personally used a 3-field flag, but I wouldn’t be surprised if there’s some crazy scenario out there where it may have been necessary for someone.

Record Status Anti-Patterns

Custom Status Pattern

A mistake I've seen by some designers is to create a completely custom status system and bypassing the out of box status functionality altogether. I was on a project once where the lead designer decided to create a custom status field based on start and end dates across multiple entities. In this system, we ignored the built-in system status fields entirely. The challenge was to determine the records that were relevant for a particular business process we had to first perform a series of date calculations on each record to determine if it was valid. This design pattern was an occurrence of a compound flag pattern as a replacement to the system status feature. Imagine you are trying to operate on relevant records, and you have to query the start and end dates on all your entities. Needless to say, this was a nightmare when it came to creating business processes that operated against relevant records.

Even with the necessity of the start and end date fields, in this case, a "better approach" would have been to use the start and end date fields to set the status field of each record. A process could have been created that would ensure records that fell out of their respective date range would represent the proper state for all other business processes that needed that information.

Volatile Status Pattern

I was on a project where we had a lead record status that flailed about like a flag on a windy day. The challenge with this was the fact that too many states needed to be represented all in a single status field. The interesting thing, in this case, is that competing business processes attempted to reset the status according to their own business rules. This scenario required one or more of the status reasons to be split out from the record status and represented as a Boolean flag. This small design change would alleviate the competitive nature of the business processes because they would effectively have their own status field to maintain. Going back to the notion of record lifespan progression if a record status has to change back and forth something probably needs to come out of the status field and represented another way.

Mass Workflow Update Pattern

Another design consideration when dealing with status and flag fields is the use of per record workflows as a maintenance mechanism. While this can make sense, it also can lead to load spikes due to the number of processes spawned. For example, if you have an entity with millions of records and you are running a job that kicks off a workflow on tens of thousands of records each day, this isn’t the most efficient way to maintain the data especially if this occurs during operating hours. A better approach from a system performance perspective is to create a separate external process that queries the data where the record status needs to change and then updates them in mass. Not only will this save your async server some load it will most likely run much faster.

Status Usage in Dashboards and Charts

Another consideration when designing your record status is how they can be used in dashboards and charts. If you think ahead and begin with the end in mind, you can make design decisions that will be in line with the direction the system is already heading. Most often as developers we don't honestly consider the end usage scenarios as we create functionality earlier on and end up having to refactor later.

Contemplation of Status Relationship Chains

When the status of a record is going to change you have to take into consideration its children. Like any good parent, you can’t have the parent go away and leave its children behind. If you’re going to archive or delete a record, its children need to be accounted for in the context of that operation. There can be exceptional circumstances where a parent can’t be archived or removed due to the significance of a child in the business context. These are things you must watch out for when designing your business functions.

Resources

• How To: Use Record Status in Microsoft Dynamics CRM
• Define Custom State Model Transitions