The Duplicates Dilemma
With duplicates I mean records in Dynamics 365 (Dataverse) that have the same important values or are exactly the same, but yet they are different records. Duplicate records in Dataverse are quite like a double-edged sword. To prevent duplicates users will get the annoying message Duplicate records found every time they create and update the duplicate record. On the other hand, if you turn duplicates detection off, the database will fill up with duplicates in short time. And that can be annoying as well. In this article you will learn to deal with duplicates, configure and automate deduplication, and prevent duplicates.
?
Deal with duplicates
In Dynamics 365 you can deal with duplicates on three levels.
These measures are not strict. The users can still decide to ignore the duplicate check and continue to save the record or import data.
You can also decide to not deal with duplicates at all and turn off deduplication completely. This is not a normal practice, but in some cases, you would want that. For example, if you particularly wish to keep duplicates because of business units, policies or integrations with other systems.
You can configure the duplicate detection settings at environment level via settings under Data management in the Power Platform admin center.
?
Configure duplicates
Configure deduplication rules
At the same place in the admin center you can configure multiple deduplication rules per table (aka entity or record type). Only published rules are used for detection. Take time to think about which values are considered for detection. For example, matching account names sounds like a good rule, but think that they can be written with or without Inc. or Plc. You could also the organization number instead. You should exclude matching of inactive records to avoid unnecessary duplicate notifications. You can add several criteria, for example the combination of Account name and Account type if you want to have the requirement that a supplier and customer should be two different records. If the column is not required you should Ignore Blank Values, to avoid unnecessary duplicate notifications if that column is not filled out.
?
Setup deduplication jobs
Because deduplication is not enforced to users, you still can get duplicates in Dataverse. Therefore you can setup deduplication jobs per entity in the admin center as well. You can use views as a basis for the job and choose to repeat the job periodically. The creation of the job will take some time and you can have an email send to you when it has finished. The duplication job run includes all the duplicates. You will have to go through them one by one to choose which record should stay and which one should be deactivated. The duplicate record is not removed, only deactivated.
?
Check relationship behavior
Before you start with deduplication check the relationship behavior with the related tables in the Power Apps Make Portal. If you merge two account records and would like to have the related records also moved to the merged record the Merge relationship behavior should be Cascade All. For the related tables that do not need to be moved the Merge relationship behavior should be Cascade None. For example, Contacts and Activities related to an Account will automatically be moved to the merged record, because the standard Merge relationship behaviors are Cascade All. Remember to look at the custom tables because they usually have undesired behavior.
?
Automate deduplication
If you don’t want to give the users the responsibility to merge records, you can Power Automate cloud flows to automate deduplication. You could trigger the flow from creation and update of a record and make it as smart as if it is a human like decision on what should be considered a duplicate record and which record to keep. Read my other article about notifications if you would like to notify users that the record is merged. This prevents a frustrated user to create a record again and again without knowing that it is merged into another record.
In Power Automate you can only merge two records at once, just like a user can in Dynamics. Use the Microsoft Dataverse action Perform an unbound action and choose Merge as Action name. In case of merging account records you can use the following example by replacing the cursive text by Dynamic content from the flow at Target, Subordinate and PerformParentingChecks in the action.
Target:
{
"name": "Name of the target account",
领英推荐
"accountid": "GUID of that account",
"@@odata.type": "Microsoft.Dynamics.CRM.account"
}
Subordinate:????????????????????????????????
{
"name": "Name of the subordinate account",
"accountid": "GUID of that account",
"@@odata.type": "Microsoft.Dynamics.CRM.account"
}
UpdateContent:?????????????????????????????????????????
{
"system name of a column that needs to be updated on the target account": "value for that column, for example from the subordinate record in case the column of the target record doesn’t have a value",
"system name of another column that needs to be updated on the target account": "value for that column",
"system name of Nth column that needs to be updated on the target account": "value for that column",
"@@odata.type": "Microsoft.Dynamics.CRM.account"
}
PerformParentingChecks:
Choose Yes if you want to check if the parent of the two records is the same. Otherwise choose No. If the parent records of the two records are different the flow will fail and will not merge the accounts.
?
Prevent duplicates
So, you’ve set up deduplication rules and are still getting duplicate records? If you haven’t done that already, educate your users to first try to find the record before creating it. You can also help them by adding the columns used for duplicate detection to look up views and putting those fields on the top of the main form. You could also change the quick create form to only include the fields used in duplicate detections to spare the users time when trying to create a duplicate record.
Also educate your developers and integration specialists in preventing duplicate records. Duplicate detection doesn’t work for synchronizations and updates by code. My experience is that duplicate records are often created because of this and harder to deal with afterwards because of dependencies.
There is a very strict solution to prevent duplicates, namely keys. Keys are actually meant to be used for integrations as an alternative way to lookup a record when you don’t have the record GUID. Keys are unique and therefore are duplicate keys are completely prevented in Dataverse. A key can exist of a combination of several columns, for example the combination of Account name and Account type. In this case should the combination of those two columns be unique and creating another record with the same values is not possible. All columns used in the key should be set to required to make this work. This solution works as well in combination with duplicate rules. First the user gets a friendly warning by the rule and after ignoring that an error message pops up not allowing the record to be saved.
?
Prevention is better than having to deal with duplicates afterwards. I hope this article makes you think about the duplication dilemma of which approach is best for which tables in your case. The soft or the hard approach? Automating deduplication or doing it manually? Or maybe you want to allow duplicate records with no detection at all in some cases.