Dynamics 365 Web API Duplicate Detection: The Complete Fix

If you've ever migrated data into Dynamics 365 using the Web API and later discovered hundreds of duplicate records quietly sitting in your system, you're not alone. Dynamics 365 Web API duplicate detection is one of the most misunderstood — and silently dangerous — gaps in the platform. Most teams discover it after the damage is done, spend days cleaning up, and then rebuild the same fix project after project.

At CRMONCE, we've seen this pattern across implementations in manufacturing, financial services, and professional services firms across India and beyond. This post gives you the definitive technical fix — not just a description of the problem — including reusable plugin code, a Power Automate monitoring flow, and a clear decision matrix so you never have to rebuild this from scratch again.

Why Duplicate Detection Silently Fails on Web API Inserts

Here's the uncomfortable truth: when you create records via the Dynamics 365 Web API, duplicate detection rules are bypassed by default. Unlike the interactive UI — where Dynamics prompts users with a duplicate warning dialog — API-based inserts skip this check entirely unless you explicitly instruct the platform to run them.

The root cause is a missing HTTP request header. Every Web API call that creates or updates a record needs the following header to trigger native duplicate detection:

MSCRM.SuppressDuplicateDetection: false

Without this header, the platform assumes you know what you're doing and skips the check. With it set to false, Dynamics will evaluate your active duplicate detection rules and — if a match is found — return an HTTP 412 Precondition Failed error with a body that identifies the conflicting records.

Here's a complete example of a correctly configured Web API POST request to create an Account with duplicate detection enforced:

POST [Organization URI]/api/data/v9.2/accounts
Content-Type: application/json
OData-MaxVersion: 4.0
OData-Version: 4.0
MSCRM.SuppressDuplicateDetection: false

{
  "name": "Contoso Technologies",
  "emailaddress1": "info@contoso.com",
  "telephone1": "+91-40-12345678"
}

And the 412 response when a duplicate is detected:

HTTP/1.1 412 Precondition Failed
{
  "error": {
    "code": "0x80040333",
    "message": "A record was not created or updated because a duplicate of the current record already exists.",
    "innererror": {
      "DuplicateRecordId": "a1b2c3d4-...",
      "DuplicateRuleId": "e5f6g7h8-..."
    }
  }
}

The critical insight: this only works if you have active, published duplicate detection rules configured in Settings → Data Management → Duplicate Detection Rules. If your rules are inactive or incomplete, the header does nothing. Both conditions must be true simultaneously.

The Real-World Risk: Bulk Migration Windows

The most dangerous window for data corruption isn't ongoing integrations — it's the post-migration period when your implementation partner runs bulk imports to seed the system. These imports almost always use the Web API or the Dataverse SDK in batch mode, and in the rush to meet go-live deadlines, the MSCRM.SuppressDuplicateDetection: false header is often omitted.

The result: thousands of records imported cleanly from a technical standpoint, but riddled with duplicates. By the time sales or service teams start working in the system, the damage is invisible — CRM looks fine until reports start showing inflated pipeline values or customers receive duplicate outreach.

Common scenarios where this occurs silently:

A Reusable, Production-Ready Plugin Solution

Native duplicate detection rules have limits — they only support simple field-match conditions and can't handle fuzzy matching, cross-entity logic, or complex business rules. For production systems, you need a plugin-based approach that survives upgrades and migrations.

The following C# plugin runs on the Create and Update messages for the Account entity in a pre-validation stage, giving you full control before any record is committed to the database.

using System;
using System.Linq;
using Microsoft.Xrm.Sdk;
using Microsoft.Xrm.Sdk.Query;

namespace CRMONCE.Plugins
{
    public class AccountDuplicateCheckPlugin : IPlugin
    {
        public void Execute(IServiceProvider serviceProvider)
        {
            var context = (IPluginExecutionContext)
                serviceProvider.GetService(typeof(IPluginExecutionContext));
            var serviceFactory = (IOrganizationServiceFactory)
                serviceProvider.GetService(typeof(IOrganizationServiceFactory));
            var service = serviceFactory.CreateOrganizationService(context.UserId);
            var tracingService = (ITracingService)
                serviceProvider.GetService(typeof(ITracingService));

            // Only act on Create and Update
            if (context.MessageName != "Create" && context.MessageName != "Update")
                return;

            var target = (Entity)context.InputParameters["Target"];

            // Extract fields for duplicate check
            string accountName = target.GetAttributeValue("name");
            string email = target.GetAttributeValue("emailaddress1");
            string phone = target.GetAttributeValue("telephone1");

            if (string.IsNullOrWhiteSpace(accountName) && string.IsNullOrWhiteSpace(email))
                return;

            tracingService.Trace($"Checking duplicates for Account: {accountName}");

            // Build query — match on Name OR Email
            var query = new QueryExpression("account")
            {
                ColumnSet = new ColumnSet("accountid", "name", "emailaddress1"),
                Criteria = new FilterExpression(LogicalOperator.Or)
            };

            if (!string.IsNullOrWhiteSpace(accountName))
                query.Criteria.AddCondition("name", ConditionOperator.Equal, accountName.Trim());

            if (!string.IsNullOrWhiteSpace(email))
                query.Criteria.AddCondition("emailaddress1", ConditionOperator.Equal, email.Trim().ToLower());

            // Exclude the current record on Update
            if (context.MessageName == "Update" && target.Id != Guid.Empty)
            {
                query.Criteria.AddCondition("accountid", ConditionOperator.NotEqual, target.Id);
            }

            // Exclude inactive records
            query.Criteria.AddCondition("statecode", ConditionOperator.Equal, 0);

            var results = service.RetrieveMultiple(query);

            if (results.Entities.Any())
            {
                var duplicate = results.Entities.First();
                tracingService.Trace($"Duplicate found: {duplicate.Id}");

                throw new InvalidPluginExecutionException(
                    PluginHttpStatusCode.PreconditionFailed,
                    $"A duplicate Account already exists with the name '{duplicate.GetAttributeValue("name")}'. " +
                    $"Existing Record ID: {duplicate.Id}. " +
                    $"Please merge or update the existing record instead of creating a new one."
                );
            }

            tracingService.Trace("No duplicates found. Proceeding with save.");
        }
    }
}

Plugin Registration Details:

Why pre-validation? Because it fires before the platform begins any transaction, meaning your duplicate check adds zero risk of partial writes. It also fires regardless of whether the call comes from the UI, Web API, Power Automate, or any SDK integration — making it the most resilient enforcement point in the system.

Power Automate Monitoring Flow for Silent Duplicates

Even with the plugin in place, you need a retroactive monitoring layer — particularly after migrations — to catch duplicates that existed before your rules were active. This Power Automate flow runs on a schedule, identifies potential duplicates using the Dataverse connector, and sends an alert to your CRM admin team.

Flow Architecture:

For the Dataverse query in Step 1, use the following FetchXML expression in a "List Rows" action with a custom FetchXML query:

<fetch aggregate="true">
  <entity name="account">
    <attribute name="name" groupby="true" alias="account_name" />
    <attribute name="accountid" aggregate="count" alias="record_count" />
    <filter>
      <condition attribute="statecode" operator="eq" value="0" />
    </filter>
    <having>
      <condition alias="record_count" operator="gt" value="1" />
    </having>
  </entity>
</fetch>

This gives you a daily safety net that keeps your data stewards proactive rather than reactive — a critical capability in regulated industries like BFSI and healthcare where data quality directly impacts compliance reporting.

Decision Matrix: Native Rules vs. Plugin vs. Azure Functions

Not every scenario needs a plugin. Here's a clear decision framework for IT Managers evaluating their options:

Scenario Recommended Approach Why
Simple field-match rules (exact name, email) Native Duplicate Detection Rules Zero code, survives upgrades, built-in UI integration
Cross-entity matching or conditional logic Custom Plugin (Pre-Validation) Full SDK access, fires on all channels including Web API
Fuzzy matching (phonetic, Levenshtein distance) Plugin + Custom Logic / Azure Cognitive Search Native rules can't handle fuzzy; requires external intelligence
High-volume batch imports (10K+ records/hour) Azure Function with Dataverse API Pre-checks records before insert, avoids plugin sandbox limits
Post-migration cleanup audit Power Automate Monitoring Flow Retroactive, no code deployment required, alerting built-in
Real-time portal/web form submissions Plugin (Pre-Validation) + UI Error Handling Synchronous enforcement with user-friendly error messages

Key Rule of Thumb: Use native rules as your baseline, add plugins for business logic that native rules can't express, and escalate to Azure Functions only when you're consistently hitting plugin execution time limits (2-minute sandbox cap) or processing thousands of records in tight batch windows.

Implementation Checklist for Your Next Project

Stop Rebuilding — Start Owning the Fix

The reason teams keep rebuilding duplicate detection solutions project after project isn't a lack of talent — it's a lack of a reusable, documented pattern that's been battle-tested in production. The plugin above, registered in your managed solution and deployed through your standard ALM pipeline, will survive upgrades, environment copies, and developer turnover.

At CRMONCE, we've packaged this pattern — along with monitoring flows, test scripts, and solution architecture guidance — into our implementation accelerator for clients who can't afford to discover duplicate data problems six months after go-live. If you're planning a Dynamics 365 implementation or migration and want to get data quality right from day one, talk to our team in Hyderabad — we'll make sure you're not rebuilding this for the fourth time.

Source reference: This post expands on duplicate detection gaps originally discussed in the CRM Software Blog, with prescriptive code solutions and a production-ready decision framework not covered in existing community resources.