In this article, we are going to share the different methods on how to remove duplicates from an array in C#. In addition, we are going to compare these methods to check their performance.<\/p>\n
Arrays contain multiple values of the same type. It\u2019s a common occurrence to have some duplicate elements in an array. Although, in some cases, we want the elements to be unique.<\/p>\n
We can achieve this manually or by using some of the .NET inbuilt methods.<\/p>\n
Let’s start.<\/p>\n
To begin with, let\u2019s create an array<\/a> containing some duplicate values. We will use this array for all the methods defined:<\/p>\n For the duplicate values in our array, we can use the LINQ This is done by invoking the HashSet is an unordered collection of a specific type. We can remove duplicates from an array by casting it to a We use Similarly, one of the constructors for By default, the compiler will eliminate the duplicate values while constructing the HashSet.<\/p>\n Another way to remove duplicate elements in an array is by combining the We use We can loop through the array and remove the items that occur more than once. We can use the initial array but shift the elements in the array once we hit a duplicate or use a temporary collection to hold the unique elements:<\/p>\n The if statement in the second loop compares all elements in the array with each other.<\/p>\n Inside the if block, we iterate through the entire array from the duplicate element index and shift the elements to one position left. The order of elements in our array changes during shifting. Once we are done with our iteration, the last element becomes obsolete. Hence, we will reduce the size of the array by 1 and cut it when the iteration is over.<\/p>\n This is far from optimal, but we’re going to see what the benchmark show at the end.<\/p>\n We can use a swapping algorithm instead of the shifting one (the one we previously used) to increase the performance of our code:<\/p>\n This is a similar code to the previous one, but as you will see in the benchmark a faster one. Thanks a lot, James Curran for suggesting this (you can read more about it in the comment section).\u00a0<\/strong><\/p>\n We can also use a dictionary within the for loop to hold the unique elements in our array:<\/p>\n The dictionary keys represent all the unique elements in the array. We’re utilizing one of the important properties of the Dictionary collection – all the keys in the dictionary must be unique.<\/p>\n If we just modify the last code line to use the Keys property, we can improve performance:<\/p>\n We add the implementation in a different method so we could compare them in our benchmark test.<\/p>\n Similarly, we can also construct a recursive method to remove duplicates in an array:<\/p>\n While the function is calling itself, we are using a list to hold all unique values in the array. During this process, we only add values missing to the list. The function stops calling itself when Now, we are going to compare the performance of the different methods discussed so far:<\/p>\nstring[] arrayWithDuplicateValues = new string[] { \"Lovely\", \"Lovely\", \"Boy\", \"Boy\" };<\/code><\/p>\nDistinct()<\/code> method to remove them:<\/p>\n\/\/ before: {\"Lovely\", \"Lovely\", \"Boy\", \"Boy\" }\r\nvar distinctArray = arrayWithDuplicateValues.Distinct().ToArray(); \/\/ after: {\"Lovely\", \"Boy\" }<\/pre>\nDistinct()<\/code> method on the array. The response is an enumerable object. ToArray()<\/code> converts the response to an array with unique values.<\/p>\n<\/a>Remove Duplicates Using HashSet<\/h2>\n
HashSet<\/code> and then converting it back to an array:\u00a0<\/p>\nvar distinctArray = arrayWithDuplicateValues.ToHashSet().ToArray();<\/code><\/p>\nToHashSet()<\/code> on the array to remove the duplicate values. The ToHashSet()<\/code> method returns an enumerable collection. ToArray()<\/code> will alter the response to an array.<\/p>\nUsing the HashSet Constructor<\/h3>\n
HashSet<\/code> accepts any collection of values of a particular type:<\/p>\nvar hashSet = new HashSet<T>(arrayWithDuplicateValues); \r\nvar distinctArray = hashSet.ToArray();<\/pre>\n
<\/a>Remove Duplicates Using LINQ GroupBy and Select Method<\/h2>\n
GroupBy()<\/code> and Select()<\/code> methods:<\/p>\nvar distinctArray = arrayWithDuplicateValues.GroupBy(d => d).Select(d => d.First()).ToArray();<\/code><\/p>\nGroupBy()<\/code> to organize the elements into groups, and then we select the first item from each of the groups and convert them to an array.\u00a0<\/p>\n<\/a>Loop Through the Array and Remove Elements Manually<\/h2>\n
var size = arrayWithDuplicateValues.Length;\r\n\r\nfor (int i = 0; i < size; i++)\r\n{\r\n for (int j = i + 1; j < size; j++)\r\n {\r\n if (arrayWithDuplicateValues[i] == arrayWithDuplicateValues[j])\r\n {\r\n for (int k = j; k < size - 1; k++)\r\n {\r\n arrayWithDuplicateValues[k] = arrayWithDuplicateValues[k + 1];\r\n }\r\n j--;\r\n size--;\r\n }\r\n }\r\n\r\n}\r\n\r\nvar distinctArray = arrayWithDuplicateValues[0..size]; <\/pre>\nUsing Swapping Instead of\u00a0 Shifting Elements to Remove Them<\/h2>\n
public string[] IterationAndSwappingElements(string[] arrayWithDuplicateValues)\r\n{\r\n var size = arrayWithDuplicateValues.Length;\r\n\r\n for (int i = 0; i < size; i++)\r\n {\r\n for (int j = i + 1; j < size; j++)\r\n {\r\n if (arrayWithDuplicateValues[i] == arrayWithDuplicateValues[j])\r\n {\r\n size--;\r\n arrayWithDuplicateValues[j] = arrayWithDuplicateValues[size];\r\n j--;\r\n }\r\n }\r\n }\r\n\r\n return arrayWithDuplicateValues[0..size];\r\n}<\/pre>\nUtilizing the Dictionary Key to Remove Duplicates<\/h2>\n
var dic = new Dictionary<string, int>();\r\n\r\nforeach (var s in arrayWithDuplicateValues)\r\n{\r\n dic.TryAdd(s, 1);\r\n}\r\n\r\nvar distinctArray = dic.Select(x => x.Key.ToString()).ToArray();<\/pre>\nImproving This Code<\/h3>\n
public string[] IterationWithDictionaryOpt(string[] arrayWithDuplicateValues)\r\n{\r\n var dic = new Dictionary<string, int>();\r\n\r\n foreach (var s in arrayWithDuplicateValues)\r\n {\r\n dic.TryAdd(s, 1);\r\n }\r\n\r\n return dic.Keys.ToArray();\r\n}<\/pre>\nRemoving Elements by Using Recursion<\/h2>\n
public string[] RecursiveMethod(string[] arrayWithDuplicateValues, List<string>? mem = default, int index = 0)\r\n{\r\n if (mem == null)\r\n {\r\n mem = new List<string>();\r\n }\r\n\r\n if (index >= arrayWithDuplicateValues.Length)\r\n {\r\n return arrayWithDuplicateValues;\r\n }\r\n\r\n if (mem.IndexOf(arrayWithDuplicateValues[index]) < 0)\r\n {\r\n mem.Add(arrayWithDuplicateValues[index]);\r\n }\r\n\r\n RecursiveMethod(arrayWithDuplicateValues, mem, index + 1);\r\n\r\n return mem.ToArray();\r\n}<\/pre>\nindex<\/code> is equal to the length of the array.<\/p>\n<\/a>Performance Metrics<\/h2>\n
| Method | Mean | Error | StdDev | Rank | Gen 0 | Allocated |\r\n|--------------------------- |------------:|----------:|----------:|-----:|-------:|----------:|\r\n| IterationAndSwapping | 828.1 ns | 2.83 ns | 2.65 ns | 1 | 0.0095 | 40 B |\r\n| IterationWithDictionaryOpt | 2,968.5 ns | 3.71 ns | 2.89 ns | 2 | 0.0648 | 280 B |\r\n| IterationWithDictionary | 3,051.0 ns | 5.13 ns | 4.55 ns | 3 | 0.0992 | 424 B |\r\n| ConversionToHashSet | 3,143.7 ns | 17.67 ns | 13.80 ns | 4 | 1.2131 | 5,080 B |\r\n| HashingMethod | 3,251.4 ns | 11.96 ns | 11.18 ns | 5 | 1.2131 | 5,080 B |\r\n| DistinctLINQMethod | 3,492.4 ns | 66.41 ns | 62.12 ns | 6 | 1.2283 | 5,144 B |\r\n| RecursiveMethod | 5,740.9 ns | 22.60 ns | 21.14 ns | 7 | 1.9302 | 8,088 B |\r\n| GroupByAndSelectLINQMethod | 7,188.4 ns | 60.13 ns | 50.21 ns | 8 | 1.1826 | 4,976 B |\r\n| IterationAndShifting | 18,046.1 ns | 172.29 ns | 152.73 ns | 9 | - | 40 B |<\/pre>\n