In this article, we will learn about situations where we may need to convert a file into a byte array. Additionally, we will learn two ways to perform the conversion in C#.<\/p>\n
If you want to learn how to convert a byte array to a file<\/strong>, check out our convert byte array to a file<\/a> article.<\/p>\n So let’s start.<\/p>\n In C#, a byte array is an array of 8-bit unsigned integers (bytes). They are often used to represent more complex data structures, such as text, images, or audio data.<\/p>\n There are several use cases in which we want to convert a file to a byte array, some of them are:<\/p>\n Generally, a byte array is declared using the This creates a byte array with 50 elements.<\/p>\n Each of our examples in this article is exercised through the unit test project. The unit tests will create temporary files for the tests and delete them when all the tests have finished executing. For more about unit testing with xUnit<\/strong> and the When converting a file into a byte array it is important to remember that if the file size exceeds <\/strong> One of the most straightforward methods for converting a file is the This method opens the file, reads all of the contents into an array, and returns it.<\/p>\n If we are working with asynchronous code, we can use the A second technique we can consider for converting a file to a byte array is to stream the file into a The code here is pretty straightforward. We open a As with the Both of the previous techniques involved creating new arrays (or in the case of the For a deeper dive into Our first option is to rent an array, load the file data into it, and return the rented array:<\/p>\n Here, we first load the file information and ensure the length is within the bounds of Next, we open a It is important to note here that to prevent memory leaks, the caller of this method must return the rented array to the <\/strong> As with our previous methods, the conversion method can also be written asynchronously:<\/p>\n Note here that because we are in an asynchronous method, we must use a Similar to our technique using First, we need to add the package to our project:<\/p>\n Here we begin by creating our Next, we create a As with our previous examples, we can also write this method as As we mentioned at the beginning of our article, sometimes it is not possible to load the entire file into memory at one time. When a file’s length exceeds What is a Byte Array?<\/h2>\n
\n
byte[]<\/code> syntax:<\/p>\nbyte[] byteArray = new byte[50];<\/code><\/p>\nSome Initial Setup<\/h3>\n
IClassFixture<T><\/code> be sure to check out our article “Differences Between NUnit, xUnit and MSTest<\/a>“<\/p>\nArray.MaxLength<\/code>, we will not be able to read all of the contents at once. Attempting to do so will result in an exception. <\/strong>In the last section of our article, we will review a technique for solving this problem by working on the file in chunks.<\/p>\nUse ReadAllBytes to Convert a File<\/h2>\n
File.ReadAllBytes()<\/code> static method:<\/p>\npublic void GivenFile_WhenConvertingUsingReadAllBytes_ThenReturnsCorrectContent()\r\n{\r\n var bytes = File.ReadAllBytes(fixture.SmallTestFile);\r\n\r\n bytes.Should().BeEquivalentTo(fixture.SmallTestFileExpectedBytes);\r\n}<\/pre>\nFile.ReadAllBytesAsync()<\/code> method to accomplish the same goal:<\/p>\npublic async void GivenFile_WhenConvertingUsingReadAllBytesAsync_ThenReturnsCorrectContent()\r\n{\r\n var bytes = await File.ReadAllBytesAsync(fixture.SmallTestFile);\r\n\r\n bytes.Should().BeEquivalentTo(fixture.SmallTestFileExpectedBytes);\r\n}<\/pre>\nConvert Using MemoryStream<\/h2>\n
MemoryStream<\/code>. While this results in additional memory pressure, if we already have an open FileStream<\/code> object, we can write the stream directly into a MemoryStream<\/code> and then return the result as an array:<\/p>\npublic static byte[] ConvertUsingMemoryStream(string filePath)\r\n{\r\n using var fs = File.OpenRead(filePath);\r\n using var ms = new MemoryStream(DefaultBufferSize);\r\n\r\n fs.CopyTo(ms);\r\n\r\n return ms.ToArray();\r\n}<\/pre>\nFileStream<\/code> for reading. Next, we initialize a new MemoryStream<\/code> using a DefaultBufferSize<\/code> constant (4096 for our example) as our initial stream capacity to reduce the number of reallocations in the MemoryStream<\/code> while processing the file. Once our initial setup is done we simply copy the FileStream<\/code> to our MemoryStream<\/code> and then convert the MemoryStream<\/code> to an array.<\/p>\nFile.ReadAllBytes()<\/code> method, we can also perform a conversion using a MemoryStream<\/code> in an asynchronous fashion:<\/p>\npublic static async Task<byte[]> ConvertUsingMemoryStreamAsync(string filePath)\r\n{\r\n await using var fs = File.OpenRead(filePath);\r\n await using var ms = new MemoryStream(DefaultBufferSize);\r\n\r\n await fs.CopyToAsync(ms);\r\n\r\n return ms.ToArray();\r\n}<\/pre>\nConvert a File Using a Rented Byte Array<\/h2>\n
MemoryStream<\/code> potentially multiple new arrays). With the introduction of ArrayPool<T><\/code> to C#, we have some additional options that will allow us to load the contents of a file into memory without additional memory pressure. This proves especially useful when processing a large number of files and only needing the data in memory temporarily.<\/p>\nArrayPool<T><\/code>, don’t miss our article: “Memory Optimization With ArrayPool in C#”<\/a>. Now, let’s see how we can optimize our memory usage while loading files into a byte array.<\/p>\nRead File Into Rented Byte Array<\/h3>\n
public static (byte[] rentedArray, int length) ConvertToPooledArray(string filePath)\r\n{\r\n var fileInfo = new FileInfo(filePath);\r\n ArgumentOutOfRangeException.ThrowIfGreaterThan(fileInfo.Length, Array.MaxLength, \"File length\");\r\n\r\n var length = (int)fileInfo.Length;\r\n var array = ArrayPool<byte>.Shared.Rent(length);\r\n var span = array.AsSpan(0, length);\r\n\r\n using var fs = fileInfo.OpenRead();\r\n fs.ReadExactly(span);\r\n\r\n return (array, length);\r\n}<\/pre>\nArray.MaxLength<\/code>. Next, we cast the file length value to an int (which we know is safe because of our previous validation). Using this length we rent an array from the shared ArrayPool<byte><\/code>, and then take a Span<byte><\/code> of the appropriate length over the rented array. We take a<\/strong> Span<\/code> over the array because arrays rented from the <\/strong>ArrayPool<\/code> may be longer than the requested size<\/strong>.<\/p>\nFileStream<\/code> for reading and read the file contents into the span. Finally, we return a ValueTuple<\/code> containing the rented array and the length of the file. We return the length because a rented array may be larger than the requested size.<\/p>\nArrayPool<\/code>:<\/p>\nArrayPool<byte>.Shared.Return(rentedArray);<\/code><\/p>\npublic static async Task<(byte[] rentedArray, int length)> ConvertToPooledArrayAsync(string filePath)\r\n{\r\n var fileInfo = new FileInfo(filePath);\r\n ArgumentOutOfRangeException.ThrowIfGreaterThan(fileInfo.Length, Array.MaxLength, \"File length\");\r\n\r\n var length = (int) fileInfo.Length;\r\n var array = ArrayPool<byte>.Shared.Rent(length);\r\n var memory = array.AsMemory(0, length);\r\n\r\n await using var fs = fileInfo.OpenRead();\r\n await fs.ReadExactlyAsync(memory);\r\n\r\n return (array, length);\r\n}<\/pre>\nMemory<byte><\/code> rather than a Span<byte><\/code>. Span<\/code> is a ref struct<\/a>, and so it cannot be used in an asynchronous context.<\/p>\nConvert to Byte Array Using ArrayPoolBufferedWriter<\/h3>\n
MemoryStream<\/code>, we can make use of ArrayPoolBufferedWriter<\/code> which is available in the CommunityToolkit.HighPerformance package<\/a>.<\/p>\ndotnet add package CommunityToolkit.HighPerformance<\/code><\/p>\nArrayPoolBufferedWriter<\/code> behaves much like MemoryStream<\/code>, but instead of allocating new arrays on the heap when the internal buffer is exceeded, it rents them from the ArrayPool<\/code>. This helps to prevent memory pressure and heap fragmentation that can result from\u00a0multiple temporary array allocations:<\/p>\npublic static byte[] ConvertUsingPooledWriter(string filePath)\r\n{\r\n using var writer = new ArrayPoolBufferWriter<byte>(DefaultBufferSize);\r\n using var stream = writer.AsStream();\r\n\r\n using var fs = File.OpenRead(filePath);\r\n fs.CopyTo(stream);\r\n\r\n return writer.WrittenSpan.ToArray();\r\n}<\/pre>\nArrayPoolBufferWriter<\/code> with our DeafultBufferSize<\/code>. We use an initial buffer size to help avoid reallocations. While in this case, we are not concerned about the memory allocations as a result of exceeding the buffer, we still want to avoid the copying required when the buffer is exceeded.<\/p>\nStream<\/code> from our buffered writer to enable stream operations on it. Following that we create a FileStream<\/code> to read the contents of our file, and stream them into our buffered writer. Finally, we return the buffered contents as a new array.<\/p>\nasync<\/code>:<\/p>\npublic static async Task<byte[]> ConvertUsingPooledWriterAsync(string filePath)\r\n{\r\n using var writer = new ArrayPoolBufferWriter<byte>(DefaultBufferSize);\r\n await using var stream = writer.AsStream();\r\n\r\n await using var fs = File.OpenRead(filePath);\r\n await fs.CopyToAsync(stream);\r\n\r\n return writer.WrittenSpan.ToArray();\r\n}<\/pre>\nConverting a Large File to a Byte Array in Chunks<\/h2>\n
Array.MaxLength<\/code>, we have to look for an alternative method to deal with the file contents. Here we present one approach making use of IAsyncEnumerable<\/a> to return the large file in chunks:<\/p>\npublic static async IAsyncEnumerable<byte[]> ConvertInChunksMemoryMapped(string filePath, int chunkSize,\r\n [EnumeratorCancellation] CancellationToken cancellationToken = default)\r\n{\r\n ArgumentOutOfRangeException.ThrowIfGreaterThan(chunkSize, Array.MaxLength);\r\n\r\n var rentedBuffer = ArrayPool<byte>.Shared.Rent(chunkSize);\r\n try\r\n {\r\n var memory = rentedBuffer.AsMemory(0, chunkSize);\r\n\r\n var fileLength = new FileInfo(filePath).Length;\r\n using var mm = MemoryMappedFile.CreateFromFile(filePath);\r\n await using var accessor = mm.CreateViewStream(0, fileLength);\r\n\r\n int bytesRead;\r\n while ((bytesRead = await accessor.ReadAsync(memory, cancellationToken)) != 0)\r\n yield return memory[..bytesRead].ToArray();\r\n }\r\n finally\r\n {\r\n ArrayPool<byte>.Shared.Return(rentedBuffer);\r\n }\r\n}<\/pre>\n