Microsoft’s Resilient File System (ReFS) was introduced with Windows Server 2012. ReFS is not a direct replacement for NTFS, and is missing some underlying NTFS features, but is designed to be (as the name suggests) a more resilient file system for extremely large amounts of data.
ReFS maintains high degree of compatibility with NTFS while providing enhanced data verification and auto-correction techniques as well as an integrated end-to-end resiliency to corruptions especially when used in conjunction with the storage spaces feature.
One example that springs to mind for extremely large amounts of data is Exchange databases, which have a theoretical maximum of 16TB per database file. In reality of course we tend to run much smaller database sizes. Microsoft reportedly uses 2TB as their ideal maximum database size in Exchange Online. In customer environments I’ve worked in the trend is towards more, smaller databases such as 200GB maximum for single server scenarios, and 500GB maximum for multi-server high availability (database availability group) deployments. Even falling well short of the 16TB maximum a 2TB or even 500GB database file is still “big”.
Support for ReFS with Exchange Server
From Exchange Server 2013 and upwards (which includes Exchange Server 2016 today) Microsoft supports the use of ReFS for Exchange servers, and in fact they now recommend it as the preferred file system for Exchange Server 2016, within the following guidelines.
For Exchange Server 2013:
- ReFS is supported for volumes containing Exchange database files, log files, and content index files.
- ReFS is not supported for volumes containing Exchange binaries (the program files).
- ReFS is not supported for volumes containing the system partition.
- ReFS data integrity features must be disabled for the database (.edb) files or the entire volume that hosts database files.
- Hotfix KB2853418 must be installed.
- For Windows 2012, the following hotfixes must be installed:
This means that you should continue to use NTFS for your operating system and Exchange Server 2013 installation volume, but you can consider using ReFS for the volumes hosting Exchange databases, log files, and index files.
For Exchange Server 2016:
- ReFS is supported for volumes containing Exchange database files, log files, and content index files.
- ReFS is not supported for volumes containing Exchange binaries (the program files).
- ReFS is not supported for volumes containing the system partition.
- ReFS data integrity features are recommended to be disabled.
- For Windows 2012, the following hotfixes must be installed:
This means that you should continue to use NTFS for your operating system and Exchange Server 2016 installation volume, and it is recommended ReFS for the volumes hosting Exchange databases, log files, and index files. Note also that at this time there is no guidance directing you to disable ReFS data integrity features or install additional hotfixes when using ReFS with Exchange Server 2016. Updated the guidance above with some additional information that is expected to be added to TechNet soon.
Creating an ReFS Formatted Volume
In Windows Server 2012 or R2 during the New Volume Wizard when you get to the step for configuring File System Settings change the file system from NTFS to ReFS.
However, using the New Volume Wizard does not give you the option to disable data integrity at the volume level. To set it at the volume level itself use PowerShell when configuring new volumes. On my demo server disk 3 has been added to the server and is ready to initialize and format with ReFS.
PS C:\> Get-Disk
Number Friendly Name OperationalStatus Total Size Partition Style
------ ------------- ----------------- ---------- ---------------
2 Microsoft Virtual Disk Online 100 GB GPT
1 Microsoft Virtual Disk Online 100 GB GPT
3 Microsoft Virtual Disk Online 100 GB RAW
0 Virtual HD ATA Device Online 95 GB MBR
PS C:\> Get-Disk 3 | Initialize-Disk -PartitionStyle GPT -PassThru | New-Partition -UseMaximumSize | Format-Volume -FileSystem REFS -AllocationUnitSize 65536 -NewFileSystemLabel Volume3 -SetIntegrityStreams $false
Confirm
Are you sure you want to perform this action?
Warning, all data on the volume will be lost!
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "Y"): y
DriveLetter FileSystemLabel FileSystem DriveType HealthStatus SizeRemaining Size
----------- --------------- ---------- --------- ------------ ------------- ----
Volume3 ReFS Fixed Healthy 99.31 GB 99.81 GB
Additional Considerations
When you are deploying an Exchange 2016 database availability group and using Autoreseed the disk reclaimer needs to know which file system to use when formatting spare disks.
C:\> Set-DatabaseAvailabilityGroup NameOfYourDAG -FileSystem ReFS
For Exchange Server 2013 DAGs, manually format the spare volumes with ReFS.
Summary
If you are running Exchange Server 2013 on Windows Server 2012/R2 today and you are using NTFS volumes then the effort to provision new ReFS volumes and migrate your data likely exceeds the benefits of ReFS over NTFS for that scenario. If you’re deploying a new Exchange Server 2013 server today you can consider using ReFS within the support guidelines mentioned in the article above.
For Exchange Server 2016 deployments ReFS is now the recommended file system, so you should plan any new deployments to use ReFS volumes for Exchange databases, log files, and content indexes.
References:
- Exchange 2013 storage configuration options
- Exchange 2016 Preferred Architecture
- Exchange Storage for Insiders: It’s ESE (Ignite video)
Thanks to Ross Smith IV from Microsoft for helping to clarify some of the missing and fragmented information on this topic.
I have a few Exchange 2016 server on Windows 2016 with ReFS disks mounted as mountpoints on an NTFS (4K blocksize) formatted disk. We have had some Microsoft guys here saying this is an issue and we should format the mountpoint disk with ReFS. Haven’t been able to find any information regarding this other than them saying it. They couldn’t point me to any article regarding it either. Does anyone know if this is true?
we are using REFS file system for Exchange server 2016 , is there any recommended hotfix or update for windows server 2012 R2.
we are continuously facing issue with content index state failed.
Verified FileIntegrity is disabled on all volume ( GPT)
You need to mention that ReFS is not supported on RAID or SAN architecture. We only talking JBOD or S2D here.
I came here to make this exact same comment!
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
I can update the article if someone has a source for that. I don’t see it on the Exchange storage config/options page on TechNet.
Have you got Craig / Aymen about this point?
Regards
https://docs.microsoft.com/en-us/windows-server/storage/refs/refs-overview#supported-deployments
Basic disks include local non-removable direct-attached via BusTypes SATA, SAS, NVME, or RAID.
If I try to create the mailbox database on ReFS formated drive then Exchange 2016 shows given below error. There is no issue if I try to create the mailbox databse on NTFS formated drive. Both drives have the same perssmissions. Is anyone else face same issue?
Both Adinistrator and System have full permissions.
There was a problem creating directory “D:\Program Files\Microsoft\Exchange Server\V15\Mailbox\REFSDB01” on server “TL-Main-EX01.thevlab.com”. In order to ensure that the Exchange data is kept secure, please create the directory, if it does not already exist, and configure directory permissions so that “SYSTEM” and the local “Administrators” group have “Full Control” for the folder, subfolders, and files. It is recomended that permission inheritance be broken so that only “SYSTEM” and “Administrators” have access to the directory.
Hi Paul,
2 questions.
1. If we change the set up of one of our Exchange 2013 servers in our DAG to use Refs on our DB and log drive/s and the other servers in the DAG remained using NTFS, will this cause any problems.
I am 90% sure it wouldn’t really matter, but I don’t know for sure.
2. The server that we want to make the changes to, which will involve formatting the disk. Whats the best way to do this.
My thoughts are to put the server into maintenance mode, remove the databases and logs.
Then format the drives to use ReFS, then reseed the DBs from another Exchange server, then bring out of maintenance mode.
You opinion would be appreciated.
Thank you
When running these updates they fail “The update is not applicable to your computer”.
Have these updates been superseded?
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
Which operating system are you running?
Hi,
I’m running Server 2012 R2…
Thanks
Hello Paul,
I´m running my Exchange 2016 on Windows server 2016. I´ve noticed that the info and hotfixes are intented for windows server 2012.
Do I need any particular KB for Windows 2016 for may DAG 2016 to work ok with Refs? or any particular warning before I move into it?
Thanks in advanced!
Hi There,
I was wondering if I could get your thoughts on the use of REFS when doing a deployment on shared storage which has RAID and some other healing smarts built in to the file system. A couple examples would be Nutanix or even just more of a traditional Hyper-V deployment with shared storage.
I have done some research and found a couple sites that did benchmarks which found ~7-10% decrease in IOPS from going with REFS instead of NTFS.
Examples:
http://www.joshodgers.com/2016/07/10/storage-performance-refs-vs-ntfs/
https://eightwone.com/2015/05/16/exchange-file-system/
They didnt argue that REFS didn’t have its place for the preferred architecture and JBOD w/ Autoreseed but reading between the lines it seemed they felt it was unnecessary when going with a virtualized deployment that leverages shared storage that has some protection mechanisms built in. Not sure if there is any other compelling reason to use REFS like upcoming features that would leverage it…
Just curios on your thoughts?
same here.. anyone has news about ReFS? I cant find something about DPM in System Center 2016 is supporting ReFS or not. Maybe Azure DPM?
“Support for protecting file server using Resilient File System (ReFS).”
What’s new in System Center 2012 SP1 – DPM
https://technet.microsoft.com/en-us/library/dn296605(v=sc.12).aspx
Does anyone know when Azure DPM will support refs for backup on exchange 2016?
Thanks
Why did not you assign a drive letter to the ReFS volume?
Why use drive letters when you have mountpoints?
Hamada,
If you want to assign a drive letter, use the following:
Get-Disk 3 | Initialize-Disk -PartitionStyle GPT -PassThru | New-Partition -DriveLetter F -UseMaximumSize | Format-Volume -FileSystem REFS -NewFileSystemLabel DB-01 -SetIntegrityStreams $false
Guyz Just sharing for your kind information!!!
Visit the below URL regarding the ReFS
https://technet.microsoft.com/en-us/library/ms.exch.setupreadiness.win2k12urefsupdatenotinstalled(v=exchg.160).aspx
Very nice article! Crystal clear, straigth to the point! Thanks
What commands could I run to determine if SetIntegrityStreams has been set to true or false after it has been formatted? I can not seem to find it anywhere.
Thanks
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
If E: is the volume, you can run
PS C:> Get-Item E:* | Get-FileIntegrity
I think that will tell you about whether files/folders have integrity streams disabled, which I guess implies that the file system has it disabled, but I’m not sure of another way to be 100% sure as I’ve never looked into it. I just make sure I format them correctly at the start 😉
I know you cannot speak to backup solutions but Microsoft’s own DPM does not yet support Exchange 2016 but an MS employee said it will be supported in just a few weeks (finally). The wrinkle is it will only support NTFS, ReFS is still a bit in the future. What is the trade off by going NTFS? I cannot gather what I am really giving up if I go ahead and deploy with NTFS. I have 77 mailboxes and right now the exchange db is 100 GB.
I did want to do HA and deploy a three server DAG group as describe in your ebook I purchased.
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
ReFS is more resilient. It’s not something I know a ton about, but I’m sure the benefits are well explained on TechNet.
That said, if you’ve got an integration that doesn’t support ReFS, like a backup product, then naturally you would not use it. NTFS is still supported, ReFS is just recommended where possible.
Hi Paul
Thanks for another good and helpful article.
Do you happen to know if Exchange 2016 Databases and Logs are supported on Windows Dynamic Disks? Or recommended?
Thanks in advance!
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
Supported, but not best practice.
Hi Paul,
do you have informations if backup systems do have any issues with ReFS?
Microsoft DPM is supporting ReFS – but no specific word on Exchange and ReFS.
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
No, you should ask your backup vendor. ReFS itself is not new so hopefully the major vendors are supporting it by now.
Thanks Paul for the excellent article, based on your customer interactions will it make sense for customers upgrading from Exchange 2010/2013 to upgrade their file system to ReFS? Is it worth the pain?
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
You don’t “upgrade” a file system to ReFS, you choose ReFS when you’re creating the volume to begin with. So there’s no pain, it’s just a slightly different step when you’re building the server before you put Exchange on it.
Should the Refs volume still be 64k allocation unit size formatted?
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
64K is the only choice with ReFS.
not with Win2016 🙂 be careful default is 4KB so your instructions can be misleading
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
Indeed. And the storage/sizing calculator spits out scripts that specify the allocation unit size as well now. I’ve updated my example. Thanks for the heads up.