Experiment: Writing a Memory Mapped File to Disk

For this experiment I used SysInternals’ Process Monitor to inspect how opening a file impacts writing a new memory mapped file to disk. I used C#’s FileStream and Win32’s CreateFile method with different options. After creating the file I mapped it to a view and wrote 4GB of ASCII character 137 to the file. I let the program pause for a few seconds and then disposed the file view.

I was hoping to see Windows immediately flush the file I opened with the WriteThrough and NoBuffering flags. However, in all cases I found that Windows used Fast I/O to write the file to disk in two 2GB chunks when I disposed the view. No earlier writes were observed. I was curious if Fast I/O was triggered because every single page was marked dirty or if I crossed a threshold of dirty pages. As a followup I tried writing 8kB of data at intervals of a 512kB offset in the file. With a patchwork of writes, Windows no longer used Fast I/O but called WriteFile individually for lengths varying from 32kB to 512kB (with a bias for the high end). The writes were mostly sequential with some periodically out of order.

To find the behavior I was originally seeking (immediate writes), I tried creating a file with the WriteThrough and NoBuffer flags and the writing to the FileStream directly. In this case, writing a 4kB buffer triggered a WriteFile call with the WriteThrough flag. Seeking forward in the file caused a synchronous zeroing of intermediate bytes.

Comments

Windows DevCenter describes Fast I/O as a parallel mechanism to IRPs which bypasses the file system and storage driver stack by copying data directly from the user buffer to the system cache. According to Windows Internals, the FlushFileBuffers method can be used to trigger an immediate disk write when opening a file with the WriteThrough flag.

The drive used for this experiment was an Intel SSD 240GB with the Microsoft driver (v 6.1.7600.16385). Write caching was enabled.

Results


C# FileStream

var fs = File.Open(g, FileMode.Create, FileAccess.ReadWrite, FileShare.ReadWrite);

Events:
CreateFile Desired Access: Generic Read/Write, Disposition: OverwriteIf, Options: Synchronous IO Non-Alert, Non-Directory File, Open No Recall, Attributes: n/a, ShareMode: Read, Write, AllocationSize: 0, OpenResult: Created=
SetEndOfFileInformationFile EndOfFile: 4,294,967,296
SetAllocationInformationFile AllocationSize: 4,294,967,296
...
FASTIO_RELEASE_FOR_SECTION_SYNCHRONIZATION
CreateFileMapping SyncType: SyncTypeOther
FASTIO_RELEASE_FOR_SECTION_SYNCHRONIZATION
ReadFile Offset: 0, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
ReadFile Offset: 32,768, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
... 32kB reads ...
ReadFile Offset: 4,294,934,528, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
FASTIO_ACQUIRE_FOR_CC_FLUSH
WriteFile Offset: 0, Length: 2,147,483,648, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
WriteFile Offset: 2,147,483,648, Length: 2,147,483,648, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
FASTIO_RELEASE_FOR_CC_FLUSH
FASTIO_ACQUIRE_FOR_CC_FLUSH
FASTIO_RELEASE_FOR_CC_FLUSH
CloseFile

Write_Through | NoBuffering

CreateFile(g,
GenericRead | GenericWrite,
Read,
IntPtr.Zero,
OpenAlways,
Write_Through | NoBuffering,
IntPtr.Zero);

Events:
CreateFile Desired Access: Generic Read/Write, Disposition: OpenIf, Options: Write Through, No Buffering, Synchronous IO Non-Alert, Non-Directory File, Attributes: n/a, ShareMode: Read, AllocationSize: 0, OpenResult: Created
ReadFile Offset: 0, Length: 0, I/O Flags: Non-cached, Priority: Normal
SetEndOfFileInformationFile EndOfFile: 4,294,967,296
SetAllocationInformationFile AllocationSize: 4,294,967,296
...
CreateFileMapping FILE LOCKED WITH WRITERS SyncType: SyncTypeCreateSection, PageProtection:
FASTIO_RELEASE_FOR_SECTION_SYNCHRONIZATION
CreateFileMapping SyncType: SyncTypeOther
FASTIO_RELEASE_FOR_SECTION_SYNCHRONIZATION
ReadFile Offset: 0, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
ReadFile Offset: 32,768, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
... 32kB reads ...
ReadFile Offset: 4,294,934,528, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
FASTIO_ACQUIRE_FOR_CC_FLUSH
WriteFile Offset: 0, Length: 2,147,483,648, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
WriteFile Offset: 2,147,483,648, Length: 2,147,483,648, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
FASTIO_RELEASE_FOR_CC_FLUSH
FASTIO_ACQUIRE_FOR_CC_FLUSH
FASTIO_RELEASE_FOR_CC_FLUSH
CloseFile

Normal

CreateFile(g,
GenericRead | GenericWrite,
Read,
IntPtr.Zero,
OpenAlways,
Normal,
IntPtr.Zero);

Events:
CreateFile Desired Access: Generic Read/Write, Disposition: OpenIf, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, AllocationSize: 0, OpenResult: Created
ReadFile Offset: 0, Length: 0, Priority: Normal
SetEndOfFileInformationFile EndOfFile: 4,294,967,296
SetAllocationInformationFile AllocationSize: 4,294,967,296
...
CreateFileMapping FILE LOCKED WITH WRITERS SyncType: SyncTypeCreateSection, PageProtection:
FASTIO_RELEASE_FOR_SECTION_SYNCHRONIZATION
CreateFileMapping SyncType: SyncTypeOther
FASTIO_RELEASE_FOR_SECTION_SYNCHRONIZATION
ReadFile Offset: 0, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
ReadFile Offset: 32,768, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
... 32kB reads ...
ReadFile Offset: 4,294,934,528, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
FASTIO_ACQUIRE_FOR_CC_FLUSH
WriteFile Offset: 0, Length: 2,147,483,648, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
WriteFile Offset: 2,147,483,648, Length: 2,147,483,648, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
FASTIO_RELEASE_FOR_CC_FLUSH
FASTIO_ACQUIRE_FOR_CC_FLUSH
FASTIO_RELEASE_FOR_CC_FLUSH
CloseFile


Normal: Writing 8kB out of every 512kB

CreateFile Desired Access: Generic Read/Write, Disposition: OpenIf, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, AllocationSize: 0, OpenResult: Created
ReadFile Offset: 0, Length: 0, Priority: Normal
SetEndOfFileInformationFile EndOfFile: 4,294,967,296
SetAllocationInformationFile AllocationSize: 4,294,967,296
...
CreateFileMapping FILE LOCKED WITH WRITERS SyncType: SyncTypeCreateSection, PageProtection:
FASTIO_RELEASE_FOR_SECTION_SYNCHRONIZATION
CreateFileMapping SyncType: SyncTypeOther
FASTIO_RELEASE_FOR_SECTION_SYNCHRONIZATION
ReadFile Offset: 0, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
ReadFile Offset: 2,097,152, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
... 32kB reads ...
ReadFile Offset: 2,145,386,496, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
WriteFile Offset: 0, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
WriteFile Offset: 2,097,152, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
WriteFile Offset: 32,768, Length: 524,288, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
.... mostly 512kB writes in no particular order, but various other sizes too ...
WriteFile Offset: 2,144,894,976, Length: 491,520, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
CloseFile