A Windows File System Driver can speed up file I/O operations?

[Dash]

Hi,

I'm not sure this is the right group.
I've been looking at some applications, like antivirus, antimalware ecc, and
I've been surprised to see how fast they are in accessing and reading the file
system. I'm not sure I can get the same speed using the traditional
CreateFile()/CloseHandle() api functions.

So my questions are:
1) do they use a file system driver (mini driver?)?
1.1) If so, how could a file system driver speed up those operations compared
to the traditional ones?

2) Or do they use Win32 file APIs with IO completion port or something like that?

Thanks

-- Dash

[JJ]

Post by Dash
Hi,
I'm not sure this is the right group.
I've been looking at some applications, like antivirus, antimalware ecc, and
I've been surprised to see how fast they are in accessing and reading the file
system. I'm not sure I can get the same speed using the traditional
CreateFile()/CloseHandle() api functions.
1.1) If so, how could a file system driver speed up those operations compared
to the traditional ones?

That's because most of the AV real time scanning process are done by the
AV's device driver which are executed at kernel level, so it's not subject
to security checks.

Post by Dash
1) do they use a file system driver (mini driver?)?

File system driver, yes. e.g. NTFS, FAT, UDFS, etc. But not mini file system
driver. Mini file system driver requires a file system driver, and is
designed for an extension to an existing file system, but it can also be
used for other purpose.

Post by Dash
2) Or do they use Win32 file APIs with IO completion port or something like that?

Win32 file APIs uses file system drivers. Not the other way around. i.e.
roughly:

Win32 File API
-> Kernel
-> File related drivers
-> Volume related drivers
-> Storage related drivers
-> Port related drivers
-> Bus related drivers

[Dash]

Post by JJ

Post by Dash
Hi,
I'm not sure this is the right group.
I've been looking at some applications, like antivirus, antimalware ecc, and
I've been surprised to see how fast they are in accessing and reading the file
system. I'm not sure I can get the same speed using the traditional
CreateFile()/CloseHandle() api functions.
1.1) If so, how could a file system driver speed up those operations compared
to the traditional ones?

That's because most of the AV real time scanning process are done by the
AV's device driver which are executed at kernel level, so it's not subject
to security checks.

There it is.

Post by JJ

Post by Dash
1) do they use a file system driver (mini driver?)?

File system driver, yes. e.g. NTFS, FAT, UDFS, etc. But not mini file system
driver. Mini file system driver requires a file system driver, and is
designed for an extension to an existing file system, but it can also be
used for other purpose.>

Post by Dash
2) Or do they use Win32 file APIs with IO completion port or something like that?

Win32 file APIs uses file system drivers. Not the other way around. i.e.
roughly:My fault: the 'they' there is referred to who develops applications I

mentioned above, AVs, backups ecc.>

Post by JJ
Win32 File API
-> Kernel
-> File related drivers
-> Volume related drivers
-> Storage related drivers
-> Port related drivers
-> Bus related drivers

In a nutshell.
Using a device driver I can speed up file I/O operations because of less
restrictions and checks compared to a relative user-land call.
I assume there must be a sort of data-communication "protocol" between the
device driver and the user-land application.
And I guess a mini file system driver might be good as well in place of a real
device driver. And I suspect that Microsoft needs to sign it.

Thanks

[JJ]

Post by Dash
I assume there must be a sort of data-communication "protocol" between the
device driver and the user-land application.

In kernel level, device drivers communicates using I/O Request Packets
(IRPs) - which are function calls to IRP functions of device drivers.

User code may communicate with device drivers using I/O control via
DeviceIoControl(). But most API functions which need to talk with devices
drivers (e.g. CreateFile(), FindFirstFile(), etc.), the communication are
done at kernel level. i.e. the API function uses system service to call a
kernel function, which in turns, translate it into IRP(s).

Post by Dash
And I suspect that Microsoft needs to sign it.

Yes. That's a one major hurdle.

[JJ]

Post by JJ
My fault: the 'they' there is referred to who develops applications I
mentioned above, AVs, backups ecc.

I/O completion port is just a better way to queue non similar I/O
operations. It doesn't provide a better performance. It may infact be slower
due to queuing and multi-thread management overhead - especially when it's
used to access anything which can not process multiple I/O at the same time.
e.g. magnetic based disk, or DVD.

[Charlie Gibbs]

Post by Dash
Hi,
I'm not sure this is the right group.
I've been looking at some applications, like antivirus, antimalware ecc, and
I've been surprised to see how fast they are in accessing and reading the file
system. I'm not sure I can get the same speed using the traditional
CreateFile()/CloseHandle() api functions.
1) do they use a file system driver (mini driver?)?
1.1) If so, how could a file system driver speed up those operations compared
to the traditional ones?
2) Or do they use Win32 file APIs with IO completion port or something like that?

I've written plenty of programs that use the C run-time library, which is
just a wrapper around the Windows filesystem APIs. They are lightning-fast -
and I've never used I/O completion ports or other such complexity.

I suspect that the performance problems you refer to lie elsewhere,
especially in this era of bloatware.

[Kaz Kylheku]

Post by Charlie Gibbs

Post by Dash
Hi,
I'm not sure this is the right group.
I've been looking at some applications, like antivirus, antimalware ecc, and
I've been surprised to see how fast they are in accessing and reading the file
system. I'm not sure I can get the same speed using the traditional
CreateFile()/CloseHandle() api functions.
1) do they use a file system driver (mini driver?)?
1.1) If so, how could a file system driver speed up those operations compared
to the traditional ones?
2) Or do they use Win32 file APIs with IO completion port or something like that?

I've written plenty of programs that use the C run-time library, which is
just a wrapper around the Windows filesystem APIs. They are lightning-fast -
and I've never used I/O completion ports or other such complexity.

I/O completion ports are for multiplexing I/O onto a variably scalable
pool of threads.

They are useful in service situations where you would otherwise face
two less efficient alternatives:

- Using polling mechanisms, like a Unix style select for sockets (even
less efficient on Windows than Unix), or WaitForMultipleObjects
(limitation or 64 handles) and whatnot.

- Creating a thread for each request context, and doing regular
blocking I/O in that thread. (Then number of threads could be large.)

With I/O completion ports, you can queue I/O requests to multiple
handles, without blocking on any of them. The blocking is done
elsewhere: by waiting on the completion port. But the waiting ona
completion port is not *specific*: you can wait for *any* request to
finish (like you do with waitForMultipleObjects to wait for any objet
to signal, or with select to wait for any socket to become ready).

[Bonita Montero]

I/O completion ports are bady implemented in Windows. I wrote two
little test-programs which keep 500 asnychronous random 4kB I/O
-requests to my SSD running (non-buffered to a 1GB-file). Once
with a I/O completion port and another with synchronous I/O from
500 threads. With I/O completion-ports I get about 7% CPU-load
and 306MB/s. With overlapped I/O and only one request per thread
I get 21,3% CPU-load and 382MB/s. But with I/O completion ports
the whole system becomes extremely sluggish and is hard to use.