CreatePipe/CreateNamedPipe unbuffered?

[e***@gmail.com]

Hi!

I'm writing a quick little Popen class in C++ on windows. Basically I
want it to:

- Spawn a subprocess
- Upon termination ("Kill"), kill said subprocess, and all children
- Gather stdout/stderr and redirect it to some function I pass in.

I've got this all working, using a mixture of CreateProcess/*JobObject
functions, and the STARTUP_INFO parameters for handle redirection.

My only problem now, seems to be that the pipes seem to get buffered
at around 1024 bytes, at least on my machine, and it tends to give
garbled output if I ever *do* combine the output later on.

I'm curious, is it possible to make the pipe unbuffered (assuming I'm
diagnosing this correctly)?

Looking at the pipe API, it looks like it's not truly possible, since
pipes get a 'suggested' buffer size that the system then does stuff
with anyhow. However, I'm wondering if some underlying implementation
of pipes would solve this (i.e., if I can get a file descriptor from a
pipe and run setvbuf on it, or some such)?

Anyhow, your opinions and knowledge are appreciated. Thanks in
advance!

-e-

[Sebastian G.]

Post by e***@gmail.com
My only problem now, seems to be that the pipes seem to get buffered
at around 1024 bytes, at least on my machine, and it tends to give
garbled output if I ever *do* combine the output later on.
I'm curious, is it possible to make the pipe unbuffered (assuming I'm
diagnosing this correctly)?

Eh, yeah? A pipe is a buffer, by its very own nature.

What you really need is synchronization. Making something unbuffered is
about the worst idea of achieving this. What's wrong with a mutex?

[e***@gmail.com]

Post by Sebastian G.

Post by e***@gmail.com
My only problem now, seems to be that the pipes seem to get buffered
at around 1024 bytes, at least on my machine, and it tends to give
garbled output if I ever *do* combine the output later on.
I'm curious, is it possible to make the pipe unbuffered (assuming I'm
diagnosing this correctly)?

Eh, yeah? A pipe is a buffer, by its very own nature.
What you really need is synchronization. Making something unbuffered is
about the worst idea of achieving this. What's wrong with a mutex?

Hrmm, perhaps I worded this incorrectly. I recognize that the pipe is
a buffer, and that's all fine. I suppose a better way of stating the
problem is that the data that seems to be getting into the pipe is
buffered at 1024 characters each time.

I am using synchronization, using a CriticalSection to maintain the
same ordering - the only issue is that because the data getting into
the pipe is buffered at 1024 bytes each time, it looks garbled on
output.

I have a hunch now that perhaps my subprocess needs to be calling
fflush after printing -- that is, perhaps the buffering is happening
on the subprocess-to-OS side (if that's truly the boundary that's
present? Forgive my ignorance if not). I'll give fflush a try, but
ideally I'd like it if I don't have to demand subprocesses be
stringent on their use of fflush, as I can't control every one's
source code...

Perhaps I'm barking up the wrong tree, but at least that's an avenue
to explore.

Feel free to correct any incorrect notions I may have. ;)

[Sebastian G.]

Post by e***@gmail.com
the only issue is that because the data getting into
the pipe is buffered at 1024 bytes each time, it looks garbled on
output.

This, on its own, bears any logical conclusion. It obviously works as a FIFO
buffer, and maintains ordering.

As I already stated: Maybe your problem simply is that multiple threads are
writing into the pipe without synchronization, or more than one thread reads
the pipe without synchronization.

Post by e***@gmail.com
I'll give fflush a try, but
ideally I'd like it if I don't have to demand subprocesses be
stringent on their use of fflush, as I can't control every one's
source code

If they don't read a pipe in a correct manner, it's their fault, not yours.
You can reasonably expect them to know how to read a pipe if they're doing so.

[boris]

Post by e***@gmail.com
Hi!
I'm writing a quick little Popen class in C++ on windows. Basically I
- Spawn a subprocess
- Upon termination ("Kill"), kill said subprocess, and all children
- Gather stdout/stderr and redirect it to some function I pass in.
I've got this all working, using a mixture of CreateProcess/*JobObject
functions, and the STARTUP_INFO parameters for handle redirection.
My only problem now, seems to be that the pipes seem to get buffered
at around 1024 bytes, at least on my machine, and it tends to give
garbled output if I ever *do* combine the output later on.
I'm curious, is it possible to make the pipe unbuffered (assuming I'm
diagnosing this correctly)?
Looking at the pipe API, it looks like it's not truly possible, since
pipes get a 'suggested' buffer size that the system then does stuff
with anyhow. However, I'm wondering if some underlying implementation
of pipes would solve this (i.e., if I can get a file descriptor from a
pipe and run setvbuf on it, or some such)?
Anyhow, your opinions and knowledge are appreciated. Thanks in
advance!
-e-

Hi,

I wrote code (using anonymous pipes), where significantly more data that 1K
bytes was sent. On sending end I didn't do anything unusual - actually, I
was spawning process (some third-party app) with redirected output - similar
to what you're doing with popen() emulation.
On receiving end I was using Win32 function PeekNamedPipe() - it returns
count of bytes currently in pipe - followed by ReadFile(). In ReadFile() I
was specifying max # of bytes to be read same as count returned by
PeekNamedPipe(). And of course I was doing it in a loop, because
PeekNamedPipe() is non-blocking function.

Boris