Discussion:
MapViewOfFile throws an "access denied" error - but only on large files
Add Reply
R.Wieser
2018-11-20 13:22:05 UTC
Reply
Permalink
Hello All,

I've got some basic code opening a file map (removed all error catching):

call CreateFileA,[@@sFile],GENERIC_READ,FILE_SHARE_READ,0,OPEN_EXISTING,0,0
mov [ebx].FM_hFile,eax
call CreateFileMappingA,[ebx].FM_hFile,0,PAGE_READONLY,0,0,0
mov [ebx].FM_hMap,eax
call MapViewOfFile,[ebx].FM_hMap,FILE_MAP_READ,0,0,0

The problem is that this works for smallish files, but when I try to open a
3 GB file the MapViewOfFile returns with an error, and GetLastError shows me
its "access denied". Which does not make any sense to me.

I've just tried to google the problem, but most of those are about coding
mistakes. And as my code seems to work perfectly for smallish files ....

The only thing I can think of is that my (old) machine doesn't have a whole
lot of memory and a 3 GBfile might simply not fit. Than again, I though
that mapping a file doesn't actally try to load everything into memory. And
besides, I would than have expected an "out of memory", not an "access
denied" error.

Does anyone know whats going on here ?

Regards,
Rudy Wieser
JJ
2018-11-20 16:08:35 UTC
Reply
Permalink
Post by R.Wieser
Hello All,
mov [ebx].FM_hFile,eax
call CreateFileMappingA,[ebx].FM_hFile,0,PAGE_READONLY,0,0,0
mov [ebx].FM_hMap,eax
call MapViewOfFile,[ebx].FM_hMap,FILE_MAP_READ,0,0,0
The problem is that this works for smallish files, but when I try to open a
3 GB file the MapViewOfFile returns with an error, and GetLastError shows me
its "access denied". Which does not make any sense to me.
I've just tried to google the problem, but most of those are about coding
mistakes. And as my code seems to work perfectly for smallish files ....
The only thing I can think of is that my (old) machine doesn't have a whole
lot of memory and a 3 GBfile might simply not fit. Than again, I though
that mapping a file doesn't actally try to load everything into memory. And
besides, I would than have expected an "out of memory", not an "access
denied" error.
Does anyone know whats going on here ?
If that system is 32-bit, it could be due to the default restriction of 2GB
user address space limit (not memory storage space). The rest of 2GB are
reserved for system/hardware. The restriction can be changed to 3GB via boot
loader setting.

But the error code is indeed doesn't make sense. However, that may be the
closest representation possible for the cause of error. Although there's
more appropriate error code, it falls into e.g. COM category - which IMO,
out of kernel's scope.
R.Wieser
2018-11-20 16:33:16 UTC
Reply
Permalink
JJ,
Post by JJ
If that system is 32-bit
It is. XPsp3, 32 bit.
Post by JJ
it could be due to the default restriction of 2GB user
address space limit (not memory storage space)
Never thought of that. I assumed that the addressing into the mapped file
would go similar to reading chunks of memory yourself. Caching the
read/written sectors I mean.
Post by JJ
But the error code is indeed doesn't make sense. However,
that may be the closest representation possible for the cause
of error.
As you have described it, its a case of being "out of virtal address space".
Something which, to me, lies a bit closer to "out of memory" than "access
denied". Than again, I do not know what the people who decided to use that
errorcode where thinking about at the time.

Thanks for the clarification.

Regards,
Rudy Wieser
JJ
2018-11-21 10:08:29 UTC
Reply
Permalink
Post by R.Wieser
Never thought of that. I assumed that the addressing into the mapped file
would go similar to reading chunks of memory yourself. Caching the
read/written sectors I mean.
Well, yes. The file data is read as chunks, but that applies only to the
memory storage. The whole address space for the entire file is preallocated.

The same error may also occur even though there are enough free address
space, but not large enough to fit an unfragmented one. To my undestanding,
the system can't defragment the allocated address space, because the system
already gave memory pointers to processes.

I think your only choice is to partially map the file, and remap it when
other part of the file is required. Similar like Expanded Memory paging in
DOS platform, if you're familiar with it.
R.Wieser
2018-11-21 13:23:19 UTC
Reply
Permalink
JJ,
Post by JJ
The whole address space for the entire file is preallocated.
Yes. But why would a virtual preallocation of 5, 9 of 16 GB be a problem ?
Current computers, even 32 bit ones, can address way more in physical memory
than that.
Post by JJ
To my undestanding, the system can't defragment the allocated
address space, because the system already gave memory pointers
to processes.
I guess the Local/GlobalAlloc method idea (with its non-fixed memory
adresses) only came later.
Post by JJ
I think your only choice is to partially map the file, and remap
it when other part of the file is required.
Yup. It just means I have to re-think everything (having a nice heap of
contiguous memory is *very* easy to manage & search)
Post by JJ
Similar like Expanded Memory paging in DOS platform, if
you're familiar with it.
Not really familiar, but know how it works. "copying " upper memory into,
and outof base memory. And ofcourse similar to working with sectors of/in
a file

Regards,
Rudy Wieser
JJ
2018-11-22 04:31:36 UTC
Reply
Permalink
Post by R.Wieser
Yes. But why would a virtual preallocation of 5, 9 of 16 GB be a problem ?
Current computers, even 32 bit ones, can address way more in physical memory
than that.
The problem is due to initial 32-bit Windows design where user processes are
given only one segment selector (i.e. one segment descriptor) for
everything, so it's limited to 4GB. Other segment selectors are provided
only to point to Windows internal data. x86/x64 CPUs have a capacity of 8192
segment descriptors (which should provide a total of 32TB address space),
but Microsoft decided that we should have only one. X(

If this is 16-bit real mode DOS, it's like each program is only allowed to
use a single memory segment value - where each is only allowed to access
64KB of memory.

The PAE and AWE was later added to circumvent the problem. PAE is only
possible if the CPU supports it, and such CPU doesn't yet exist when Win32
was made. AWE is just like EMS paging. Yet, none of them allows processes to
use more than one segment selector.
Post by R.Wieser
I guess the Local/GlobalAlloc method idea (with its non-fixed memory
adresses) only came later.
It's actually very early. The 16-bit Windows 2.0 SDK already has the
GMEM_MOVABLE constant.
R.Wieser
2018-11-22 06:57:18 UTC
Reply
Permalink
JJ,
Post by JJ
The problem is due to initial 32-bit Windows design where user
processes are given only one segment selector (i.e. one segment
descriptor) for everything, so it's limited to 4GB.
Well, just assigning a single selector explains the limit (somewhat. I
think). Odd that it has never changed with the progression of OSes though.
Post by JJ
Yet, none of them allows processes to use more
than one segment selector.
I would have imagined MS would have developed a transparent method alike we
thought of, just grabbing the part of the mapped file we're interested in
(and make it look as if all is available). Pretty-much the same as how we
can seek in files and read the memory block we want.
Post by JJ
It's actually very early. The 16-bit Windows 2.0 SDK already has
the GMEM_MOVABLE constant.
I knew it was old (have programmed under W95), but not that it was *that*
old.

Regards,
Rudy Wieser

Loading...