Mapping a file to memory

[gbl08ma Profile Website]

I would like to know if any of the knowledgeable minds in this forum (Simon, TeamFX, ...) know of any way to map a file in storage memory to a contiguous location in the virtual memory of the Prizm, from inside an add-in, and preferably with as little side effects as possible.

I ported a scripting engine to the Prizm and what I'm trying to do is to run a script without having to copy it to RAM first. (I'm only copying to RAM because the file contents are not contiguous in the storage, otherwise I'd just find their address and point the interpreter to that...). The interpreter also expects the whole file to be available at the same time, so it can't follow the pointers in the file allocation table.
If it could be run from a mapped file these problems would be solved, and more RAM would be available for the script to use. And the scripts could actually be larger than my RAM buffer, eventually even larger than the stack size.

Note that the MMU has support for this (add-ins are mapped into memory and then run from the mapped region), I just wonder if there's any syscall that would allow a second file to be mapped at another address, without destroying the execution of the current add-in. My guess is no, but I guess asking doesn't hurt... just in case anyone knows about anything.

[AmazoNKA Profile]

Picking up on the last point, is it actually possible to force add-in to restart on Prizm please, in other words actually destroy current execution of the add-in without causing system errors but to be able to start the same last executed add in from the beginning, I probably could try long jump etc but legacy system calls seemed to indicate it was possible, or perhaps someone could guide me what flags could be utilised for the same or recommend other workarounds. Thanks in advance

[SimonLothar]

Picking up on the last point, is it actually possible to force add-in to restart on Prizm please, in other words actually destroy current execution of the add-in without causing system errors but to be able to start the same last executed add in from the beginning, I probably could try long jump etc but legacy system calls seemed to indicate it was possible, or perhaps someone could guide me what flags could be utilised for the same or recommend other workarounds. Thanks in advance

Perhaps you could use APP_EnableRestartFlag (see below). I excluded OS version 2.02 because I did not verify it, yet. Perhaps the exclusion can be omitted. I just checked the function with version 2.00.

Code: Select all

//
short*APP_EnableRestart(){
unsigned int ea;
unsigned int j;
short*pEnableRestartFlag;
  if ( OSVersionAsInt() < 0x02020000 ){
    ea = *(unsigned int*)0x8002007C;
    ea += 0x1e6e*4;
   
    ea = *(unsigned int*)( ea );
    j = *(unsigned char*)( ea + 1 );
    j *= 4;
    j = ( ea + j + 4 ) & 0xFFFFFFFC;
    pEnableRestartFlag = (short*)(*( unsigned int*)( j ) + 8 );
  }else{
    pEnableRestartFlag = 0;
  }
  if ( pEnableRestartFlag ) *pEnableRestartFlag = 1;
  return pEnableRestartFlag;
}


[SimonLothar]

I ported a scripting engine to the Prizm and what I'm trying to do is to run a script without having to copy it to RAM first...

Couldn't you read the script-file line by line using Bfile-functions?

[gbl08ma Profile Website]

Couldn't you read the script-file line by line using Bfile-functions?

I looked into this before and got the idea the parser needed the whole file to be available at once. But now I went to look at it again and it seems to have a mode where it is interactive, i.e. to be used when the script is typed from a terminal. Perhaps I can use this mode to "feed" the parser one line at a time as if it was being typed by the user, and achieve the same effect as reading it all at once.

https://code.google.com/archive/p/picoc ... Picoc.wiki doesn't mention any difference in behavior between interactive mode and whole-file mode, so maybe this is indeed possible.

The problem is that because the Bfile functions begin to show strange behavior once more than one file is open, then scripts wouldn't be able to use Bfile effectively (because the script would need to be open in order to speed up its access; I can't be opening and closing it every few bytes...). It will still be much slower if it has to wait for ReadFile every time it needs to parse some bytes... so maybe I'll leave it with the RAM buffer and limit script size to 64 KB. Kind of limits the size of the software one can build with the scripting engine, but oh well...

So mmap on the Prizm would still be much better, but one has to live with these limitations.

[AmazoNKA Profile]

Many thanks for APP_EnableRestartFlag code

[SimonLothar]

So mmap on the Prizm would still be much better...

I did not find a syscall, which virtualizes any file to any address space.
I found the following:
At best syscall 0x0026 could serve as template. 0x0026 is used to virtualize a G3A. Important syscalls in this course are 0x1DAA, which enumerates the 4096-byte blocks of an open file, and 0x0009, which sets the TLB.
Possibly syscall 0x0027 is informative, too. It virtualizes the static RAM (8*64k) of a G3A and uses 0x000A to set the TLB.

I did not unravel the way of 0x0026 in detail, therefore I cannot present examples. These things are rather complicated.

[gbl08ma Profile Website]

Thank you very much for taking the time to research this.

All these syscalls look very interesting, but as you say none of them appear to be generic, i.e. built to work with any kind of file and address. We also don't know whether they would still be of any use for setting things up from inside an add-in, without destroying the execution context for the add-in.

Then there's the additional problem of having to reconfigure the TLB every time a Bfile operation occurs, because as we know very well, filesystem optimization moves the file sectors around in the flash. The OS is smart enough to reconfigure the TLB for the currently running add-in (but not smart enough to pause timers set up by the add-in: see the lengthy discussion about this we had on this forum months/years ago...). I highly doubt it would have the logic to reconfigure the TLB for our custom allocations too, or to destroy these settings when the add-in is exited.

Since the PicoC environment I've ported has access to Bfile syscalls (and IMO would be quite limited without it), one could always intercept each system call from PicoC (very easy to do) and see if TLB reconfiguration is necessary before proceeding with the script execution. So this last point about the TLB needing reconfiguration would not be the worst problem. (Enumerating all of the syscalls that use Bfile functions internally would still be a lot of work, but oh well).

And then there's the problem of not being able to emulate the OS shutdown/standby and wake-up on the emulator, which would mean testing that part on hardware, and last time such a thing happened I ended with a brick (these companies which do not write-protect flash sectors when given the opportunity... ).

Of all the syscalls you listed, the one that sounds more promising for generic use is 0x1DAA, even though it does little to help me with this problem - unless I could, through a clever data structure (sort of a map) and modification of the PicoC parser, make it execute the script from the 4K-sectors directly, jumping to the right sectors as needed. The point about file system optimizations still stands, except now the problem has shifted to a slightly more amenable matter of updating that data structure (which, unlike the TLB, I would have full control over).

I'll take a look at these syscalls with our favorite SH-4A disassembler when I have time, but I highly doubt I'll find anything of interest, as my database is certainly well less complete than yours and I have much less experience with SH4 assembler...

[AmazoNKA Profile]

Hi gbl08ma, I'm wondering if any of the above plus the other post by Simon regarding detecting registered add-in tables and the add-in's restart enablement are of any help with finding even safer ways of disabling add-ins functiontionality in your utilities add-in?

[SimonLothar]

Of all the syscalls you listed, the one that sounds more promising for generic use is 0x1DAA...

The interface is int Bfile_GetBlockAddress( int handle, int offset, int*address );.
handle: file handle, opened in readonly mode (1)
offset: block offset (0, 0x1000, 0x2000, 0x3000...)
*address (on return): block pointer

Hint: If you open a G3A in readwrite mode (3) and use 0x1DAA beyond the last block (in this case 0x1DAA returns -2), the G3A vanishes from the main menu!

You are right: The automatic optimization seems to be a serious problem.