Problems with file copying function

[gbl08ma Profile Website]

The problem is definitely inside or after CreateFile, as making the add-in display debug messages had already revealed. I knew there were some problems with manipulating g3a files from inside an add-in (that's why the "add-in manager" function is so unstable, and after your explanation I may remove it entirely as my suspicions are confirmed), even simply renaming them (changing the file extension) frequently makes the calculator reboot.

I'm not sure why Bfile_CreateEntry calls syscall 0x0028, shouldn't the scan only occur when the main menu is open? Or does the OS always keep a list of the add-ins in RAM? I'm guessing Bfile_CreateEntry only does it because it is also used in 3-pin file transfer, and people may transfer g3a files.

At some point I will add restrictions to the file manager so that it shows a warning when manipulating g3a files, or disables their manipulation entirely.

However, the file copying problem happens with files other than g3a! It happens very often with some g3p (picture) files and I've seen it happen with other big files too. This is on the emulator running OS 1.02 (I have seen the error happen on real hardware too, on earlier builds). This is my biggest issue with the code now.
May it has something to do with the fact that I have two files open at the same time (with different handles, of course), and the OS is buggy and doesn't like this? I have already tried code that opens and closes files as it reads and writes, and the problem existed too (I recall it was even worse).

EDIT: or maybe it has something to do with the fact that I'm creating a file (the destination file) while the original is open?

EDIT 2: during some experiments, I stumbled upon a behavior I didn't know about... looks like the size of a file can be dynamically extended by calling WriteFile with a size bigger than the original file one. That is, one can use CreateEntry to create f.i. a 5 bytes file, then extend it to 100 bytes by doing a 100 bytes write. Can someone confirm this?

EDIT 3: looks like creating a file when another is open isn't the problem, as I changed my code to avoid this and yet managed to get a system error when trying to copy a g3p (not g3a) file. Is it possible that g3p files are an "exception" too?
I also stopped declaring variables as static on this new file copying code, and now the PC and target are:
TARGET=00309514
PC=881D0828

See the new code here: https://github.com/gbl08ma/utilities/bl ... r.cpp#L124 (want a compiled g3a?)

Renaming the file I wanted to copy from "Amsterdam.g3p" to "Amsterdam.aaa" did not avoid the error either, so I don't think it is a special case with g3p files. I think it may have more to do with flash sectors or the way files are put in the flash - this file I'm trying to copy, is one of those in the "CASIO" folder that come preinstalled from factory (on the emulator too). Maybe these have something special to them...

[SimonLothar]

I always get the error with g3a-files.
I never got the error with any other file-type yet. I copied a 1M-textfile as well as a standard-8k-g3p-file several times without any problems.

I'm at a loss with this, as long as I cannot trigger the error systematically.

[gbl08ma Profile Website]

Yeah, I don't get the error with files other than the factory-default g3p, either. (I have not tried copying g3a's).
On which OS version are you testing? Maybe it's a bug they fixed. Have you tried copying the files in the CASIO folder, or did you get rid of it already?
I think I'll record the emulator screen, I'm not lying...

EDIT: here's a video showing the problem (MKV container, V8 codec, VLC can open it but I can upload to YT if someone can't view):
http://tny.im/dl/casioprizm/devel/utilsCopyError.mkv

Note how it copies a random g3e file fine, then it copies a g3p file fine (most amusing, that file usually fails, so I copied another), and then it breaks with another g3p file (the target and PC of the error are no longer in the static RAM because on my new copying code, no variables are declared as static).
Also note how the storage memory goes into a strange state after the error (OS can still get the free size and create folders, but doesn't show any files).
I then rebooted the emulator, proceed to showing how the problematic file has a different content than the original, and most interesting (I only noticed this after I record, so you'll have to pay attention to see it while I scroll), you can see how the "AAA" folder I created is still there.
I am willing to call this some bug of Bfile functions... unless it has to do with the optimization state of the filesystem (the filesystem on this emulator hasn't been optimized in ages, on purpose, so I can test if this sort of things work with the FAT fragmented - they should).

[SimonLothar]

On which OS version are you testing?

2.00.0200

Have you tried copying the files in the CASIO folder, or did you get rid of it already?

No, I copied to folders named ABC and ABCDEFAB. I did not touch the folder CASIO.

I think I'll record the emulator screen, I'm not lying...

I do not think that you are lying. I followed your video and could observe the error with copying a g3p-file from the CASIO\g3p-folder. But in this case the error occurred after the call of Bfile_WriteFile_OS.

the target and PC of the error are no longer in the static RAM because on my new copying code, no variables are declared as static).

Usually target is the address immediate following the call of Bfile_CreateEntry_OS (somewhere in the add-in's virtual ROM range (starting at 0x00300000)). PC depends on the content of the stack, when the exception occurs. It is more or less unpredictable.

I am willing to call this some bug of Bfile functions... unless it has to do with the optimization state of the filesystem (the filesystem on this emulator hasn't been optimized in ages, on purpose, so I can test if this sort of things work with the FAT fragmented - they should).

I am not sure. Perhaps there is some flag, which must be set before any call of SMEM-changing-functions to prevent auto-optimization. PictPlot.g3a must have the same problem, because it creates and writes g3p-files too. Maybe PictPlot.g3a reveals some important information.

[gbl08ma Profile Website]

Geometry.g3a reads and writes g3p files, too. It's the only way to store a geometry file in SMEM I think, is to open a g3p file, do some geometry on top of it and save it again (the geometry items will be saved on the g3p).
I find it strange that the error does not happen with files that didn't come from factory (the files that come in the CASIO folder are probably very well aligned at the beginning of the filesystem in flash).
Also, remember my add-in can read and write files with at most 50 KB, in the calendar thing. It has never showed any problems with that since my last thread with Bfile problems (where the problem came to be the fact that I was reading from a buffer in flash). So if optimization happens, it would occur too when creating and editing calendar events...
Furthermore, if it were related to optimization, it would happen with other big files without g3p extension (I have successfully copied files with over 300 KB), I think. But everything is possible...

[helder7]

Today I did download the new version of the emulator (v2.00) from edu.casio.com.
It is independent from the previous version of the emulator, by other words, the files of the previous version are not imported.
The first differences I found were: a new option to lock/unlock the main memory, screenshots with more resolution/quality (400x228) and auto-nofitication of future updates. I think it has more default g3p images, but im not sure.

I installed your Utilities 1.1, and I started copying images g3p from the default CASIO folder to to folder "DEMO", one by one. Once I have copied some, I decided to copy a group with 12 images (if im sure) and had no problem. After, I went back to copy images one by one until I obtained an error with a g3p image:

System ERROR
REBOOT: [EXIT]
INITIALIZE: [EXE]
TLB ERROR
TARGET=003080D4
PC=881CDBFC

After restarting the add-in, also got an error with a g3b file, with same target and different PC.

But yeah, an error just with image files .g3p/.g3b is a bit strange. With add-ins files is understandable, but not with images. So I decided to do one more test, with eActivity files. Also got an error with *.g2e files, after copying 9 files successfully. I did a video: https://app.box.com/s/0tyscu1z1mnlmfsqwcsu

[gbl08ma Profile Website]

Today I did download the new version of the emulator (v2.00) from edu.casio.com.

I had no idea they had already released the 2.00 for the emulator, the last time I checked (when 2.00 was released for real hardware) there was version 1.04 only, and it only worked as an update to people who had the full version and not the 90 day trial (I even took the installer apart to retrieve the ROM files, but they would not work on the 1.02 emulator).

Now we know the problem is independent of OS version, and from your problem with copying a g2e file, I'm more inclined to believe it has less to do with file extension and more to do with the position of the file in flash. I know that some time ago I tried to copy a file using Utilities (it was not a g3p file) on my calculator, when my calculator (before it died and had the PCB replaced with a new one with a new hardware revision) still had OS 1.04, and got an error too.
It could be a problem from Utilities v1.1 that is not present on my new copying code... but seeing as the new code errors in exactly the same way as the old one, there should not be much difference.

The more one looks into it, the more difficult this error gets to debug... and I'd really like to take the copying function out of the "advanced tools", because I and other people often need it, and it better be stable.

I would be amazed if there's really some flag that Casio add-ins set before manipulating the SMEM, because if there is, it's surprising that Bfile has worked so well for so long on so many add-ins, and the thing would only be discovered now!

[gbl08ma Profile Website]

I added lots of debug messages to my code. For some reason, Amsterdam.g3p now never seems to fail.
Amusement_park.g3p, on the other hand, creates successfully, ReadFile and WriteFile work well in the loop for over 40 KB... at which point it magically stops working with a system error (the file has 53 KB).

If we analyse the bad copy of Amusement_part.g3p after a reboot, it indeed is written up to the 0xd000 offset (53248 in decimal), when it still has 0x660 bytes to write (that are all 0x00). Remember that in this code, the reads and writes happen in blocks of 0x1000 (4096) bytes.

I made further changes to my code, so that it tries to write only what ReadFile can read (ReadFile return value), instead of having my code calculate the amount that's yet to be copied based on what GetFileSize returned at the beginning.
With the new code, it copied right up to 53248, then system error: that is, the same thing happened.

I repeated the procedure with a slightly altered code that would sum to the current position counter not the fixed buffer size, but the return of ReadFile (that counter is used for nothing, except knowing when to get out of the loop). This made no change... except that this time it failed after having successfully copied 16384 bytes.

I'm officially out of ideas

EDIT: on files that copy successfully, it appears that some read/write cycles take longer, in comparison to the other cycles. Maybe it is doing some operation on these cycles, which causes the error. The only operation I could see it doing, is filesystem optimization... still, I don't know what to do.

[TeamFX Profile]

Today I did download the new version of the emulator (v2.00) from edu.casio.com

So I looked at those updated DLLs and the only interesting stuff seems to be in LCDR61524A.dll

This is what they added:

Code: Select all

0000D450   00 00 00 00 00 00 00 00  00 00 53 61 76 65 4C 65             SaveLe
0000D460   66 74 4F 66 66 73 65 74  00 4C 65 66 74 20 6F 66   ftOffset Left of
0000D470   66 73 65 74 20 69 6E 20  77 69 6E 64 6F 77 20 66   fset in window f
0000D480   6F 72 20 73 61 76 65 00  53 61 76 65 54 6F 70 4F   or save SaveTopO
0000D490   66 66 73 65 74 00 54 6F  70 20 6F 66 66 73 65 74   ffset Top offset
0000D4A0   20 69 6E 20 77 69 6E 64  6F 77 20 66 6F 72 20 73    in window for s
0000D4B0   61 76 65 00 53 61 76 65  57 69 64 74 68 00 57 69   ave SaveWidth Wi
0000D4C0   64 74 68 20 69 6E 20 77  69 6E 64 6F 77 20 66 6F   dth in window fo
0000D4D0   72 20 73 61 76 65 00 53  61 76 65 48 65 69 67 68   r save SaveHeigh
0000D4E0   74 00 48 65 69 67 68 74  20 69 6E 20 77 69 6E 64   t Height in wind
0000D4F0   6F 77 20 66 6F 72 20 73  61 76 65 00 00 00 00 00   ow for save

0000E0A0   53 63 72 6F 6C 6C 00 53  63 72 6F 6C 6C 20 73 65   Scroll Scroll se
0000E0B0   6E 73 6F 72 20 63 6F 6E  74 72 6F 6C 00 46 69 6E   nsor control Fin
0000E0C0   67 65 72 31 00 46 69 72  73 74 20 66 69 6E 67 65   ger1 First finge
0000E0D0   72 20 70 72 65 73 73 00  46 69 6E 67 65 72 32 00   r press Finger2
0000E0E0   53 65 63 6F 6E 64 20 66  69 6E 67 65 72 20 70 72   Second finger pr
0000E0F0   65 73 73 00 4C 65 66 74  50 6F 73 00 4C 65 66 74   ess LeftPos Left
0000E100   20 73 74 61 72 74 20 70  6F 73 69 74 69 6F 6E 20    start position
0000E110   69 6E 20 77 69 6E 64 6F  77 00 52 69 67 68 74 50   in window RightP
0000E120   6F 73 00 52 69 67 68 74  20 65 6E 64 20 70 6F 73   os Right end pos
0000E130   69 74 69 6F 6E 20 69 6E  20 77 69 6E 64 6F 77 00   ition in window
0000E140   54 6F 70 50 6F 73 00 54  6F 70 20 73 74 61 72 74   TopPos Top start
0000E150   20 70 6F 73 69 74 69 6F  6E 20 69 6E 20 77 69 6E    position in win
0000E160   64 6F 77 00 42 6F 74 74  6F 6D 50 6F 73 00 42 6F   dow BottomPos Bo
0000E170   74 74 6F 6D 20 65 6E 64  20 70 6F 73 69 74 69 6F   ttom end positio
0000E180   6E 20 69 6E 20 77 69 6E  64 6F 77 00 00 00 00 00   n in window


[gbl08ma Profile Website]

I got the idea somewhere that the hardware of the fx-CP400 is the same used in the Prizm, but with a touchscreen controller. It even has the same amount of RAM (which probably means the system is much more optimized than that of the Prizm, otherwise they would not be able to fit giant VRAM buffers for the huge screen).