Register

Problems with file copying function

Discuss issues related to the Casio Prizm 3rd party development
Senior Member
Posts: 141
Joined: Wed May 16, 2012 2:50 pm
Location: Portugal
Calculators: Casio fx-CG20

Problems with file copying function

Postby gbl08ma » Wed Nov 13, 2013 10:34 pm

I have developed the following code to copy files:

Code: Select all
//copy file
        char newfilename[MAX_FILENAME_SIZE] = "";
        strcpy(newfilename, browserbasepath);
        strcat(newfilename, name);
        if(!strcmp(newfilename, clipboard[curfile].filename)) {
          //a file with that name already exists on the new location.
          curfile++; continue; //skip
        }
        unsigned short newfilenameshort[0x10A];
        unsigned short oldfilenameshort[0x10A];
        Bfile_StrToName_ncpy(oldfilenameshort, (unsigned char*)clipboard[curfile].filename, 0x10A);
        Bfile_StrToName_ncpy(newfilenameshort, (unsigned char*)newfilename, 0x10A);
       
        int hOldFile = Bfile_OpenFile_OS(oldfilenameshort, READ, 0); // Get handle for the old file
        if(hOldFile < 0) {
          //returned error: couldn't open file to copy.
          curfile++; continue; //skip this file
        } else {
          //file to copy exists and is open. get its size.
          int copySize = Bfile_GetFileSize_OS(hOldFile);
          int hNewFile = Bfile_OpenFile_OS(newfilenameshort, WRITE, 0); // Get handle for the destination file. This should fail because the file shouldn't exist.
          if(hNewFile < 0) {
            // Returned error, dest file does not exist (which is good)
            int BCEres = Bfile_CreateEntry_OS(newfilenameshort, CREATEMODE_FILE, &copySize);
            if(BCEres >= 0) // Did it create?
            {
              //created. open newly-created destination file
              hNewFile = Bfile_OpenFile_OS(newfilenameshort, READWRITE, 0);
              if(hNewFile < 0) // Still failing?
              {
                //skip this copy.
                Bfile_CloseFile_OS(hOldFile);
                curfile++; continue;
              }
              //File to copy is open, destination file is created and open.
              //copy 4 KB at a time. Write more bytes in the last loop because WriteFile doesn't like writing few bytes.
              unsigned char copybuffer[4096*2+5];
              int curpos = 0;
              while(curpos < copySize) {
                memset( &copybuffer, 0, sizeof( copybuffer ) );
                int writesize = 0;
                if(copySize - curpos > 4096) {
                  // decide on whether to copy 4 KB, or find out if less than 8 KB are yet to be copied
                  // if less than 8 KB are yet to be written, write them all at once (instead of writing 4 KB then having a write that may be too small, causing problems with WriteFile)
                  if(copySize - curpos < 4096*2) {
                    // less than 8 KB are yet to be copied.
                    // write them all at once.
                    writesize = copySize - curpos;
                  } else {
                    // more than 4 KB are yet to be copied.
                    // write yet another 4 KB...
                    writesize = 4096;
                  }
                } else {
                  writesize = copySize - curpos;
                }
                Bfile_ReadFile_OS( hOldFile, copybuffer, writesize, -1 );
                Bfile_WriteFile_OS(hNewFile, copybuffer, writesize);
                curpos = curpos + writesize;
              }
              //done copying, close files.
              Bfile_CloseFile_OS(hOldFile);
              Bfile_CloseFile_OS(hNewFile);
            }
          } else {
            //destination file exists and is now open. Close open files and skip.
            Bfile_CloseFile_OS(hNewFile);
            Bfile_CloseFile_OS(hOldFile);
          }
        }


You can see the function where this code is used, here: http://tny.im/F1

This code works well, most of the time creating perfect copies of files (I'm checking with a sha256 sum of the files). However, this function doesn't always run right. For some files (usually it's always the same files, but it appears to depend on their position in the flash, as optimizing it moves the problem to other files) it crashes with a system error. One can access the main menu, but 90% of the time the storage memory will have been unmounted (no add-ins appear on menu and everything in the OS that involves the smem just results in memory errors), and only a reboot fixes it.

With all the lessons learned on my previous thread related to Bfile problems, I think I managed to avoid all the known pitfalls:
- No timers are installed while the file copy code is running;
- Small writes to the flash are avoided (when reaching the end of the file, a bit more than the standard write size is written at once, to avoid a smaller write request in the end);

May this have something to do with writing an odd number of bytes? Simon once said the problems with that have been fixed on the Prizm, and I'm yet to notice any problem related to that in my other code that uses Bfile (most notably, the calendar events code in Utilities, which in the latest version has not caused any system error so far) and where once in a while, an odd number of bytes is written (as it writes user-generated content which is not word-aligned).

Having the file system optimized or not appears to make no difference (all it does is make certain files less problematic, and others more).

Does anyone have any idea on to fix this? You can see the problem happen by downloading version 1.1 of Utilities, enabling the advanced tools in the settings, then going into the file manager and copying files until you encounter an error.

Senior Member
User avatar
Posts: 605
Joined: Sat Sep 15, 2012 6:59 am
Location: Krautland ****
Calculators: Casio fx-7400GII, Casio fx-7400GII (SH4), Casio fx-9750GII, Casio fx-9750GII (SH4), Casio fx-9860G, Casio fx-9860G SD, Casio fx-9860G Slim, Casio fx-9860GII SD, Casio fx-9860GII SD Power Graphic 2, Casio Classpad 330 plus, Casio fx-CG20, Casio fx-CG50, Casio Classpad fx-CP400

Re: Problems with file copying function

Postby SimonLothar » Thu Nov 14, 2013 10:44 am

Possibly a MsgBoxPush() / MsgBoxPop() - mismatch.

Code: Select all
void filePasteClipboardItems(File* clipboard, char* browserbasepath, int itemsInClipboard) {
  if (itemsInClipboard>0) {
.
.
.
   closeProgressMessage(); //we opened it, no matter if copying or moving.
  }
}

If closeProgressMessage() is a counterpart of some MsgBoxPush(), t. i. closeProgressMessage() contains some MsgBoxPop(), there could be a problem. F. i. if MsgBoxPush() has been called unconditionally before filePasteClipboardItems() is called and (itemsInClipboard>0) is not true, then closeProgressMessage() would not be called, giving a MsgBoxPush() / MsgBoxPop() - mismatch. MsgBoxPush() / MsgBoxPop() - mismatches lead to bad system failures.

BTW
I'd rather not suspect the writing of small amounts of data to be a cause of problems. The Bfile-functions are quite stable.
I'll be back!

Senior Member
Posts: 141
Joined: Wed May 16, 2012 2:50 pm
Location: Portugal
Calculators: Casio fx-CG20

Re: Problems with file copying function

Postby gbl08ma » Thu Nov 14, 2013 1:43 pm

Right now closeProgressMessage is just an alias for MsgBoxPop, but in the future it may be extended to do more things.

While what you said represents a problem, that situation never runs because filePasteClipboard items is not called without items in the clipboard. Still, I'm going to fix it and see if it solves the problem.

As for the problem with Bfile, meanwhile I remembered that the problem was not with small writes, but with writing from a buffer in flash (e.g. static const char array). Again, it's not the case here.

Senior Member
Posts: 141
Joined: Wed May 16, 2012 2:50 pm
Location: Portugal
Calculators: Casio fx-CG20

Re: Problems with file copying function

Postby gbl08ma » Thu Nov 14, 2013 9:33 pm

I fixed the problem with the possible MsgBox leak, but the problem still occurs.

One of the files I can't manage to copy without resulting in an error, is the Amsterdam.g3p file included in the example pictures folder. However, because I suspect it has something to do with the position of the file in the flash, I don't know if you're going to be able to reproduce the problem with the same file, but on another calculator.

Senior Member
Posts: 141
Joined: Wed May 16, 2012 2:50 pm
Location: Portugal
Calculators: Casio fx-CG20

Re: Problems with file copying function

Postby gbl08ma » Thu Nov 28, 2013 9:13 pm

Since you said that writing few bytes isn't a problem for WriteFile (and in fact, I recall that in the end we found out the problem was with writing a buffer located in flash and not in RAM), I simplified the code. I also reduced the scope of the copy buffer, and I'm now saving on the use of a memset. The problem still persists, with some files copying correctly and others almost always failing, as before. In the cases when it doesn't copy, the files are always created, but the WriteFile never seems to occur as the files are empty (all 0x00).
The target and PC are always the same (the PC may vary a bit depending on what code I have, of course):
TARGET=003094B2 (why is it even jumping here?!?)
PC=081000FC (definitely inside my add-in)

See the new code here: https://github.com/gbl08ma/utilities/bl ... r.cpp#L144

I can provide a g3a of the Utilities add-in build manifesting the problem, on request. (The latest public release also has the problem, but with a different target and PC and they vary from time to time, I think).

I am fully sure this has nothing to do with MsgBoxPushes and Pops, as this errors even with that code commented.

Senior Member
User avatar
Posts: 605
Joined: Sat Sep 15, 2012 6:59 am
Location: Krautland ****
Calculators: Casio fx-7400GII, Casio fx-7400GII (SH4), Casio fx-9750GII, Casio fx-9750GII (SH4), Casio fx-9860G, Casio fx-9860G SD, Casio fx-9860G Slim, Casio fx-9860GII SD, Casio fx-9860GII SD Power Graphic 2, Casio Classpad 330 plus, Casio fx-CG20, Casio fx-CG50, Casio Classpad fx-CP400

Re: Problems with file copying function

Postby SimonLothar » Fri Nov 29, 2013 7:33 am

gbl08ma wrote:TARGET=003094B2 (why is it even jumping here?!?)
PC=081000FC (definitely inside my add-in)
The (virtual) address 0x081000FC is inside your add-in's static RAM.
Do you use code in RAM?
What is the size of static RAM, which your add-in requires?
I'll be back!

Senior Member
Posts: 141
Joined: Wed May 16, 2012 2:50 pm
Location: Portugal
Calculators: Casio fx-CG20

Re: Problems with file copying function

Postby gbl08ma » Fri Nov 29, 2013 2:58 pm

Do you use code in RAM?


I do not intentionally run code from RAM, unless GCC is trying to optimize it for me by making certain things run from there without my request.

What is the size of static RAM, which your add-in requires?


I'm not sure how to check, is it the size taken by all the things I specifically declared as "static" in my code? If I recall correctly, I mostly have static ints declared (the settings variables), but there are no large buffers declared as static.

Senior Member
User avatar
Posts: 605
Joined: Sat Sep 15, 2012 6:59 am
Location: Krautland ****
Calculators: Casio fx-7400GII, Casio fx-7400GII (SH4), Casio fx-9750GII, Casio fx-9750GII (SH4), Casio fx-9860G, Casio fx-9860G SD, Casio fx-9860G Slim, Casio fx-9860GII SD, Casio fx-9860GII SD Power Graphic 2, Casio Classpad 330 plus, Casio fx-CG20, Casio fx-CG50, Casio Classpad fx-CP400

Re: Problems with file copying function

Postby SimonLothar » Fri Nov 29, 2013 9:24 pm

gbl08ma wrote:I'm not sure how to check, is it the size taken by all the things I specifically declared as "static" in my code? If I recall correctly, I mostly have static ints declared (the settings variables), but there are no large buffers declared as static.
Your SDK's linker should produce a map-file, where the information should be stored.

I did not get the hang of how to copy files with your utility, hence I cannot reproduce the error. Gimme some hints, please.

According to the memory locations of your official release, 0x081000FC is inside of the uninitialized static RAM. The GCC is not allowed to use this range on its own account.

Perhaps you should upload the actual version of the g3a somewhere, as you already proposed.
I'll be back!

Senior Member
Posts: 141
Joined: Wed May 16, 2012 2:50 pm
Location: Portugal
Calculators: Casio fx-CG20

Re: Problems with file copying function

Postby gbl08ma » Fri Nov 29, 2013 9:52 pm

SimonLothar wrote:Your SDK's linker should produce a map-file, where the information should be stored.
The linker is GCC's LD, I'm not sure where that map-file goes or if it even exists.
I do have the rules that tell the linker where each section of code goes, and I also have the crt0.S that is used on the add-in start.

SimonLothar wrote:I did not get the hang of how to copy files with your utility, hence I cannot reproduce the error. Gimme some hints, please.
I'm sorry, copying files is currently more or less hidden because it's unstable (the whole reason why I'm here).
You need to go in the add-in settings (press Shift then Menu) then enable the option "Show advanced tools". Now, go back to the main screen with the clock, press F4 and select the first menu option.
To copy a file, you first select it (press F1), then you press F3 (the copy function only appears when "advanced tools" are enabled). The selected files will be added to clipboard (you may see a "hint" message, ignore it). Now move to another folder (or create one with F4 if necessary) and do Shift then 9 to paste the clipboard contents (it will either move or copy depending on what you pressed when you added the files to clipboard).
Note that as I said, not all files show the problem. If you don't get an error with one file, try with another, until you get an error...

SimonLothar wrote:Perhaps you should upload the actual version of the g3a somewhere, as you already proposed.
I will send you a PM with a link to the latest build of Utilities.

Senior Member
User avatar
Posts: 605
Joined: Sat Sep 15, 2012 6:59 am
Location: Krautland ****
Calculators: Casio fx-7400GII, Casio fx-7400GII (SH4), Casio fx-9750GII, Casio fx-9750GII (SH4), Casio fx-9860G, Casio fx-9860G SD, Casio fx-9860G Slim, Casio fx-9860GII SD, Casio fx-9860GII SD Power Graphic 2, Casio Classpad 330 plus, Casio fx-CG20, Casio fx-CG50, Casio Classpad fx-CP400

Re: Problems with file copying function

Postby SimonLothar » Sat Nov 30, 2013 2:15 pm

I tried to trace down the problem and located it to be inside of
sys1dae:Bfile_CreateEntry_OS.

The problem should occur with g3a-files only.
sys1dae:Bfile_CreateEntry_OS calls sys0daf:SMEM_CreateEntry (in case of fls0-files). After SMEM_CreateEntry returns, sys0028 is called. sys0028 scans especially for g3a-files and registers the add-ins (with sys0015) anew. That operation kills any running add-in, because the TLB is affected, too. Bfile_CreateEntry_OS runs properly to its RTS and then tries to return to the calling add-in, which has attained nirvana meanwhile, where the calculator follows with good grace.

Solution: never create or delete g3a-files during the run of your add-in.
I'll be back!

Next

Return to Casio Prizm SDK

Who is online

Users browsing this forum: No registered users and 11 guests