Some hints concerning the G1S-type

[SimonLothar]

The emulator of the CASIO SDK used a file default.G1S to emulate the storage memory.
This file contains zeros up to 0x00270000. The space beginning at 0x00270000 is used as storage memory. This corresponds with the location of the storage memory in real fx-9860-calculators.
This seems to be the single official use.

G1S has been used as extension for complete image-backups of fx-9860 calculators, too.

Some information concerning the structure of the storage memory (hence the format of the significant parts of default.G1S) can be found here:
(3.edit)

G1S walk: Show

Code: Select all

/*
The storage memory on a fx-9860 starts at 0x00270000.
The first block consists of 25 (1.edit) TG1SDirectoryLocator.TG1SSectorDef elements (ID:0x_200).
Then follow TG1SDirectoryLocator.TG1SDirectoryDef (ID:0x_110) or TG1SDirectoryLocator.TG1SFileDef (ID:0x_120) structures.
A TG1SDirectoryLocator.TG1SDirectoryDef (ID:0x_110) structure consists of the directoryname only.
A TG1SDirectoryLocator.TG1SFileDef (ID:0x_120) structure consists of a dirref1, a dirnum and the directoryname. dirref1 is 0x0110 if the file resides in a subdirectory. The dirnum represents the consecutive number of the TG1SDirectoryLocator.TG1SDirectoryDef-structures. dirref1 and dirnum are 0xFFFF both, if the file resides in the root.
A TG1SDirectoryLocator.TG1SFileDef (ID:0x_120) structure is immediately followed by one or more TG1SDirectoryLocator.TG1SFragment (ID:0x_130) structures, which can be used to localize the file fragments.
Search TG1SSectorDef.LogicalSectorNumber == TG1SFragment.HI_logicalsectornumber in the TG1SSectorDef list, giving the hi word of the physical fragment address (TG1SSectorDef.SectorStartAddress).
TG1SFragment.LO_physicalsector represents the lo word of the physical fragment address (refer to the function GetFragnmentAddress below).
TG1SFragment.blocksize+1 is the size of the fragment (thus a short can repesent a blocksize of 0x10000).
If the high nibble of ID of a (ID:0x_120) structure is zero, the file has been marked as deleted. These entries are eliminated when optimizing.
Using this information, a deleted file can be recovered, as long as no optimization has been performed since the file deletion.
*/

#define idSectorDEF 0x0200
#define idDirectoryDEF 0x0110
#define idFileDEF 0x0120
#define idFragmentDEF 0x0130

typedef struct {
   unsigned int SectorStartAddress;
   unsigned int LogicalSectorNumber;
   char dummy[20];
} TG1SSectorDef;

typedef struct {
   unsigned int dirref;
   short Name[12];
} TG1SDirectoryDef;

typedef struct {
   unsigned short dirref1;
   unsigned short dirnum;
   short Name[12];
} TG1SFileDef;

typedef struct {
   unsigned int FileID;
   unsigned short FileType;
   unsigned short FragmentCount;
   unsigned short FragmentNumber;
   unsigned short HI_logicalsectornumber;
   unsigned short LO_physicalsector;
   unsigned short BlockSize;
   char dummy[12];
} TG1SFragment;

typedef struct {
   short ID;         
   short UniqueNumber;   
   union{
      TG1SSectorDef Sector;
      TG1SDirectoryDef Directory;
      TG1SFileDef File;
      TG1SFragment Fragment;
      unsigned char dummy[28];
   };
} TG1SDirectoryLocator;

#define SIZE001 30000

//
int GetFragnmentAddress( TG1SDirectoryLocator*pFragment ){
   int result = 0;
   int i;
   TG1SDirectoryLocator*pSector;
   
   // search the sectorlist for a specific logical sector number
   pSector = (TG1SDirectoryLocator*)0x80270000;
   for ( i = 0; i<25; i++ ){ // (1.edit)
      if ( (*pSector).Sector.LogicalSectorNumber == (*pFragment).Fragment.HI_logicalsectornumber ){
         result = (*pSector).Sector.SectorStartAddress;
         break;
      }
      pSector++;
   }   
   // on success calculate the address of the fragment
   if ( result ) result += (*pFragment).Fragment.LO_physicalsector;
   return result;
}

// ************************************************
// enumerate the storage memory
int F06_3_Handler(){
char buffer[SIZE001];
char hb[40];
void*p;
int iii, j;
short w;

TG1SDirectoryLocator*sectorlist;

   memset( buffer, 0, SIZE001 );
   
   strcpy( (char*)hb, "SMEM walk" );
   strcat( buffer, hb );
   strcat( buffer, "\x0D\x0A" );
   
   p = (void*)0x80270000;
   iii = 0;
   while ( iii++ < 0x800 ) { // (2.edit)
      if ((*(TG1SDirectoryLocator*)p).ID != (short)0xFFFF ){
         WordToHex( (*(TG1SDirectoryLocator*)p).ID, (unsigned char*)hb );
         hb[4]=0;
         strcat( buffer, hb );
         strcat( buffer, " " );
         
         switch ( (*(TG1SDirectoryLocator*)p).ID & 0x0FFF ){
            case idSectorDEF :
               IntToHex( (*(TG1SDirectoryLocator*)p).Sector.SectorStartAddress, (unsigned char*)hb );
               hb[8]=0;
               strcat( buffer, hb );
               if ( (*(TG1SDirectoryLocator*)p).Sector.LogicalSectorNumber != 0xFFFFFFFF ){
                  strcat( buffer, " " );
                  IntToHex( (*(TG1SDirectoryLocator*)p).Sector.LogicalSectorNumber, (unsigned char*)hb );
                  hb[8]=0;
                  strcat( buffer, hb );
               }   
               break;
               
            case idDirectoryDEF :
               if ( ( (*(TG1SDirectoryLocator*)p).ID & 0xF000 ) == 0 ){ // (3.edit)
                  strcat( buffer, "(D) " );
               }
               if ( (*(TG1SDirectoryLocator*)p).Directory.dirref != 0xFFFFFFFF ){
                  IntToHex( (*(TG1SDirectoryLocator*)p).Directory.dirref, (unsigned char*)hb );
                  hb[8]=' ';
                  hb[9]=0;
                  strcat( buffer, hb );
               }   
               
               memset( hb, 0, sizeof( hb ) );
               for ( j=0; j<12; j++ ){
                  w = (*(TG1SDirectoryLocator*)p).Directory.Name[ j ];
                  if ( w == 0 ) break;
                  if ( w == (short)0xFFFF ) break;
                  hb[ j ] = w;
               }
               strcat( buffer, "[" );
               strcat( buffer, hb );
               strcat( buffer, "]" );
               break;
               
            case idFileDEF :
               if ( ( (*(TG1SDirectoryLocator*)p).ID & 0xF000 ) == 0 ){
                  strcat( buffer, "(D) " );
               }
               if ( (*(TG1SDirectoryLocator*)p).File.dirref1 != (unsigned short)0xFFFF ){
                  WordToHex( (*(TG1SDirectoryLocator*)p).File.dirref1, (unsigned char*)hb );
                  hb[4]=' ';
                  hb[5]=0;
                  strcat( buffer, hb );
               }
               if ( (*(TG1SDirectoryLocator*)p).File.dirnum != (unsigned short)0xFFFF ){
                  WordToHex( (*(TG1SDirectoryLocator*)p).File.dirnum, (unsigned char*)hb );
                  hb[4]=' ';
                  hb[5]=0;
                  strcat( buffer, hb );
               }   
               memset( hb, 0, sizeof( hb ) );
               for ( j=0; j<12; j++ ){
                  w = (*(TG1SDirectoryLocator*)p).File.Name[ j ];
                  if ( w == 0 ) break;
                  if ( w == (short)0xFFFF ) break;
                  hb[ j ] = w;
               }
               strcat( buffer, hb );
               break;
               
            case idFragmentDEF :
               
               IntToHex( GetFragnmentAddress( (TG1SDirectoryLocator*)p ), (unsigned char*)hb );
               hb[8]=' ';
               hb[9]=0;
               strcat( buffer, hb );
            
            
   //            IntToHex( (*(TG1SDirectoryLocator*)p).Fragment.FileID, (unsigned char*)hb );
   //            hb[8]=' ';
   //            hb[9]=0;
   //            strcat( buffer, hb );
   //            
   //            WordToHex( (*(TG1SDirectoryLocator*)p).Fragment.FileType, (unsigned char*)hb );
   //            hb[4]=' ';
   //            hb[5]=0;
   //            strcat( buffer, hb );
   //            
   //            ByteToHex( (*(TG1SDirectoryLocator*)p).Fragment.FragmentCount, (unsigned char*)hb );
   //            hb[2]=' ';
   //            hb[3]=0;
   //            strcat( buffer, hb );
   //            
   //            ByteToHex( (*(TG1SDirectoryLocator*)p).Fragment.FragmentNumber, (unsigned char*)hb );
   //            hb[2]=' ';
   //            hb[3]=0;
   //            strcat( buffer, hb );
   //            
   //            ByteToHex( (*(TG1SDirectoryLocator*)p).Fragment.HI_logicalsectornumber, (unsigned char*)hb );
   //            hb[2]=' ';
   //            hb[3]=0;
   //            strcat( buffer, hb );
   //            
   //            ByteToHex( (*(TG1SDirectoryLocator*)p).Fragment.LO_physicalsector, (unsigned char*)hb );
   //            hb[2]=' ';
   //            hb[3]=0;
   //            strcat( buffer, hb );
   //         
   //            strcat( buffer, "\x0D\x0A" );
   //            strcat( buffer, "     " );
         
               j = (*(TG1SDirectoryLocator*)p).Fragment.BlockSize;
               j++;
               IntToHex( j, (unsigned char*)hb );
               hb[8]=' ';
               hb[9]=0;
               strcat( buffer, hb+3 );
               
               break;
               
            default : // (3.edit)
               WordToHex( (*(TG1SDirectoryLocator*)p).UniqueNumber, (unsigned char*)hb );
               hb[4]=0;
               strcat( buffer, hb );
               strcat( buffer, " " );
            
               IntToHex( (int)p, (unsigned char*)hb );
               hb[8]=' ';
               hb[9]=0;
               strcat( buffer, hb );
               
               break;
         }
         
   //      IntToHex( (int)p, (unsigned char*)hb );
   //      hb[8]=0;
   //      strcat( buffer, hb );
   
         strcat( buffer, "\x0D\x0A" );
      }   
      p = (void*)((unsigned int)p + 0x20);
   }
   
// now buffer could be written into a file.   

   return 1;
}


[cakeisalie5 Profile Website]

Oh, storage memory! I somehow understood it was raw MCS. I will use this to understand then, thanks <3

[cakeisalie5 Profile Website]

In fx_legacy_SMEMstructure.htm, you say there are 25 sectors ("not in the AU"?), and in your source code you say that there are 27...? (if you look at the history of this message, you'll find other questions I was able to answer myself :p)

Also, based on what you've found, I've started implementing parsing for storage files in libg1m: format description / parsing functions
Don't hesitate if you find something wrong :p

EDIT: So my parsing gave me that for a demo file:

Code: Select all

[libg1m info]  g1m_parse_storage: [01,01] Is a sector.
[libg1m info]  g1m_parse_storage: - Starts at address 0x00270000
[libg1m info]  g1m_parse_storage: - Logical number: 0x16711681
[libg1m info]  g1m_parse_storage: [02,02] Is an unused sector.
[libg1m info]  g1m_parse_storage: - Starts at address 0x00280000
[libg1m info]  g1m_parse_storage: [03,03] Is a sector.
[libg1m info]  g1m_parse_storage: - Starts at address 0x00290000
[libg1m info]  g1m_parse_storage: - Logical number: 0x00000001
...
[libg1m info]  g1m_parse_storage: [26,00] Is a active directory.
[libg1m info]  g1m_parse_storage: - Directory name: DEMOS
...
[libg1m info]  g1m_parse_storage: [29,00] Is a active file.
[libg1m info]  g1m_parse_storage: - File name: COMPLEXE.g1e
[libg1m info]  g1m_parse_storage: - Is in directory #1
[libg1m info]  g1m_parse_storage: [30,00] Is a fragment.
[libg1m info]  g1m_parse_storage: - Points at sector #1
[libg1m info]  g1m_parse_storage: - Is 1/1 (1615o at offset 0x0400)
...
[libg1m info]  g1m_parse_storage: [33,00] Is a active file.
[libg1m info]  g1m_parse_storage: - File name: COMPLEXE.g1e
[libg1m info]  g1m_parse_storage: - Is in directory #1
[libg1m info]  g1m_parse_storage: [34,00] Is a fragment.
[libg1m info]  g1m_parse_storage: - Points at sector #1
[libg1m info]  g1m_parse_storage: - Is 1/1 (2307o at offset 0x0e40)
...
[libg1m info]  g1m_parse_storage: [61,01] Is a active directory.
[libg1m info]  g1m_parse_storage: - Directory name: DEMOS
[libg1m info]  g1m_parse_storage: [62,01] Is a active file.
[libg1m info]  g1m_parse_storage: - File name: COMPLEXE.g1e
[libg1m info]  g1m_parse_storage: - Is in directory #1
[libg1m info]  g1m_parse_storage: [63,01] Is a fragment.
[libg1m info]  g1m_parse_storage: - Points at sector #1
[libg1m info]  g1m_parse_storage: - Is 1/1 (1615o at offset 0x5910)
[libg1m info]  g1m_parse_storage: [64,02] Is a active file.
[libg1m info]  g1m_parse_storage: - File name: INTEGRAT.g1e
[libg1m info]  g1m_parse_storage: - Is in directory #1
[libg1m info]  g1m_parse_storage: [65,02] Is a fragment.
[libg1m info]  g1m_parse_storage: - Points at sector #1
[libg1m info]  g1m_parse_storage: - Is 1/1 (915o at offset 0x5f60)
[libg1m info]  g1m_parse_storage: [66,03] Is a active file.
[libg1m info]  g1m_parse_storage: - File name: GENDARME.g1e
[libg1m info]  g1m_parse_storage: - Is in directory #1
[libg1m info]  g1m_parse_storage: [67,03] Is a fragment.
[libg1m info]  g1m_parse_storage: - Points at sector #1
[libg1m info]  g1m_parse_storage: - Is 1/1 (1271o at offset 0x6300)
[libg1m info]  g1m_parse_storage: [68,04] Is a active file.
[libg1m info]  g1m_parse_storage: - File name: SUITES.g1e
[libg1m info]  g1m_parse_storage: - Is in directory #1
[libg1m info]  g1m_parse_storage: [69,04] Is a fragment.
[libg1m info]  g1m_parse_storage: - Points at sector #1
[libg1m info]  g1m_parse_storage: - Is 1/1 (915o at offset 0x6800)

Facts about my parsing:
- Between brackets are the iterator and the ID.
- I consider a sector unused if `logical_sector_number == 0xFFFFFFFF`.

Questions:
- It looks like the first and second sectors are always used by the index, is that correct?
- Why is the directory there two times?
- Also, entry #26 to #60 have an ID equal to zero, and entries appear several times, then the ID is increment for each folder or file (independently; I don't have examples with several folders by I assume it does). Maybe you know more about this?

Thanks in advance!

[SimonLothar]

Of course there are 25 sectors, 27 has been a typo (I corrected the code snippet).

The AU-types featured only 14 sectors to reduce the amount of available storage memory (giving about 800 KiB of SMEM).
This has been done to meet Australian school regulations.

[SimonLothar]

I consider a sector unused if `logical_sector_number == 0xFFFFFFFF`.

This seems to be true.

It looks like the first and second sectors are always used by the index, is that correct?

Only the sector 00270000 is used by the index The sector 00280000 is used for data, but not necessarily used at first even if empty. The last entry in sector 00270000 features an ID == 0100, obviously designating the very end of the directory.

Why is the directory there two times?

A deleted directory is marked as deleted by setting the upper nibble of the ID to zero, too. I failed to heed for that in the first version of my code snippet. I corrected that. This could explain the occurence of multiple directory entries (one active and the others deleted). But this could not explain the occurence of IDs being zero. I never saw such ID==0-entries up to now. But you never can tell.

[SimonLothar]

Possibly I overlooked something. But what I did not understand is this:
(special == 0x05) ? "deleted" : "active"
should be
(special == 0) ? "deleted" : "active"

[cakeisalie5 Profile Website]

As your test files look different, here are mine (they come from a random course archive for french baccalauréat, I found it on Planète Casio, it's a used archive so it should be valid), so you can try to parse them using your tools and see what the differences are. I'm based on what's in there and I supposed no file/directory in it is deleted, but as I haven't tested it myself (having no PC left with MS-Windows on it), I can't really tell - that's why I used 0x05 as the deleted mark. ^^

[SimonLothar]

There are only deleted items (high nibble == 0) in Maths.g1s and Demonstrations.G1S.
I usually test with real storage memories. This is more reliable.

The fact, that the zero is the delete mark, is not random.
The flash allows to change set bits to cleared bits by flashwriting a short value only. So you can easily mark a file as deleted in the directory (flashwrite the ID, where the high nibble is cleared).
You cannot set bits in the flash this easy.

[cakeisalie5 Profile Website]

So that means FA-124 doesn't read the "deleted" special, and ignores the ID == 0 files and directories... right? Oo

Oh, and while I'm at it: is it possible to have less sectors, like: if two are needed, four sectors (first two for indexes, other two for data)? Or are the 25(/14) sectors mandatory/hardcoded? (that could be useful for making G1S smaller)

[SimonLothar]

So that means FA-124 doesn't read the "deleted" special, and ignores the ID == 0 files and directories... right? Oo

I do not know much about FA-124. I ceased using FA-124 shortly after I started the CASIO thing in 2008.

Oh, and while I'm at it: is it possible to have less sectors, like: if two are needed, four sectors (first two for indexes, other two for data)? Or are the 25(/14) sectors mandatory/hardcoded? (that could be useful for making G1S smaller)

The limitation to 800 KiB has been a software (OS) thing. If fxRemote would have existed in these days...