|
Post by TrekkiesUnite118 on Sept 10, 2018 4:05:08 GMT
This is a repost from what I posted on SegaXtreme. But over the weekend I thought I'd snoop around in Grandia's files and do some comparing and contrasting with the US and JP PS1 releases to see how different they are. The thought process was to see if it would be possible to extract the English script out of the PS1 version, and possibly use that as a base to translate the Saturn version. So far this is what I found: - The File Structure between the two games is actually rather similar, unlike that of Lunar Silver Star Story.
- That said there are still some differences. Some files don't exist on the Saturn version and some don't exist on the PS1 version. However this seems to be more aimed at bin files and system specific formatted stuff like audio files, video files, fonts, etc.
- I believe I've found the script for the English version. In both versions there's a Field directory. In this directory there are numerous files with hex values for their file names. In the Saturn data these are .MDT files, in the PS1 version these are .MDP files. I'm pretty sure these contain the games script.
- The English PS1 script is completely uncompressed in the .MDP files. You can bring them up in a Hex Editor and read the ASCII encoded script.
- The font for the Saturn version is in the FDATA.BIN directory as mentioned before by CyberWarriorX on SegaXtreme, and his compression and decompression tools that he wrote appear to work just fine. If you run them on FDATA.BIN you can get the 4BPP font and can load it up in a tile editor of your choice.
- The FONT does have an all Caps English font in it, but from the looks of things I'd bet this one isn't a variable width font.
- I've not been able to really find where the menu text lives. In the PS1 version I've found a couple pieces here and there. For example in the Battle directory there's some pieces in it's DAT files, but these files appear to be one of the few areas where the Saturn and PS1 versions are structured differently. Basically some files the PS1 has don't exist on the Saturn and vice versa.
The Saturn version also recently had a Korean Translation released. So I took al peak at the Korean translated version and this is what I've found: - File sizes appear to be identical to the Japanese version.
- The font has definitely been changed as I'd expect it to be.
- The MDT files have been modified, which would seem to confirm that those contain the script.
- ITGET.BIN has been modified. Looks like they changed character names from Japanese to somewhat English versions. Which looking at the changes and gameplay footage, I'm guessing this is where the menu text lives. This is probably the equivalent of the ITEM.BIN file in the PS1 version.
- CMDMN.BIN has been changed. Again this looks like either Menu text, guessing command related based on what's in it and the name.
- CNFIG.BIN has also been changed. This again looks to be like menu text, guessing Config menu based on the name.
- SHOP.BIN has been changed. This is most likely Shop Menu text by the looks of things.
- SVLD.BIN has been changed. Again looks to be Menu text, I'm guessing this is for the Save/Load screen based on whats in it and the name.
- All the AMAP#.ADD files have been modified. My guess is this is text in the map screens or dungeons?
- WDATA.DAT has been altered, my guess is this is Weapons info maybe?
- SWN6M.DAT has been altered, no idea what this one is.
I then dug into the English PS1 version a bit more. I was able to get the PS1 English font. After looking at it I'm starting to wonder if the English version actually supports VWF. Reason being the English font is 8x16 in size, but every letter is scaled to that size and is perfectly centered. And when watching it in action you can actually see some spots where the spacing looks a bit odd, specifically around thin letters like Lower case L's and I's: And here are the fonts for each version of interest. Original Japanese Saturn: Hacked Korean Saturn Font: English PS1 Font: There were some 8x16 Japanese characters in the English PS1 font as well, but I didn't see a reason to include them. More findings continued in the next post...
|
|
|
Post by TrekkiesUnite118 on Sept 10, 2018 4:05:27 GMT
In comparing the script files between the Korean version and the Japanese version I think I've figured out some of the encoding. I still need to figure out the control codes for the text boxes, but the text itself is actually pretty simple. Basically the hex values for the text correspond to the sequence number of the font character in the font table. However the values are the decimal value of the font sequence number + 31, and then turned into Hex. So for example looking at the font charts above, the character A is 11th in the font table. If you want to have A displayed, you need to put the value 2A in the MDT file. This is because 11 + 31 = 42, and 42 in Hex is 2A. I was able to test this theory by replacing the text in the intro scene so in the first two text boxes Baal should now say: And low and behold: As I said, I still need to figure out the control codes, but if someone wants to take a look at it, the file for this area is BA38.MDT in the FIELD directory. The offset for this cutscene and where I started making changes is x5AA. Now the odd thing here is there's over 1000 characters in the font if I'm counting correctly. But this encoding can at most go up to 255 if I'm understanding it correctly. So I'm not sure how it's dealing with characters beyond that limit. However for an English hack we might not even need to care about that as we wouldn't need over 1000 characters to begin with. So honestly, once the control codes are figured out I think this might actually be a pretty straight forward hack. The biggest issue to do first would be to change the game to operate on an 8x16 font instead of a 16x16 font so the English PS1 font can be used. After that it would just be injecting the new font and then editing the script files. This is probably what they did for the Korean translation, but without the font size change, as Korean is still a Fixed Width Font so they could stick with the 16x16 font size easily. So this is what I've found over the weekend. I figured I'd post it here as well since I figured you guys would find this of interest.
|
|
|
Post by bultje112 on Sept 10, 2018 7:57:53 GMT
holy shit. I would sell organs for this to happen! I never played the ps1 version since I hate the english dub there. this is a true saturn game!
|
|
|
Post by TrekkiesUnite118 on Sept 11, 2018 4:56:37 GMT
So doing some more digging I've found a couple more nuggets of info: - The Japanese PS1 and Japanese Saturn version use the same encoding for their text.
- The English PS1, Japanese PS1, and Saturn versions seem to use the same control codes with some catches
- The PS1 versions seem to add additional values in the codes between text lines/new boxes.
- Some codes are also used differently.
So with the control codes, I've figured this out looking at the file BA38.MDP/BA38.MDT in the different versions. This is the file that contains the intro scene I edited earlier among other things. When the scene starts, we can see this code before the text across all 4 versions: 0901 090B 8020 0F03 0A0C 0030 It would seem that this indicates the following: - Start a new text box for a new character.
- Text Box is on Top
- Character Portrait is on the Left
- The character is Baal.
- Face palming.
When he speaks again however, we now see this: 0901 0F03 0A0C 0030 This seems to indicate the following: - Start a new text box
- Same Position
- Same Portrait Position
- Character is Baal.
- Face Palming.
The next big code we see is this: 0901 0F03 0A0C 002E This seems to indicate the following: - Start a new text box
- Same Position
- Same Portrait Position
- Character is Baal.
- Relaxed face.
Next in this scene the character Mullen is the one speaking so we now see this: 0901 090B 8021 0F21 0A0C 0022 This seems to indicate the following: - Start a new text box for a new character.
- Text Box is on the Bottom
- Character Portrait is on the Right
- The character is Mullen.
- Serious? Face.
Next, Baal speaks again, only now with this code and it's slightly different in the English PS1 version. In the Japanese and Korean versions it's: 0108 0F03 0A0C 002F In the English version it's this: DC08 0F03 0A0C 002F This would seem to reflect the following: - Use the previous text box settings (Position, Portrait Position, etc.).
- Character is Baal again.
- Mild Chuckle Face?
Next Mullen speaks again and we see a code that differs between the English version and the other versions. In the Japanese and Korean versions it's: 0108 0F21 0A0C 0021 In the English version it's this: AA08 0F21 0A0C 0021 This seems to reflect the following: - Use the previous text box settings (Position, Portrait Position, etc.).
- Character is Mullen again.
- Relaxed face.
So this would seem that the value 0108 does mean use the previous text box settings. Next we see this code come up quickly: 01 0F21 0A0C 0026 This seems to indicate that in the same text box mid sentence we need to change the face portrait to a different one. This is used in the cutscene to change Mullen's expression. It would seem that the value 01 is an interrupt to tell the game to change portraits. Now I can keep going but I think the patterns should start to become obvious. I've also noticed the following values coming up to signify the end of a line in the text box or the end of a textbox: 0E00
090F The problem however is I can't seem to find consistency for which is used where. The PS1 sometimes uses 0E00 where the Saturn version uses 090F, but then there's instances where both are using the same values. Now after the end code however, there seems to be varying bytes of data. The size of bytes seems to be the same in the Japanese and Korean Saturn versions, and in many cases the same values. However when we compare to the Japanese PS1 version, the size of bytes between these gaps is different. There's additional data added in. However there is still some data that matches, however bytes are flipped which is consistent with endian differences. This would seem to indicate these are perhaps pointers. I have no idea what the additional data that's added into the PS1 version is though. Obviously more research is needed here as well as some experimentation, but this seems to be consistent when stepping through the script file in sync with the cutscenes so far.
|
|
|
Post by TrekkiesUnite118 on Sept 13, 2018 3:38:29 GMT
So I decided to play around with the font tonight. I did a simple test where I took the English PS1 font and stretched it to now be 16x16 in size. I then copy pasted it into the Japanese Font file in ASCII order. This way in theory the script files could be edited into English with ASCII encoding to make things easy. And other than a few letters being off, it proved to be successful: I'm attributing the few incorrect characters here to two things: 1) I was probably off in my count for the ... character. 2) I probably missed a control code that I haven't figured out yet that controls Kanji characters. Obviously I didn't edit much beyond this point but the game still prints the font out ok. It's just now everything is truly gibberish as expected:
|
|
|
Post by digichan on Sept 15, 2018 0:07:29 GMT
wow nice hope you can keep this up! Currently playing this again because of the switch announcement
|
|
|
Post by TrekkiesUnite118 on Sept 15, 2018 7:19:45 GMT
So I've started putting together a font table for the game as I'm figuring out the encoding more. I have it done for the English portions of the Font, the Hiragana, and the Katakana as well as the special characters. I think I have an idea of how the Kanji encoding works, but it's going to be some trial and error to get it right. I figure if I can get this made it should make it at the very least easier to identify the Japanese text in a hex editor, as well as possibly make it easier to dump the script. It could also make it easier to spot the control codes. I've also been thinking about the issue of getting the font size changed from 16x16 to 8x16. CyberWarriorX has some good notes in this old thread about how the font and it's compression system works: segaxtreme.net/threads/saturn-hacking-notes.4934/Basically I think this is going to require some ASM hacking of the text decompression routine and possibly the text drawing routine. The Decompression routine works by decoding the first 8x16 portion of the character, followed by the next 8x16 portion. I'd imagine this would need to be changed so only the first half is decoded and treated as one character. I'd also imagine that this would impact the pointer table of the FDATA.BIN file. Finally depending on if the game is spacing the text based on font width or if it's spacing it in 16x16 increments regardless of it's size, would determine if the text drawing routine would need to be modified. This is something that is way above my skill level so I'm probably going to need some help on this one. I was planning on reaching out to CyberWarriorX and seeing if he had any thoughts on it or if he wanted to jump in and help. If anyone here wants to give it a shot feel free to jump in. I was also thinking about reaching out for help on ROMHacking.net. I have however had some success figuring out a bit more about the control codes. The code 090F that I before thought was end of text I believe is actually 1 code plus the start of another. The first code is 09. This seems to indicate a pause in text drawing. It seems to be used to pause the text to keep it in sync with pauses in the voices. The next code is actually 0FXX. The XX I've seen be any value from 01 up to 0A so far. This seems to be used for various different things. What I've found so far though is the following: 0F03 seems to be related to controlling the Left Portrait. 0F21 seems to be related to controlling the Right Portrait. The next code I was able to figure out is this one: 0A 0C 00 XX This code tells the game to switch to a different portrait, where XX is the value for the portrait you want displayed. So for example if you see the following: 0F 03 0A 0C 00 XX It would appear to mean "change the left portrait to portrait XX". I was able to confirm this in testing by changing the portraits in the intro cutscene to arbitrary values: There's still quite a few different values for 0F XX however, and I haven't quite been able to nail down what they all do yet.
|
|
|
Post by zyrobs on Sept 15, 2018 11:27:59 GMT
Reminds me of the progress I made on Madou Monogatari, but that one has the script files (and everything else) compressed in a custom format, so I couldn't actually put stuff on the disc, only edit them in memory.
|
|
|
Post by TrekkiesUnite118 on Sept 16, 2018 1:18:29 GMT
So after making a topic on RomHacking.net it was revealed that the 8x16 English font is actually in the Saturn version, and it's in ASCII format. To use it you just need to start your text line with the code 03 and then you can write ASCII encoded text. I tested this and got these results: So basically this makes things a lot easier I'd imagine. Now it's just figure out the control codes, any pointers in the files for text size and location. With that information it probably wouldn't be too difficult to write a tool to dump the JP script, and then another to inject an English script with pointer values adjusted for expanded text. Though it would still probably be beneficial to make some kind of hack to make the 8x16 font the default font so you don't have to waste a byte with the 03 control code. I'm guessing this is what the English PS1 version did as I'm not seeing any evidence of a control code like that being used in it's files. EDIT: So I think I might be figuring out how the script pointers work based on comparing the PS1 versions, and then spot checking with the Saturn version. So here is what I believe to be one complete text routine from the file BA38.MDT at x05B4: This highlighted portion I believe is a header of some kind. It seems to change from text sample to text sample, so there's probably optional values that are in some text boxes but not in others. Now this next highlighted portion is the important part. This value I believe represents how many bytes until the end of this text box. In this example it's 5A: If we then move forward 5A that will bring us to the end of this block of data. The end of a block of text data seems to be tagged with this possible footer: After this the next text box starts. I've been able to keep following this pattern and consistently keep finding what appears to be the correct number of bytes and the start of the next text box. I'm going to keep testing this but so far it's looking promising. Also, another interesting thing of note is that it seems that the English PS1 versions files seem to be close to if not the same size as the Japanese PS1 files. It looks like Sony removed unnecessary padding to keep the files close to the same size from what I can tell.
|
|
mick_aka
Kickin' it lively!
"Mick is moderately adequate."
Joined: April 2007
Posts: 9,817
Location:
XBL: mickloaf
PSN: mickloaf
Nintendo ID: segamick
|
Post by mick_aka on Sept 17, 2018 7:02:03 GMT
This is very encouraging! keep us posted
|
|
|
Post by TrekkiesUnite118 on Sept 18, 2018 4:43:48 GMT
Ok, so I think I've cracked the code of the MDT/MDP file headers. This information seems to hold true for both the Saturn and PS1 files, the only difference being that in the PS1 files pointers are Little Endian where as in the Saturn files they're Big Endian. - The header is 512 Bytes in size.
- The header seems to consist of just a pointer table
- Each entry is 8 bytes total in size
- The first 4 bytes seem to be the place in the file where the data lives
- The second 4 bytes seem to be the size of the data.
- The entry that tells us where the text lives and how big it is seems to be at x60 every time.
- There's an entry at x58 that seems to point to a 24 byte piece of data right before the text. Perhaps this is a header or something?
- Consistently I'm seeing in the English PS1 version that any addresses that come after the address defined at x60 have their values increased by the increase in size of the text data, and any values that come before it are untouched.
- I'm seeing that padding is being removed in the PS1 files of the exact amount of bytes added to the text.
- Another oddity I'm seeing is that the last entry in the table at x1F8 seems to also have it's size increased by the increase in text size. But I'm not sure what that value represents or what data was added.
Some other oddities I'm seeing is that some entries don't seem to make sense as the addresses are beyond the size of the file, but they're still updated in the English PS1 version. With this info I imagine I should be able to make a program that can do the following:
- Parse the MDT files.
- Extract the script portions.
- Put the MDT file back together with a new script portion.
- Recalculate the pointer table.
The hard part to automate in code I'd imagine would be to find a portion of padding in the file that can be removed to keep the file size the same.
|
|
|
Post by TrekkiesUnite118 on Sept 22, 2018 20:36:58 GMT
So I wrote a program that allows me to parse the MDT and MDP files from the Saturn and Playstation versions. This basically strips the first 512 bytes and reads them as a pointer table. This information is then used to parse the rest of the file. Knowing that 0x60 marks the table entry for the script, I wrote some code to copy the bytes defined in the table and write them to a file. This has essentially allowed me to dump the raw script for the game out of all the MDT/MDP files.
I've made it so it also writes the pointer table to a file, as well as the pre-script and post-script sections to files. This way it should be easy to make some code to do the reverse and put all the pieces back together and recalculate the pointer table.
On to the more interesting stuff though. After dumping the script for the Japanese Saturn, Japanese PS1, and English PS1 versions, I discovered that the Saturn and Japanese PS1 script data is identical. The initial cutscene I was looking at appears to be a special snowflake where they've added a bunch of additional control codes in the PS1 version. After that, things start lining up between the two. The only difference is the non-script parts of the data are byte swapped due to endian differences.
This is making it really easy to start identifying pointer values and control codes by comparing the different versions. I'm still figuring things out in these files, but I'm starting to think it might be possible to write a program that could automatically convert the English PS1 script files into a format the Saturn is expecting.
|
|
|
Post by darkman01 on Sept 23, 2018 9:41:39 GMT
Grandia is a favourite of mine, I wish you the best with this project.
|
|
sych
"Living for the fantasy" -NIGHTS INTO DREAMS- The Classic Saturn Gem!!!
Joined: July 2009
Posts: 2,660
Location:
|
Post by sych on Sept 23, 2018 23:08:00 GMT
Likewise. I absolutely loved Saturn Grandia. As someone that slogged through the game with a translation script and completed it, it would be nice to revisit it without having to repeat the same formula. I wish you all the best with this.
|
|
|
Post by darkman01 on Sept 25, 2018 8:09:35 GMT
Eventually I might end up getting a secondary Saturn just for playing homebrew games, either on CD-R or flash based solution , as Id rather not wear out my main Saturn's laser with burnt homebrew discs.
|
|