Editing the Database / Quick_Start .dat Files

This is the place to discuss all aspects of Franchise Hockey Manager by OOTP Developments.
Post Reply
User avatar
archibalduk
TBL Admin Team
Posts: 20372
Joined: Tue Jul 06, 2004 8:44 pm
Custom Rank: Seaside + Fruit Juice Mode
Favourite Team: Guildford (EPL) / Invicta (NIHL)
Location: United Kingdom
Contact:

Editing the Database / Quick_Start .dat Files

Post by archibalduk »

I've been taking a bit of a look through the FHM database files (those located at 'quickstart_games\Majors.quick'). It looks like editing them is a real possibility. So far, I've been able to extract the data from arenas.dat and save it to a csv spreadsheet:

Image

This is probably only of interest to those wanting to edit/explore the dat files via a hex editor or writing custom programs to edit the files:

Editing the files is going to be a little harder than with EHM. The bytes in the dat files are un-swapped (I think this is called big endian - whereas EHM, like most things, is little endian). So any tools have to byte swap before doing anything with the data. It does however mean that the bytes are easier to read manually via a hex editor. For example, the capacity of the Bell Centre is 21,273. In hex this is 53 19. With the usual binary files (such as EHM's dat files), this would be written as 19 53 because the bytes are swapped. However, in FHM's data files this is written as 53 19. When loading the files using C++ code, the function that loads the data (std::fstream) would expect the bytes to be swapped and so it reads FHM's dat files incorrectly. So you have to manually swap the bytes using your own code (it seems a little odd that you have to swap un-swapped bytes to view them, but there you go!). Here's an example of how to swap an unsigned short, such as the Capacity field, in C++:

Code: Select all

void endian(unsigned short &x)
{
	x = ((x << 8) & 0xFF00FF00 ) | ((x >> 8) & 0xFF00FF );
}
So once you've read a Capacity field from a dat file, you then call the above endian function to get the bytes the right way around. E.g. if you have an unsigned short named arena_capacity, here is how you'd call the function:

Code: Select all

endian(arena_name);
As with shorts, integers are also un-swapped. So whereas 1,527 (= 00 00 05 F7 in hex) would be stored in as F7 05 00 00 in an EHM dat file, it is stored as 00 00 05 F7 in FHM.

Additionally, text fields are of variable size. For those who have edited EHM, the text fields are of fixed sizes. In FHM, there is a short field (i.e. 2 bytes long) directly before the Arena Name field which states how long the Arena Name field is. I'm guessing this is going to be the case with other text fields in other dat files, but I haven't looked yet. Also, each character in a text string is stored as 2 bytes. For example, the letter 'A' is 41 in hex - but in FHM it is stored as 00 41.

I'm only a novice at programming, but I'm guessing there's more to this than how I've explained it. For example, the text fields may be 2 bytes because they're wide characters (UNICODE). And the big endianess may also have some logical reason/explanation. But at least this is a start for anybody wanting to write editors for FHM. I know I'd like to eventually try writing an FHM version of my EHM Updater so that we can collaboratively edit FHM.
User avatar
Alessandro
Olympic Gold
Posts: 2865
Joined: Sat Aug 27, 2005 1:54 pm
Custom Rank: TBL Rosters Man
Favourite Team: Team Russia
WHL Team: Calgary Flames

Re: Editing the Database / Quick_Start .dat Files

Post by Alessandro »

How much I do hope that we'll be able to get an editor... in-game editing is awful, it takes a huge lot of time even for the simplest tasks
User avatar
archibalduk
TBL Admin Team
Posts: 20372
Joined: Tue Jul 06, 2004 8:44 pm
Custom Rank: Seaside + Fruit Juice Mode
Favourite Team: Guildford (EPL) / Invicta (NIHL)
Location: United Kingdom
Contact:

Re: Editing the Database / Quick_Start .dat Files

Post by archibalduk »

Alessandro wrote:How much I do hope that we'll be able to get an editor... in-game editing is awful, it takes a huge lot of time even for the simplest tasks
Yeah the in-game editor isn't really suitable for any real roster update editing. It's only really ideal for in-game updating (e.g. updating league alignments and moving/folding teams, etc). But that's not to take away anything from the in-game editor - it's going to be fantastic to keep the league structures, etc in saved games up to date. 8-)

As for editing the data outside of the game, this is looking very possible. I'm working on figuring out the structure of players.dat at the moment. The data is structured very differently in FHM compared to EHM. The variable field lengths and variable fields are killing me, but once I've figured it out it'll be fine (I've figured out much of the structure for skaters within the file, but I'm not quite there with the goalies because it seems there are additional career history fields for goalies). Whereas EHM kept player data across a number of different files (e.g. contract/bio data in one file, career history in another, draft data in another, etc), FHM keeps all of this data in one file (players.dat). This makes it harder to figure out the structures; but once I've got it, it'll be so much easier than EHM in terms of writing an editor. And there's no darn index.dat file (instead the first four bytes of each .dat file keeps the record count for each file).

EDIT: It seems the players.dat file in the database has the exact same structure as players.dat in the saved game. This will lead to some interesting possibilities - and it makes figuring out the structure oh so much easier.
User avatar
nino33
Mr. Goalie
Posts: 6088
Joined: Sat Aug 07, 2010 3:37 am
Custom Rank: Retro Rosters Specialist
Favourite Team: 1970s hockey

Re: Editing the Database / Quick_Start .dat Files

Post by nino33 »

I'm certainly glad to see you working on an FHM editor Arch 8-)
User avatar
archibalduk
TBL Admin Team
Posts: 20372
Joined: Tue Jul 06, 2004 8:44 pm
Custom Rank: Seaside + Fruit Juice Mode
Favourite Team: Guildford (EPL) / Invicta (NIHL)
Location: United Kingdom
Contact:

Re: Editing the Database / Quick_Start .dat Files

Post by archibalduk »

Wow, there really is a lot of data in players.dat. So many variable fields (some of which I haven't figured out what they do yet). I've figured out a lot of it however (salary data, draft rights, career histories, attributes, TAs, PAs, fitness, positions, reputation, contentment, etc). I thought I'd figured out enough of the structure to start exporting data (being able to successfully export data indicates that I can start editing the data) but it crashes after correctly exporting the first 1,622 records. So it looks like I haven't quite got it yet... :-k
User avatar
Alessandro
Olympic Gold
Posts: 2865
Joined: Sat Aug 27, 2005 1:54 pm
Custom Rank: TBL Rosters Man
Favourite Team: Team Russia
WHL Team: Calgary Flames

Re: Editing the Database / Quick_Start .dat Files

Post by Alessandro »

Hurry up Archi, the game will need a lot of editing. I also wonder if they'd let us use the players templates in the game, it would be good for speeding things up
User avatar
archibalduk
TBL Admin Team
Posts: 20372
Joined: Tue Jul 06, 2004 8:44 pm
Custom Rank: Seaside + Fruit Juice Mode
Favourite Team: Guildford (EPL) / Invicta (NIHL)
Location: United Kingdom
Contact:

Re: Editing the Database / Quick_Start .dat Files

Post by archibalduk »

Alessandro wrote:Hurry up Archi, the game will need a lot of editing. I also wonder if they'd let us use the players templates in the game, it would be good for speeding things up
I'm getting there very slowly. The variable fields make progress really slow. This morning I was able to export about 22,400 of the 22,800 or so records with the v0.9.12 DB. However since the private beta released since then, the structure has changed subtly and so I can only export about half of the records. Nonetheless, I've figured out the vast majority of the structure - it's just the complex things that I need to work on.

Btw there's a lot of unexpected stuff in the DB - for example, some players have multiple sets of contract data (ie they have multiple sets of team contracted to fields and multiple sets of normal/minor years 1-10 salary fields). Also, there are some players in the file that don't appear in game and don't seem to exist on Elite Prospects (eg Landon Palmer and Mike English).
User avatar
archibalduk
TBL Admin Team
Posts: 20372
Joined: Tue Jul 06, 2004 8:44 pm
Custom Rank: Seaside + Fruit Juice Mode
Favourite Team: Guildford (EPL) / Invicta (NIHL)
Location: United Kingdom
Contact:

Re: Editing the Database / Quick_Start .dat Files

Post by archibalduk »

Alessandro wrote:I also wonder if they'd let us use the players templates in the game, it would be good for speeding things up
Yes, it looks like we can. :thup:

The templates are used in the database. There is an entry for every player in players.dat specifying the template for each player. So presumably when you start a new game, it generates the attributes for each player based on the templates.
User avatar
Alessandro
Olympic Gold
Posts: 2865
Joined: Sat Aug 27, 2005 1:54 pm
Custom Rank: TBL Rosters Man
Favourite Team: Team Russia
WHL Team: Calgary Flames

Re: Editing the Database / Quick_Start .dat Files

Post by Alessandro »

Of course it is used... i just wonder if in the future we'll just be able to enter the template and the game does its magic... like we do now in the researchers editor
User avatar
archibalduk
TBL Admin Team
Posts: 20372
Joined: Tue Jul 06, 2004 8:44 pm
Custom Rank: Seaside + Fruit Juice Mode
Favourite Team: Guildford (EPL) / Invicta (NIHL)
Location: United Kingdom
Contact:

Re: Editing the Database / Quick_Start .dat Files

Post by archibalduk »

Alessandro wrote:Of course it is used... i just wonder if in the future we'll just be able to enter the template and the game does its magic... like we do now in the researchers editor
I would expect so. If it were just a researcher thing, I wouldn't expect it to be in the DB. The fact that it is in the DB and used by the game suggests that it is a permanent feature.

With regards to figuring out the rest of the players.dat structure, I think I'm close to hitting a brick wall. There is a little more I'll be able to figure out, but the rest is a mystery. I think we'll be very dependent on OOTP publishing the structure. I looked at OOTP Baseball and it seems no one has ever successfully figured out the structure and OOTP has never published it - so I'm not sure how likely it is that OOTP will share FHM's structure. Having said that, there is still a fair chance that I'll be able to figure out enough to allow us to edit most of the data.
User avatar
archibalduk
TBL Admin Team
Posts: 20372
Joined: Tue Jul 06, 2004 8:44 pm
Custom Rank: Seaside + Fruit Juice Mode
Favourite Team: Guildford (EPL) / Invicta (NIHL)
Location: United Kingdom
Contact:

Re: Editing the Database / Quick_Start .dat Files

Post by archibalduk »

I've figured out the source of the crash! :joy: Writing an updating tool is now a real possibility. Even though I don't know the full structure, I have enough to be able to edit attributes and other bits. Contracts might be a bit more complex, but I think I should at least be able to implement trade/transaction updates.
User avatar
Alessandro
Olympic Gold
Posts: 2865
Joined: Sat Aug 27, 2005 1:54 pm
Custom Rank: TBL Rosters Man
Favourite Team: Team Russia
WHL Team: Calgary Flames

Re: Editing the Database / Quick_Start .dat Files

Post by Alessandro »

Fantastic. What about adding new players?
User avatar
archibalduk
TBL Admin Team
Posts: 20372
Joined: Tue Jul 06, 2004 8:44 pm
Custom Rank: Seaside + Fruit Juice Mode
Favourite Team: Guildford (EPL) / Invicta (NIHL)
Location: United Kingdom
Contact:

Re: Editing the Database / Quick_Start .dat Files

Post by archibalduk »

Alessandro wrote:Fantastic. What about adding new players?
Possibly. The difficulty is that I don't know the full structure and so I'd have to borrow data from existing records. This would mean that created players might end up having certain things by default (eg they might automatically have a particular clause in their contract).

The first thing I need to solve is the Eastern Euro accents. It's a little harder to sort out than with EHM. I'm not expecting to have anything up and running for some time - I need to get back to FHM researching.
User avatar
Alessandro
Olympic Gold
Posts: 2865
Joined: Sat Aug 27, 2005 1:54 pm
Custom Rank: TBL Rosters Man
Favourite Team: Team Russia
WHL Team: Calgary Flames

Re: Editing the Database / Quick_Start .dat Files

Post by Alessandro »

archibalduk wrote:
Alessandro wrote:Fantastic. What about adding new players?
Possibly. The difficulty is that I don't know the full structure and so I'd have to borrow data from existing records. This would mean that created players might end up having certain things by default (eg they might automatically have a particular clause in their contract).

The first thing I need to solve is the Eastern Euro accents. It's a little harder to sort out than with EHM. I'm not expecting to have anything up and running for some time - I need to get back to FHM researching.
They should solve it in the game itself...
User avatar
M Ace
Minor League
Posts: 233
Joined: Sat May 01, 2010 3:43 pm
Favourite Team: Edmonton Oilers
Location: Örebro, Sweden

Re: Editing the Database / Quick_Start .dat Files

Post by M Ace »

Great work, Archi!

I really hope you can figure it out so we can edit the most important things as trades, create new teams/players, insert draft/rights, stats history and maybe even create new leagues as a default outside the game. All honour to the in-game editor but we really need to be able to have this things edited before we even start the game!
cavaelen
Learning to skate
Posts: 3
Joined: Fri Oct 14, 2011 5:20 am

Re: Editing the Database / Quick_Start .dat Files

Post by cavaelen »

Thanks for the advice Archi. Helped me get started.

I have now got the arena table mapped too and outputting to a csv as a PoC.
Image

I started looking at the players table and found that the names etc were not stored directly in there so decided it would be better to figure out some of the other files first.

So have got part way through the names table. Figuring that the bit i dont have completely worked out (the last 9 bytes of each record) is something to do with countries/regions and their probabilities and possibly first/last name but just guessing at this stage till i do some more work. Also the first two bytes of each record always seem to be 0. This could be due to the name length being 4 bytes not the 2 im using currently or because its something not implemented yet.
Image
Image

Note also that the name.dat file changes structure towards the end completely. No idea what this is about yet. If anyone has any thoughts let me know.
Image

Figure i will map some of the sub tables like these first before i go back to working on the players table itself.
User avatar
archibalduk
TBL Admin Team
Posts: 20372
Joined: Tue Jul 06, 2004 8:44 pm
Custom Rank: Seaside + Fruit Juice Mode
Favourite Team: Guildford (EPL) / Invicta (NIHL)
Location: United Kingdom
Contact:

Re: Editing the Database / Quick_Start .dat Files

Post by archibalduk »

Nice work! =D>
cavaelen wrote:I started looking at the players table and found that the names etc were not stored directly in there so decided it would be better to figure out some of the other files first.
I'm glad you pointed that out. It looks like the DB structure has changed because when I was last working on editing the DB, the player names were in fact stored in the players table. It's things like this that have made me hold back from putting any more work into editing. The DB is still a work in progress (especially the teams/leagues/rules tables) and so I've been waiting things to settle down a little more - or at least until next month anyhow! :D
cavaelen wrote:Also the first two bytes of each record always seem to be 0. This could be due to the name length being 4 bytes not the 2 im using currently or because its something not implemented yet.
The name length will be 4 bytes (i.e. it's an int). All of the name lengths I've come across in the DB have been 4 bytes. The only 2 byte (i.e. shorts) I've across are Nation IDs (although this may have changed), player attributes, TAs (but PAs are/were 4 bytes...) and few other things - but I've only looked at players, drafts and arenas.
cavaelen wrote:Figuring that the bit i dont have completely worked out (the last 9 bytes of each record) is something to do with countries/regions and their probabilities and possibly first/last name but just guessing at this stage till i do some more work.
There was a public thread on the OOTP Forums asking for people to research names. The data they wanted us to research were names and name counts (the higher the name count, the higher the probability of it being chosen). The name count is based on national statistics and not how many players/staff in game have the name - I assume this is the same case with the names table. I did the research for the British & Irish names and so might be able to help you narrow some of the data down. Try finding the first name "Naoise" - according to the research I gave OOTP, the name count was 3 and the Nation ID for Ireland was 5. Another example is the Scottish name "Ruaridh" with a count of 78 and the Nation ID for Scotland was 4. I don't know if these Nation IDs have changed, but maybe that'll give you a head start. I think the plan was for names to be linked to regions as well as nations (e.g. so you get French Canadian names in Quebec) - I wouldn't be surprised if there is a Region ID in there too.
cavaelen wrote:Note also that the name.dat file changes structure towards the end completely. No idea what this is about yet. If anyone has any thoughts let me know.
What's the ID number of the final name entry in the table? Looks like the bits at the end might be ID numbers with blank entries perhaps?
cavaelen wrote:Figure i will map some of the sub tables like these first before i go back to working on the players table itself.
I think Nations and Regions might be worth looking at if you're wanting to figure out the whole of the Names table.

Keep us posted on how you get along. I plan on picking things up again later this summer. Perhaps we can swap notes. :-)
cavaelen
Learning to skate
Posts: 3
Joined: Fri Oct 14, 2011 5:20 am

Re: Editing the Database / Quick_Start .dat Files

Post by cavaelen »

The name length will be 4 bytes (i.e. it's an int). All of the name lengths I've come across in the DB have been 4 bytes. The only 2 byte (i.e. shorts) I've across are Nation IDs (although this may have changed), player attributes, TAs (but PAs are/were 4 bytes...) and few other things - but I've only looked at players, drafts and arenas.
Thanks, was assuming that the name length would be an int but couldnt be sure.
There was a public thread on the OOTP Forums asking for people to research names. The data they wanted us to research were names and name counts (the higher the name count, the higher the probability of it being chosen). The name count is based on national statistics and not how many players/staff in game have the name - I assume this is the same case with the names table. I did the research for the British & Irish names and so might be able to help you narrow some of the data down. Try finding the first name "Naoise" - according to the research I gave OOTP, the name count was 3 and the Nation ID for Ireland was 5. Another example is the Scottish name "Ruaridh" with a count of 78 and the Nation ID for Scotland was 4. I don't know if these Nation IDs have changed, but maybe that'll give you a head start. I think the plan was for names to be linked to regions as well as nations (e.g. so you get French Canadian names in Quebec) - I wouldn't be surprised if there is a Region ID in there too.
Well based on these two records its NationID then NameCount. Will do a more thorough check later once ive done some of the other files.
[table][th]Name Size[/th] [th]Name[/th] [th]Name ID[/th] [th]NationID[/th] [th]NameCount[/th][th]??[/th] [th]?[/th]
[tr][td]12[/td] [td]Naoise[/td] [td]4533[/td] [td]5[/td] [td]3[/td][td]0[/td] [td]1[/td][/tr]
[tr][td]14[/td] [td]Ruaridh[/td] [td]4221[/td] [td]4[/td] [td]78[/td] [td]0[/td][td]1[/td][/tr][/table]
What's the ID number of the final name entry in the table? Looks like the bits at the end might be ID numbers with blank entries perhaps?
176682 which matches the total count from the start of the file given the records start at zero.

The next int is 8541 which if it was an id number would duplicate the id number for 'Christian'. The next int after that is 26114 which would duplicate the id number for 'Aabye'. They then count up after that. The very end of the file changes again. Will probably ignore these parts at the moment and come back later.
I think Nations and Regions might be worth looking at if you're wanting to figure out the whole of the Names table.
They were where i was planning on going. Will follow the rabbit hole.
HotPoopAuditor
Learning to skate
Posts: 1
Joined: Sat Sep 28, 2024 3:39 pm
Favourite Team: New York Islanders

Editing the Database / Quick_Start .dat Files

Post by HotPoopAuditor »

Any development with his?

Maybe there is an editor by now?
User avatar
archibalduk
TBL Admin Team
Posts: 20372
Joined: Tue Jul 06, 2004 8:44 pm
Custom Rank: Seaside + Fruit Juice Mode
Favourite Team: Guildford (EPL) / Invicta (NIHL)
Location: United Kingdom
Contact:

Editing the Database / Quick_Start .dat Files

Post by archibalduk »

There's an in-game editor but I'm not aware of an external/third party editor.
Post Reply