Archi's EHM 2007 Data Editing Blog

[archibalduk]

I have added to the first post a list of the various tools I have written, those I'm working on and those I'm planning to work on: http://www.ehmtheblueline.com/forums/vi ... 85#p130785

I'm starting to lose track of things and so it'll be useful to have it all listed in one post.

[archibalduk]

This evening I've been taking another look at the Date of First Pro Contract field in the staff.dat file. As I mentioned in a recent blog post, I managed to fixed the "overwritten players bug" (where a player is overwritten using the Editor with new player details but they can be signed by NHL teams without first clearing the NHL Draft). The solution it seems is to reset the date.

I have written a very simple tool which resets all of the Dates of First Pro Contracts. Whilst testing I noticed something interesting: If I set the Date of First Pro Contract for every staff/player in the database as 01-Feb-1900, it allows any player to be signed without first going through the NHL Entry Draft (i.e. rather than fixing the bug, it makes every player in the DB become affected by the bug). The reason I'd gone with 01-Feb-1900 is because that is the date used by the Pre Game Editor if you want EHM to generate a random date.

I noticed very recently in the EHM 2007 database structure released by SI that there is something called "Invalid Date" which equals the 31st day of 1900 (in other words 31-Jan-1900). So I set all of the Dates of First Pro Contract to 31-Jan-1900 and sure enough it fixed the overwritten players bug. Perhaps we are supposed to be using 31/01/1900 as the blank/random date in EHM rather than 01/02/1900. This is certainly the way I'll be doing things with my tools from now on.

Anyway, I've had a bit of a test with the Manimal v4.1b DB with all Dates of First Pro Contracts set as 31-Jan-1900. It seems this fixes the bug and, as far as I can tell, there aren't any negative effects. I can, for example, sign NHL UFAs (such as Sergei Samsonov) like normal (i.e. there isn't suddenly a requirement that they also first clear the NHL Draft!). It does need more testing, but it seems that simply clearing all of the Dates of First Pro Contract is the solution to the bug. It must also be kept in mind that all of the new players created using the Editor will have a blank Date of First Pro Contract (i.e. 31/01/1900) and we haven't, as far as I know, encountered any unusual/unexpected issues with them. So I don't expect there will be any issues.

Out of curiosity, I'm going to take a look at what the cut-off date is for the cause of the bug. The bug seems to be caused by the Date of First Pro Contract being too close to the player's date of birth. For example, The Date of First Pro Contract for Oliver Bohm was only two years after his year of birth. I wonder if the bug would still be present if the date was, for example, 16 years after his year of birth.

[archibalduk]

Further to my previous post about the Date of First Pro Contract (DoFPC), I've been taking a closer look at what effect it has on NHL Entry Draft eligibility. Note that all of the testing done below was done on the first day in game (i.e. 31 August 2011) with the NHL loaded as Enhanced and Sweden as Standard using Manimal v4.1b with updated Player Info.

If I set the DoFPC as 31-Jan-1900 then it all works fine. Undrafted players under the threshold age (i.e. 20 in N. America and 22 in Europe - thanks Manimal) cannot be signed and instead you get the error message saying that they're still draft eligible etc. Overage undrafted players such as Eric Healey (again thanks for the example Manimal) can be signed okay. NHL UFAs can be signed okay (e.g. Samsonov). I also make Nugent-Hopkins and Landeskog UFAs and could sign them okay.

So all seems well with the DoFPC set as 31-Jan-1900. It makes me wonder what this field is for and whether there is some as yet unknown issue that will crop up from every player being set like this. Having said that, all of the newly created players in the Manimal DB will have this DoFPC and I'm not aware of any issues that have cropped up with those players in the past.

I have now tried setting the DoFPC for all players as 01-Sep-2010. If we take this field at face-value, those players playing for pro teams should have their DoFPC set as the date on which they first signed. Take for example the Swedish Elitserien - any player playing in this league should surely have a DoFPC. Unfortunately, by setting the DoFPC to this date, it allows draft-eligible players to be signed without first clearing the NHL draft (i.e. we get this bug again). However, this doesn't apply to every single draft-eligible player. For example, Fredrik Svantesson (who wasn't the subject of the overwritten player bug before) can now be signed even though he is draft-eligible. On the other hand, Maksim Moller (who is also Swedish and born on the same date and year as Svantesson) is draft-eligible. The only difference I can see between the two players is that Svantesson is a UFA whereas Moller is on Stromsund's roster. This doesn't explain the difference because Robin Winstedt is on Rogle J20's roster and he can be signed without first clearing the draft.

Only 1,548 players are listed as draft-eligible on the NHL 2012 Draft preview screen. Compare this with when the DoFPC is set as 31-Jan-1900: 5,263 draft-eligible players.

As things stand, it seems the best solution is to set the DoFPC as 31-Jan-1900. Setting it to any other date seems to cause issues. Maybe the DoFPC is only supposed to be set for a narrow selection of players and perhaps what teams are regarded as 'pro' is also quite narrow...

[archibalduk]

Progress on the EHM Updater is going extremely well. In addition to contract and attribute updating, I have added the ability to import player career history. Although it had taken some time to re-write the Updater, the re-write is now really paying dividends. With the new framework I've set up, it has only take two or three evenings to write the career history updating function - under the old code (i.e. without the framework) it would have taken at the very least a week. Thus I hope to be able to continue adding updating functions at a good speed.

With regards to the career history updating, the basic idea is that you list the new history in a spreadsheet and then the Updater will add this to the existing history in the database. As with the contract and attribute updating spreadsheets (I posted screenshots of these previously here), you simply enter player, team and league names in plain English - there's no need to worry about ID numbers, etc. This should make the process of updating the career histories a lot simpler than with our previous history updating project. It will also make it much easier to manage. Additionally, unlike the previous History Editor, the Updater supports international career histories.

In order to help make updating the histories as easy as possible, the Updater can generate a spreadsheet listing the latest season each player has history for (in addition to various other information). Below is a screenshot of the spreadsheet generated from the Manimal v4.1b Rosters which shows there are a couple of players without career histories up to 2010/11. This information is shown in the second-to-right column. In the right-most column, the total number of career history entries for each player is listed (note that the playoffs and regular season are each counted as individual entries). For example, you can see that Mark Stone does not have any staff history and that Louie Caporusso does not have any career history entries beyond 2004/05.



As you may have spotted in the screenshot, Louis Domingue has eight entries but you'll see that a zero is listed as his latest season. This is because it seems when his data was added to the database, the year was set incorrectly. Presumably this was because the years were inadvertently entered as zero when the research was added to the database. However, such an issue won't arise with the Updater because it will not allow invalid dates to be imported (i.e. any seasons earlier than 1901). It also won't allow a season to be entered with zero games played.

The error-checking won't stop with the updating. I plan on shortly adding some additional checking to the Updater which will include checking for invalid career history data already in the database (such entries without a valid year or games played) and deleting histories for non-playing and retired staff. This will help ensure that the data is valid and that the database does not become bloated full of useless data (this will keep the database file size and game loading speeds down).

Another feature of note is that data can be exported from one database and then used as an update file for another. This applies to all of the areas that can be updated. For example, you can export the player attributes from one database and apply it to another separate database.

If all goes well, I hope to have a beta version available tomorrow. This is however subject to the extensive testing I plan on carrying out this weekend. Fingers crossed!

[archibalduk]

Having taken a look at adding career history entries for national teams, it would seem this is not really possible. Although you can add career history entries, they appear in game under the wrong section. Ordinarily, national team career history entries only appear if you click on the Scope drop down menu -> International when viewing a player's career history in EHM. However if I add any national team career histories using my Updater, they appear under the Regular Season section rather than the International section. It doesn't seem to make a difference if I mark the entry as playoffs or regular season - they still don't appear under the International section.

In addition, there do not seem to be any national team career histories included within the original rosters (which suggests SI didn't implement the ability to add national team career history to the database).

Thus it would seem that adding national team histories is not feasible.

UPDATE: It seems the Pre Game Editor does not support nation team career histories either. If I add national team career history to the database in the Updater and then try and view the career history for that player in the Editor, I get an error and some of the histories appear invisible (i.e. the boxes are blank). The Editor doesn't allow the adding of national team career history and so I guess it can't process databases with this sort of history because it cannot cross-reference the names.

Seeing as this all seems to be a dead end, I have disabled national team career histories in the Updater.

[archibalduk]

Just a quick message to say that I have added a short update to my previous post. It seems the Pre Game Editor doesn't support national team career histories and so I've disabled the ability to add this sort of history to the database.

I've been working hard today to finalise and test the Updater in order to get a beta released today. Unfortunately I encountered a bug which has taken some work to track down. It is all fixed now, but I'm not sure if I'll be able to finish testing and write the readme file before the end of today. It might not be until tomorrow that it gets released.

[archibalduk]

Following testing, the Updater is working very well and it imports data very quickly. It only slows down if, for example, you want to import 250,000 career history records - in which case it takes roughly a minute.

I think however the contract updating may need some final testing. Although it works correctly, I encountered a database error when I tried to import all of the contract data from Manimal v4.1 into the original SI v3.0.4 rosters. I expect the cause of the error may well relate to a player who was in the original database but isn't in Manimal v4.1 and is therefore not updated (perhaps a young player in a junior team has been left on the junior team and, because I used a 2011 start date, he is now too old for the rosters). Or perhaps it relates to a defunct team. It's hard to pin down because I'm dealing with 45,000 staff contract updates. However, if I narrow down the number of players updated, it works fine. Thus I'm guessing the cause of the error is something obscure which in reality the Updater may not be used for (I don't envisage in reality we'd need to dump all 45,000 contract data entries from one database to another). In any event, I've made some addition changes/improvements to the Updater with regards to retired/invalid players (i.e. it won't allow them to be moved on to a roster) which may well resolve the issue - I haven't had a chance to test this again yet.

I'd hoped to release the latest beta today. However, I got stuck in a huge traffic jam for nearly two hours and so I haven't had a chance to finish the readme this evening! Fingers crossed I find the time to finish it off tomorrow...

[archibalduk]

Here is the download link for the latest beta version of the EHM Updater (v0.3.3): https://skydrive.live.com/redir.aspx?ci ... B049FF!122

The zip file includes sample update files and a readme file - so I won't explain too much here. I think the attribute and staff history updating is now perfected (although I will add one or two additional ease of use changes to the staff history update). I think the contract update function is also bug-free but I plan on carrying out some additional testing and experimentation. If anybody has any feedback/questions/etc please feel free to send me a PM (or the Manimal researchers can post in the Research Forum).

In the mean time, I'm going to continue working on the Updater...

[archibalduk]

This evening I've been working on the player career history validation function. The aim of this function is to check and remove invalid data as well as ensuring that the data is in the correct order. Invalid data is regarded as any of the following:

• The history relates to a non-player (i.e. they're neither a player nor a player/coach)
• The data is corrupt (i.e. it doesn't link to a valid staff, team or competition)
• The season is invalid (i.e. the season is earlier than 1901)
• The number of games played is less than 1 (because otherwise it won't appear in game)

Having applied this to the Manimal v4.1b rosters, the Updater has identified 40,164 invalid entries out of the 239,718 entries (17%). Roughly 37,000 of these relate to non-players whereas the remaining relate to invalid data (all but one of those entries are invalid because of an invalid year or zero games played). By removing this data, it has reduced the staff_history.dat file from 8.6mb to 7.2mb. Hopefully by using this function for each update, we can keep the size of the database down (i.e. so we don't fill it full of useless data, thereby keeping game loading times down). Obviously there is a lot of post 2005/06 history that still needs to be added to the database, but at least the size will be lower than it otherwise would have been.

The invalid data is saved to a CSV file so that the deleted entries can be reviewed.

The final part of the function which I need to now work on is to check for duplicate entries and remove them. I'll count a duplicate as an entry that has the same player, club, competition, year, playoff/regular season setting and games played. Theoretically there could be duplicate entries with different numbers of games played, but there is no way for the Updater to determine which one is the correct one and so it won't be able to check for these.

[archibalduk]

I've spotted a bug with regards to the staff history updater. It will only affect you if a staff_history_errors file is created. I've fixed this now and so it won't be a problem for the next release.

[archibalduk]

Here is version 0.3.5 - https://skydrive.live.com/redir?resid=B ... B049FF!135

Here is a rough changelist (I can't remember every minor thing I changed, but these are certainly the headline changes):

New Features:

• Competition history updating (i.e. club and international leagues and tournaments);
• Player award history updating;
• Player career history validation (removes duplicate entries, entries for non-players, corrupt entries - thereby helping to keep the size of the database down and resolving the difficult task of ensuring that no duplicate data is in the game);
• Detection of possible incorrect player career history entries - i.e. those that may not have the Playoffs setting marked as 'Yes' or 'No' correctly;
• Redundant player attribute removal / validation. This removes player attributes for players who have since been changed to non-players in game. This means that their Profile in game will no-longer show player attributes because they are of no relevance or use (here is an example before the fix and after - see how Maltby no longer has a Playing Profile). I understand this all fixes an issue with the database that results in "super regens". Alessandro and Manimal have both mentioned to me that that super regen issue is the number one issue with roster updating. Hopefully this fix will eradicate this issue.

Other Changes:

• Very significantly updated code behind virtually every function. This won't be noticeable to the end-user but it makes the whole coding process much quicker, easier, more efficient and less error-prone);
• Support for imperial height and weight measurements in the attribute updating sheet (details on how to do this are on page 7 of the readme);
• Various accented letters are no longer case sensitive;
• Exported spreadsheets now have text with capital letters (i.e. it is no longer purely lowercase);
• Team and competition columns in the staff_history_update.csv can be left blank (or you can enter _none) if you want the team and/or competition to appear "unknown" in game;
• Fixed a major (but simple) bug with the player history updating which would result in incorrect ID numbers (thereby causing the Pre Game Editor a great deal of confusion);
• Updated readme file.

Up next will be player and non-player creation. Progress might slow down slightly as I'll also be working with Calv to research the British leagues for Manimal v5.x.

Now that we have the competition history and player awards history updating functions up and running, I think this is an area that would benefit greatly from some sort of collaboration between members of the community. Both updates can be shared with any database and so if a number of volunteers were interested, they could research some awards/competitions, put all of their research into one spreadsheet and then this could be applied to any and every database (note that the team names and league names would have to be present in the database however - but it doesn't matter whether the players are present or not, so long as the names are in the database's name set). This point equally applies to every function of the Updater - any part of any update can be applied to other databases.

Development of the player awards and competition history functions has actually been the toughest work so far because of the interesting nuances of the way the game deals with the data and of the data that is present in the database. Here are some notes:


Competition History
Within the default database (and the Manimal Rosters), there is competition history for many non-playable competitions. Although this data does not appear on any of the Competition Information screens, much of it does appear on the Competition screens for each team and nation. On the Team Roster screen, click on History -> Competitions and you will see a list of the competitions won. Therefore it is worth updating the competition histories for many non-playable competitions too. However, this seems to work with some non-playable leagues and not others:

E.g. it works with competitions such as the Alberta Junior Hockey League (Rogers Wireless Cup), Austrian Nationalliga, NAHL, and Belarus Hockey League (mysteriously the 2004/05 data doesn't appear in game but the 2005/06 data does ) but not the Australian League, Romanian League or Swiss NLA. This is strange because there is data for the Australian League, Romanian League or Swiss NLA, etc in the default database but it just doesn't appear in game. Maybe this had been added/researched with a view to making it appear in a future version of the game.

In light of the above, I recommend that you test one entry of data for a non-playable competition before researching the whole history so that you’re not wasting time on data that won’t appear in game.

There is also data in the default database relating to playable leagues that doesn't appear in game. Here are two examples which I came across:

1) The host of the EIHL Playoffs final is recorded under the Hosts column of the database. However, this data is not displayed in game (the game only shows the champion and runner-up). I guess it is probably not worth bothering with the Hosts data for the EIHL (although it is always held at Nottingham and so it isn't too much effort to add this to each entry!).

2) Although the game does not display the Hosts data for the NHL, SI Games seem to have included data in the Hosts column for some of the more recent NHL competition history entries (1993/94 onwards). The Hosts data seems simply to be the name of the champions and not the location where the final game of the Playoffs was won. My guess is that you can ignore this column because it isn’t displayed in game.

There are tables on pages 10 and 11 of the readme which show what data is displayed for each playable competition.


Player Awards History
This was a real pain to get up and running!


Just one important note: No awards prior to the 1975/76 season are displayed in game. It doesn't matter what start date you use or whether you delete all of the post 1975/76 data, the pre-1975/76 data never appears in game.

Disclaimer: The following is probably not of any real use/interest to anybody who doesn't want to hex edit the database or write their own tools:

Before I explain an interesting characteristic of EHM (it's probably bug, to be honest), I'll explain the basics of how the database works in relation to staff. Every member of staff (i.e. both players and non-players) in the database is assigned an incremental Staff ID number. The first member of staff in the database has a Staff ID of 0, the second will have ID 1, the third will have ID 2, the hundredth will have ID 99... Well, you get the picture.

Both the player awards .dat file and the team records .dat file (the team records is the one where you list the player who holds the record for most Pts, PIM, etc for the team) can accommodate not only staff who exist in the database but also staff who do not. The reason for this is because many of the past record holders may have retired from the sport and so you need some way of ensuring that they can still appear in the awards and team records data. The way EHM does this is to store the following data for each record:

• Staff ID - the ID of the staff member (I will explain how we deal with staff that don't exist in the database in a moment - this is the source of the bug);
• First Name ID - the ID number of the staff member's forename (similar to the staff data, each forename in the database's name set has an incremental ID number starting from 0);
• Second Name ID - the ID number of the staff member's surname (similar to the staff data and first name data, each surname in the database's name set has an incremental ID number starting from 0).

For staff that exist in the database, it is very straightforward: You simply list the Staff ID of the staff member and then the First and Second Name IDs for his name.

With regards to non-existent staff, things ought to be straightforward too, but they're not for the player awards...

For the team records, things are indeed straightforward: You list the Staff ID as -1 and then you set the First and Second Name IDs for whatever the staff member's name is. The -1 ID number has a special meaning in the EHM database. It means that the staff does not exist in game (in other contexts it can mean an unknown team or competition name - such as when you want "Unknown Team" to appear in a player's career history - as well as other unknown/blank things). So if you add an entry for John Doe and set the Staff ID as -1 in the team records, he will appear in game when viewing the team records, but you won't be able to click on his name (because he hasn't been linked to an existing staff ID).

With regards to the player awards: When I looked through the default database, I found that the vast majority of non-existent staff in the database also had the -1 ID. This is what I was expecting to find. However, the awards for these staff did not appear in game.

Having looked deeper into the player awards data, I have found that some non-existent staff have been linked to a Staff ID other than -1. In other words, there is a John Doe who has been linked to a staff ID relating to an existing player such as Jaromir Jagr. I found that this data does appear in game. However, because the John Doe name doesn't match Jaromir Jagr, you cannot click on John Doe because his name doesn't match Jagr's (for example). So in essence you get the same effect as the -1 Staff ID in the team records data, but instead you have to use an existing Staff ID. This doesn't seem quite right and, seeing as much of the default data in the player awards .dat file does use the -1 Staff ID, I'm guessing this is a bug.

In order to get around this bug and to allow non-existent staff to appear in the player awards in game, the Updater will set all non-existent staff to have a Staff ID as 0. As far as I can tell, this doesn't cause any issues with the game (and in fact it is the only reason a minority of the non-existent staff have appeared in the game up until now).

Another thing I have noticed is that there are lots of entries that should be linked to an existing staff member but aren't. If you look at the Art Ross history in game, you'll see that you cannot click on Jagr's name to view his profile. This is because his entry hasn't been linked to the correct Staff ID. This can easily be fixed by exporting the data using the Updater, adding the correct spelling of the name (the á has been missed in the spelling of Jágr), checking that the corresponding DOB is correct and then reimporting using the Updater.

[archibalduk]

I've spotted a couple of issues relating to the player awards updating:

1) Certain circumstances result in the Pre Game Editor being unable to save the database (it comes up with an error). I don't know what the cause is yet, but only a particular set of circumstances with the staff awards data causes this problem. The rest of the time it works fine. You can test by opening and saving the database in the Editor after you have made changes to the staff awards data.

2) It seems as though the team name for the first placed staff member will be copied to the second and third placed staff. This is a simple mistake which will be simple to fix. To be honest, there isn't any point adding second and third placed staff data to the awards because EHM only shows second and third for the most recent season and then it discards it.

If you make any changes to the staff awards data, make sure you make plenty of backups (both of the database and your updater CSV files). Also, regularly test the database in the Editor by simply opening and saving.

Hopefully I will have both issues fixed for the next version.

[archibalduk]

Here are some updates:

The EHM Updater - Staff Creation
I've been taking a break from working on the Updater for the past fortnight - partly because there are other things I want to work on and partly because I need a break! However, I have resumed work on the staff creation functions. This is fairly complex and so progress is a bit slow. I'd hoped to have something uploaded this week, but I suspect it will be another week or so before it's ready.

The task of staff creation consists of three different parts:

1) Adding the staff to the database. This will work with both players and non-players. I won't add the ability to add invalid and retired staff because I don't see any value in this.
2) Updating the name set data (adding new names, updating the name counters, re-ordering the data) and the nation counts (nation.dat keeps a count of how many staff exist in the database for each nationality).
3) Re-ordering the staff data and updating all of the linked .dat files.

Currently I'm concentrating on the first two points listed above and this is what will feature in the first test version. Number three is going to be a very big task as there are so many different files in the database that are linked to staff ID numbers. It seems as though the Pre-Game Editor doesn't re-order any of the staff data (perhaps because it is a bit of a headache to sort out) but as it is something that SI stipulated in their guidance (link), it is something I will add to the Updater. My guess is that the order of the data affects loading speed (but this is just my guess).

Once I have completed the three tasks above, I will then work on a validation function which will remove all invalid and retired staff from the database. This links in with task #3 above to quite an extent. Again, this is going to be a particularly complex and time-consuming task.

With regards to the player creation CSV file, here is how it is looking at the moment (note that this is subject to change):



The spreadsheet is basically the same as the contract updating sheet but with additional columns for the player creation. I thought it made sense to do it like this because we'll inevitably want to set up some initial contract details for each new staff created. Any attributes you want to allocate to the new staff can be done in the standard way using the attribute updating function.


The EHM Updater - Version 0.3.6
On a separate note, I recently uploaded a new version of the Updater here (prior to starting work on the staff creation function): https://skydrive.live.com/redir?resid=B ... B049FF!138

Alessandro helpfully pointed out that the updating was no longer working for semicolon delimited files (i.e. for those countries/computers which use a comma as the decimal point). It seems I inadvertently broke the function whilst making some improvements to the code. This latest update fixes this issue and also the issue with the player awards whereby the wrong team is added for the second and third placed staff (see point #2 from my previous post).


Date of First Pro Contract
I now have some clarity with regards to the DoFPC. As previously suspected, this field relates solely to the date of a player's first NHL contract. It has the following effects in game:

• Ensures correct NHL Entry Draft eligibility.
• Ensures that rookies can only ask for new contracts within the rookie maximum salary and the rookie maximum contract length. Rookie contracts are always two-way contracts.
• Affects when the player is eligible for arbitration.
• Ensures the correct free agency type of players with NHL rights still valid.
• Determines regular season waiver eligibility.

It also has various other effects in the game where there is a distinction between a rookie and a veteran.

I'm going to add DoFPC updating to the Player Info Generator. The Generator will be able to generate the DoFPC based upon the player's career history. Hence it will be important that we keep player career history up to date (for the NHL at least).

[archibalduk]

Unfortunately progress has been quite slow this week - partly because I've been too busy and partly because I'm lacking motivation. Thankfully I'm starting to get my motivation back and so I really want to get the basic staff creation completed by the end of this weekend. I'm not entirely sure how realistic that is at this stage though...

Whilst working on the staff creation, I've come across some interesting-looking attributes/data that I hadn't noticed before. I'll need to take a look at many of these before I can finalise the staff creation (because with every staff member created, default settings will need to be set for each attribute relating to them). Here's what I've found:

• Preferred Tactics (non players) - We all know how this looks in the game (you'll see it on the staff profile screen), but IIRC this isn't something that can be edited via the Pre-Game Editor. The database structure published by SI doesn't go into the possible settings for this attribute and so it will take some experimentation in order to find the full range of these.
• Playing Squad (players and non-players) - I've no idea what this is for. It doesn't relate to a team per se (because the data type isn't a team pointer / team ID). It must be some sort of switch where there are multiple settings - maybe something like Main Team / Affiliate 1 / Affiliate 2... Similar to the Morale attribute, it might be something that is left blank in the database but is instead used once a game has been created.
• Squad Selected For (players and non-players) - My notes regarding Playing Squad apply equally to this.
• Club Valuation (players and non-players) - This is different to the Estimated Value. It doesn't seem to relate to money (because the range of this particular attribute is from -127 to 127). I wonder if this is used to set whether the team regards the player as a Backup, Hot Prospect, Regular, etc.
• Estimated Wage Weekly (players and non-players) - This is different to the Estimate Wage setting in the Pre-Game Editor. I wonder why there are settings for both weekly and annual wages in the game...

In addition to these unknown attributes/settings, there are also several settings relating to contracts to national teams. For some reason the Pre-Game Editor doesn't allow you to contract non-players to national teams or place players on their rosters (maybe it cannot be set in the database?). These settings are on the Contract Details tab in the Editor, but they're always disabled. It would be interested to try these out at some point. Here are the various settings (note that these are in addition to the club contract details - presumably a staff member can be contracted to both a club and a national team at the same time):

• Nation Contracted To
• Date Joined Nation
• Nation Contract Expires
• Job For Nation
• National Team Job Level

Normally I try to post some sort of interesting screenshot with my blog posts. However I don't have any new screenshots to share at the moment and so I'll post a screenshot of my hard work instead!

EDIT: Heh I've spotted a mistake in that screenshot!

[archibalduk]

The task of staff creation consists of three different parts:

1) Adding the staff to the database. This will work with both players and non-players. I won't add the ability to add invalid and retired staff because I don't see any value in this.
2) Updating the name set data (adding new names, updating the name counters, re-ordering the data) and the nation counts (nation.dat keeps a count of how many staff exist in the database for each nationality).
3) Re-ordering the staff data and updating all of the linked .dat files.

Currently I'm concentrating on the first two points listed above and this is what will feature in the first test version.

I have made very pleasing progress over the past couple of days and have now almost achieved bullet points one and two above. There is still some work to be done, but the Updater now appears to successfully create new staff in the database. It needs a lot more testing but the early signs are looking good. I have only tested that the new staff appear in the Pre-Game Editor so far (i.e. I haven't tried starting a new game in EHM with the new staff yet) but I use the Editor as the benchmark for all of my initial testing because I find that the Editor is typically more sensitive to any data that might not quite comply with the normal standards.

This weekend I will be finalising the staff creation feature by generally tidying/polishing off things as well as adding some additional columns to the new staff updater spreadsheet (i.e. I'll be allowing the ability to add some basic data such as CA/PA, height/weight, positional data, etc - thanks for the feedback Nino ). All being well, I'll have a test version ready by the end of Sunday.

In the mean time, here's a screenshot of the Editor showing the new staff member created by the Updater. Note that the unusual name is simply to test the Updater's ability to create new names in the name set.

[archibalduk]

I now have EHM up and running with my newly created staff appearing absolutely fine in game!

The name sets (i.e. first_names.dat and second_names.dat) are used for a number of other parts of the database in addition to the staff details. Thus these must also be updated:

• Officials
• Player awards
• Team records

The names for the player awards and team records are particularly important where a player who doesn't exist in game is added to an award/record. The database allows you to enter these players (such as Gordie Howe and Phil Esposito who don't exist in game IIRC) into the awards/records even though they don't exist in the database because they link to the name sets (i.e. the forename "Gordie" exists in first_names.dat and the surname "Howe" exists in second_names.dat). If I don't correctly update the above dat files or staff.dat then the names will appear incorrectly in the database whenever the Updater adds a new name to the name sets (because the ID numbers in the player awards, etc won't match up to the correct updated ID numbers in the name sets).

The Updater now updates all of the aforementioned files and so the names appear correctly in all areas of the game except for one: Mysteriously, certain records in the team records are not displaying in game correctly. At first I thought this was a bug with the Updater but it turns out that it's a bug with EHM.

The bug affects certain categories of team records where the record holder does not exist in game. For example, the most career wins for the Boston Bruins is held by Tiny Thompson. I have double-checked the original v3.0.4 rosters (the rosters included with the final EHM patch) and the Manimal 4.2 rosters. Looking at the relevant dat file for both sets of rosters (club_records.dat), the data is correct. In other words, the rosters each have Tiny Thompson listed as the record holder for the Bruins' most career wins. Yet in game, a seemingly random name appears. If you take a look at the career wins record for the Bruins in your own saved game (regardless of which roster update you're using) you'll see what I mean.

For some reason, the game appears to have swapped the ID numbers for the forenames and surnames. So it uses the ID number of the forename and links this to the surname - and it uses the ID number of the surname and links this to the forename. Well, kind of. It seems this is the case for various other examples of this bug in the database, but not quite in relation to Tiny Thompson because it seems to have plucked the forename randomly out of the air - presumably because the ID of the his surname "Thompson" is higher than any of the forename IDs and so it cannot cross it over / mix it up quite like I've mentioned.

This bug doesn't seem to affect every category of team records; just a couple such as career wins and career PIM. Other categories seem to work fine. For example, Tiny Thompson's name correctly appears for most career shutouts.

After spending a good hour or so looking at this bug and my Updater, it is clear the bug lies with EHM and not the Updater or the Manimal Rosters. The game clearly mixes up the forenames and surnames for some categories of team records. I guess the solution to fixing this bug is to switch the ID numbers stored in the dat file for the offending categories - so the forename ID is stored in the surname field and the surname ID in the forename field. This is something I could perhaps look at resolving in the future with a simple tool.

In the mean time, we can continue to edit using the Editor and Updater like normal - neither tool will make the issue worse or irrecoverable.

On a more positive note, here's a screenshot of the success of the Updater in relation to creating new players:

[archibalduk]

Time to answer some of the questions regarding some unknown fields which I posted previously here.

I found this post from Graeme Kelly on the SI Forums:

Divisions are not used for national teams, their entry into competitions is either hard coded or uses the Final Position fields.

Playing squad is used internally only, so you should just hide it in an editor. Same with the squad selected field.

National Team Job level is 0;Men; 1;U20; 2;U18. So, set the job to GM and need to set this field to determine which team it is.

[archibalduk]

Version 0.3.8 of the EHM Updater is now available here: https://skydrive.live.com/redir?resid=B ... B049FF!141

This features the player creation function. It follows the same general principles as the other updating spreadsheets, but there is a fuller explanation on pages 19-21 of the readme.pdf.

I must make clear that the staff creation function is still rather experimental. I cannot emphasise enough the importance of making backups of the database each time before you apply the function. The function works fine when I have tested it, but I just feel that because it is such a radical addition to the Updater that it would benefit from additional testing and plenty of caution.

I will find some time this week to double-check that the imperial height/weight option works okay (I expect it will do as it is the same as the option for the player attribute updating) and also that the Updater handles Head Scouts okay (it should set them as Scouts because the database shouldn't have Head Scouts set).

Next I will be looking at a function to remove all invalid and retired staff from the database. I think this will probably take approximately two weeks. I will also look at how to deal with player rights and draft data updating.

If anybody tries out the staff creation function then please do let me know how it goes (even if it is just to say that it works fine). It would be useful to know other users' experiences of the Updater.

[archibalduk]

Following on from my post above, I thought I'd post a list of things to check for when checking that all is working okay after applying the Player Creation Function. Note that you don't have to check every single player, just check a very small sample for each of the five points below.

1) Check that the players listed at the top and the bottom of the list in the Pre-Game Editor still have the correct names. A good way to check this is to check one or two players with a surname or forename beginning with 'A' who are Canadian and a couple beginning with 'Z' or a letter with umlauts in it (or alternatively any special character that does not have an acute/obtuse accent above it). Do the same for a couple of players who are from the USA. This might sound a bit odd, but the name sets in the database are grouped by nation and then ordered alphabetically. By checking a group that appears near the beginning of the database (e.g. Canadian or Austrian) and a group that appears at the end (e.g. USA or a country beginning with V, W, X, Y or Z) then you will be able to pick up if any of the names have gone out of sync.

2) Check that the officials at the top and the bottom of the list in the Pre-Game Editor still have the correct names.

3) Check that the player names in the team records screen in game still have the correct names (although keep in mind the bug mentioned earlier).

4) Check that the player awards history still show the correct names in game (e.g. the Norris, Vezina, etc).

5) Ensure that your newly created players show up with the correct names.

These five key checks are the main checks that I have been performing during my testing. As far as I can tell, none of these should arise but I post them here in case anybody else is feeling super cautious.

[archibalduk]

It looks like I've nearly completed the staff validation function. This deletes all invalid and retired staff from the database. There is just some minor checking I want to do before saying for sure that it is completed.

The validation function involves updating a large number of files:

• Staff.dat (basic staff data)
• Club.dat (roster sizes and captain details)
• Club_records.dat (team player records)
• Drafted_players.dat (draft data)
• First_names.dat (counts the number of times each name is used)
• Nation.dat (counts the total number of staff for each nation in the database)
• Nonplayer.dat (non-player attributes)
• Player.dat (player attributes)
• Player_info.dat (player stats)
• Player_rights.dat (team player rights)
• Second_names.dat (counts the number of times each name is used)
• Staff_comp_history.dat (staff award history)
• Staff_history.dat (player career history)
• Staff_info.dat (tracks the previous team which non-player played for)
• Staff_languages.dat (staff language proficiency - this doesn't actually hold any records)
• Staff_preferences.dat (the liked/disliked staff for each staff member)

The Updater will remove all data associated with each of the invalid and retired staff members. However, it leaves things such as club records and staff award data intact (i.e. their names still appear in the records even though they have been deleted).

One particular oddity of the database I've had to deal with are the first and second name counters. If you look at the first and second name data in the Pre-Game Editor, you will see that there is a Count field. This simply counts how many staff and officials have that particular first/second name in the database. I not 100% sure why, but SI has given this field a range of only -128 to 127. In other words, the field cannot be set higher than 127. I expect the Count is used by EHM when assigning names to regens (perhaps the Count affects the probability of a particular name being used). Perhaps the idea is that any names used more than 127 times would have a disproportionate weighting if the Count were to be greater than 127. For example, if the name "John" had a count of 700 and the name "Dave" was only something like 50, then there is a chance that every regen would be called John...

Anyway, the Pre-Game Editor seems to treat this field slightly differently in that it will allow a range of 0 to 255. I won't go into the technical reasons for this, but basically I think it is treating the field as an unsigned char rather than a signed char. However, seeing as EHM uses a range of -128 to 127, this is what I have used for the Updater.

It seems many of the Counts in the database are incorrect (for example, the Manimal v4.2b DB has a Count of 191 for Mikes from Kazakhstan!!!). I don't know whether this is an issue with the original EHM database or the Pre-Game Editor. In any event, the Updater corrects these counts within the limits laid down by SI's database structure. This may have ramifications for the regens in EHM (assuming I have understood/guessed correctly the role of this field) in that the names generated might be more representative of the spread of names in the database.

[archibalduk]

Version 0.4.0 can now be downloaded here: https://skydrive.live.com/redir?resid=B ... B049FF!146

The big new feature in this update is the Staff Validation function. This deletes all Invalid and Retired staff (along with all related data) and ensures that there is a Player Info entry for every player. It also re-orders all of the affected database files correctly and updates/corrects the name set counters.

There are also some general improvements/fixes included:

• Staff creation function: A Player Info entry is now added for every new player created.
• Staff creation function: The Updater will no longer create a staff member if an identical staff member already exists in the database, thereby avoiding inadvertent duplicates. An staff member is regarded as duplicate if they have the same name, primary nationality and date of birth.
• The Updater now accepts text qualifiers used by OpenOffice (it encloses all text in quotation marks - this was previously preventing the csv from being correctly processed).
• Blanked out cells with text qualifiers in OpenOffice no longer cause a problem.

I now intend to start work on an improved Player Info generator function within the Updater which will incorporate a Date of First Pro Contract generator function. I will also work on polishing the Updater a bit (the functions need to be re-ordered in order to make more logical sense) in addition to making some minor improvements. I plan to release that version as version 1. Then I'll be taking a break from coding to work on some other things, including an improved site header and some roster research.

Once I've finished these final bits for the Updater, I plan to re-commence work on additional features probably around about September or October. I want to particularly look at draft data updating and non-player attribute updating.

[archibalduk]

Alessandro has kindly reported a minor bug in relation to the EHM Updater:

when you put a player as dual national, and add a nation into the declared field, the editor will put him declared to the first nation listed no matter what.

This is something I will fix when I next work on the Updater. I expect this will be in maybe October as I'm so busy with other things right now.

Update: We found that there isn't a bug; it works fine.

[archibalduk]

Just a quick post to say that I've found another minor bug with the Updater which I'll need to fix. The bug doesn't affect any real functions, but it may be a bit confusing:

If an attribute_errors.csv is generated, all of the attributes that were actually applied okay are listed in the file as UNKNOWN ERROR (in addition to the attributes that weren't successfully applied). If you encounter any UNKNOWN ERROR listings in this file, you can safely ignore these as they will have been applied successfully anyway. Note that this problem only arises if something in your attribute_update.csv file does in fact contain an error. Basically, the Updater is simply listing all of your attribute updates in the error file rather than just showing those that were erroneous.

[archibalduk]

Three very quick updates in the relation to the EHM Updater:

1) It turns out that the Declared Nationality works fine. There isn't a bug.

2) We have found an unusual issue with the most recent development version of the Manimal v5 rosters. I don't know whether it is an issue with the Updater (I can't say for sure, but I don't think it is). You can find out whether your own database has this issue by loading it in the Pre Game Editor, and then click on Tools -> Check Database for Errors. If everything is okay, nothing will appear. However you might encounter occurrences of the following error message: "xxx is contracted to an invalid national team (10). Reset to none.". If you get this error, exit the Editor (don't save the database) and download this patch: https://skydrive.live.com/redir?resid=B ... B049FF!150

Put the above patch in the same folder as your database and run it once. This will fix the problem.

3) I have also come across a minor mistake in relation to the staff creation function of the Updater. I have fixed this and so any staff created from v0.4.1 onwards will be fine. In the meantime, if you apply the aforementioned patch, it will resolve this very minor issue (apply the patch after you have created new staff - you can apply the patch as many times as you like). It doesn't matter if you don't want to use the patch; the issue is just a consistency issue and won't have any negative impact on the database, Editor or game. It is perfectly safe to continue creating staff using the current version of the Updater (v0.4.0).

[archibalduk]

I haven't really been doing much data editing recently as my top priority is helping with the Manimal Roster Update and the OOTP game. However, I have been working on two relatively small things:

Blank Database
A particular problem Nino had to overcome with his 1974 database was to remove all of the modern players. In order to make this easier in the future, I have added a function to the EHM Updater which removes all players from the database. I still need to do some work and make some minor bug fixes to the Updater and so I won't be uploading an updated version of the Updater quite yet. However, to make life easier, I have uploaded a blank database here: http://www.ehmtheblueline.com/forums/vi ... 115&t=9924

When testing, I found that the Pre Game Editor was unable to open the database if no players/staff existed (the Editor would simply crash). To get around this problem, the Updater leaves ten players/staff in the database.

During testing, it was interesting to note that the only notable players in the game (i.e. those with a high enough reputation to appear on the Player/Staff Search Screen) were always almost entirely British:




EHM SuperPatch
This is something I have been working on with Alessandro and Hobbit13 (he has done some incredible work with modifying league rules and structures). It is still very much in the beta testing phase, but if you would like to try the league rules patch (it also expands the Austrian league to 12 teams), you can download it here: http://www.ehmtheblueline.com/forums/vi ... 31#p145031