Archi's EHM 2007 Data Editing Blog

[archibalduk]

Wow, it's been quite some time since I last posted in my blog (nearly 9 months in fact!). This reflects the fact that I've spent more time working on improving the website (hopefully I'll have it ready this summer) and updating databases (i.e. the British leagues for both the TBL Rosters and for FHM) - not to mention the fact that my free time has been restricted due to me returning to university last September (only 2 exams and 3 weeks to go!! ). However, I haven't stopped work on hacking EHM altogether. Mostly I've been working on an EHM Saved Game Editor (I'll have a regen patch ready when I have some spare time to finish it off - it can also export various bits of data from saved games, such as schedules and results) and also, to an extent, on editing FHM. As for the EHM Updater:

I worked on the EHM Updater a few months ago, but chose only to share the new version with the TBL Rosters guys and Nino as I'd planned to make some additional improvements and then release it as version 1. However, I got sidetracked by other things and so I didn't get back to looking into it until about a week ago. When I started the Updater in March last year, my knowledge about how to organise a program internally was very basic. Since then my knowledge has improved (although still only at a basic level) and I can see that I organised the internal code for the Updater in a very poor way. As more and more functions are added to the Updater, the code gets messier and messier. So I've taken the plunge and have begun re-writing the Updater from scratch. A lot of the code can be easily ported/copied across and so it shouldn't be as bad as it sounds.

In addition to re-writing the Updater, I'm adding a GUI to it. Something that has always bothered me about the Updater is that you're limited to one function at a time (i.e. you have to reload the Updater for every function you want to use). This isn't really very helpful when you're looking to import data from lots of different files. Additionally you're limited to strictly following the file names of the updater csv files (i.e. the contract update file has to be named contract_update.csv) and this is a pain if you have several csv files of the same type. It'd be much better if you could call things whatever you want (e.g. contract_update_01.csv, contract_update_02.csv, etc - or even NHL Contracts.csv). There's also the problem of having to have a copy of the EHM Updater exe file in every database folder (rather than just having one copy of the exe in a central place). And then there's the obvious issue of the menu being limited to a list in a command prompt and you having to enter the corresponding number.

I've started using something known as Qt to develop a GUI for the Updater. It's very simple to use and it has the benefit of being compatible with Linux and OS X. This isn't a major advantage for the EHM Updater, but it'll be neat to be able to write an FHM Updater that'll work on all three operating systems. I haven't gotten all that far with the new Updater yet, but it can load and save databases as well as update the index. It's not much right now, but that's mostly because I'm putting effort into getting the behind-the-scenes framework correct. This will make life much easier when it comes to adding new functions. A big behind-the-scenes shift is that the Updater now loads the entire database (whereas before it would just load the files it needed for a particular function). This will allow the Updater to make unlimited changes to the database without having to constantly reload it. It also has a proper load/save dialogue window so that you can keep a single copy of the Updater and use it to access any of your databases.

I don't expect to make any progress over the next few weeks as I'll of course be concentrating on my exams, but hopefully I'll have something worth uploading at some point over the summer. In the meantime, here's a screenshot of how the basic program is looking at the moment:

[archibalduk]

I've made a little bit of progress since my last post which I thought I'd mention whilst I remember:

The most immediate change which the new Updater will feature is the use of XML files instead of CSV. CSV files have been really useful over the past few years, but the format does have three disadvantages which have always been a little problematic:

1) CSVs only really support ASCII format rather than Unicode. This limits the range of characters/letters that can be used; most notably it doesn't allow the use of Eastern European letters with carons (whereas Unicode allows a much wider range of characters). Strictly speaking, it is possible to use Unicode with CSV files, but it can be difficult to get them to work in Excel and LibreOffice/OpenOffice.

2) Depending on whether you live in UK/North America or Europe, there is a different type of delimiter/separator used. Although this isn't a problem for the Updater, it does make sharing files with different people a bit of a pain.

3) CSVs don't save any formatting data. For example, if you make some text bold or a different size, it can't be saved to the file. It's just plain text.

XML format overcomes all three of these problems. It supports Unicode, it doesn't use delimiters and it allows for basic formatting (including preserving column widths). By switching over to XML, it'll make life a little easier, allowing us to create staff names with Eastern European characters, have tidier updater files, and make it easier to share updater files. An additional (hidden) benefit is that it will prevent a user importing the wrong file. The Updater will no longer limit what file name you use for files (so they don't always have to be called 'contract_update', etc) but this brings with it the risk that a user may accidentally select the wrong file and this could result in a crash (and you'll lose any unsaved DB changes). XML will allow me to put some hidden text in the file which the Updater can use to check that the file is the correct one for the function selected (e.g. it'll prevent a user accidentally selecting an attribute_update file when trying to use the contract update function).

Obviously it might be a bit of a pain switching from CSV to XML, but it won't be that bad. I'll include a conversion function in the Updater. The XML format is compatible with both Excel and LibreOffice (and I believe also OpenOffice).

A second thing I want to briefly mention is with regards to editing saved games. With the way the Updater is now internally organised, it should be possible to edit saved games using the Updater. This would allow all of the current updater functions to be used with saved games. Not all functions will be suitable for the saved game (such as attribute updating which uses a different and complex range of numbers) but it'll still be possible to update simple things like CAs, PAs, heights, weights, etc - and possibly also to create new players. There will be plenty of exporting functions too. But this will be a little way off yet - I'm not going to begin on saved game functions until I've got the original updating functions up and running.

[archibalduk]

I've made quite a lot of progress since my last post. Almost all of the main updating functions are complete now. I just need to add the new staff creation ("staff_new.csv") function and that'll be all of the spreadsheet functions complete (e.g. attribute_update, contract_update, staff_history_update, etc). The staff creation function will be a little better than the previous one in that we'll now be able to create staff with the correct Eastern European characters (i.e. the carons). I'm then going to move on to the other updating functions; the Date of First NHL Contract function, the player stats function, and the general optimisation function (this function deletes retired players, deletes player attributes for non-players (and non-player attributes for players) and re-orders the database, thereby fixing the super-regen bug). Hopefully I'll have all of this complete within the next week or two.

Once I have the original features complete, I'll be moving on to some new features, namely:

• Saved game editing (this is potentially complex, but hopefully I'll be able to edit saved games at some basic level using the Updater. Theoretically, it should be possible to use most of the existing Updater functions with the saved games)
• League error checking [DONE]
• League-by-league CA distribution stats [DONE]
• Xml template creation [DONE]
• Automatic .cfg file repair (replacing any incorrect apostrophe characters with the correct one) [DONE]
• Global date adjustments (so that all of the years in the database can be adjusted for different start dates) [DONE]
• A function to delete all staff from the database [DONE]
• Automatic strength calculation
• Draft history and/or team rights updating
• Retired numbers updating
• Nations updating
• Ability to change a non-player's job type using the contract updating spreadsheet
• Ability to modify staff's nationalities (perhaps via staff_update.csv) and also add it to the exported xml file.

The list is obviously very long and so there's no guarantee if/when these features will be added. I expect it'll just be a case of adding them here and there whenever I have a spare moment. I expect I'll release a beta version of the Updater at some point in July.

Finally, I thought I'd mention another way in which the new Updater will be an improvement over the old one. As you'll see from the screenshot below, there is just the one Import button. Whereas with the previous Updater you had to select the type of update you wanted to apply by entering the corresponding number, the new Updater uses a single button and detects the type of updater spreadsheet selected. Hopefully this will make things a little quicker and easier. The export function is also relatively similar in that you now select the chosen data from the drop-down menu (including an option to 'Export All') and then click on the Export button.

Note: You can view a larger version of the screenshot below here: http://s209.photobucket.com/user/archib ... a.png.html

[archibalduk]

Something I'm particularly interested in adding to the Updater is the ability to view stats about the loaded database or saved game. Today I've been working on two such functions. The first functions shows information about the Current Ability Ranges for players in every league. The second function flags up any problems with the number of teams in each playable league (as we all know, having the wrong number of teams in a playable league is an easy way of making EHM crash). The screenshots below provide more detailed explanations.

[archibalduk]

I'd hoped to get the Updater complete and uploaded today, but unfortunately working on the CSD Patch has delayed things somewhat. I also ran into a bug with the Player Info and Date of First Pro Contract function which took some time to sort out. However, the Updater is still very close to being ready. All but one of the old functions have been added (I'll be adding the missing one next week - this is the history validation function). Next week I'm also going to add date of birth editing to the attribute_update spreadsheet as well as non-player attribute and DOB editing. If I have time, I'll add official (i.e. referee) creation/editing/deletion.

In addition to adding the existing features to the Updater, I've also added some new and expanded functions; such as different database erasing options, year adjustment (so that databases can be converted for different start years), basic cfg file validation (i.e. correcting any rogue apostrophes which have previously caused problems) and a free agent utility.

[archibalduk]

I expect to be able to release the new version of the Updater either this evening or tomorrow. I've been running some final tests of the different features and have been correcting some minor mistakes. Once I've corrected the last of these, it'll be ready to upload.

I have however come across one major issue; the Database Optimisation function crashes the Updater. It used to work fine and I don't recall making any subsequent changes to it (but I guess I must have!). That one may take some time to fix. I don't want this to hold up the release of the Updater because I want to spend time on finalising some logos and research for FHM. So, unless it is quick and easy to fix, the Optimisation function will be disabled for today/tomorrow's release.

I plan on releasing an updated version following the release. This will probably be within the next week or two. It'll include the ability to use CSV files instead of XML and also a relaxed text matching option which will allow you to enter names into Updater Spreadsheets without having to use any accents. It will be better not to use the relaxed option because this option will increase the admittedly small chances of two players having identical names and DOBs (but one with accents in his name and the other player without). However, this option will be really useful for importing histories because the data available online doesn't always contain the correct accents and it is such time-consuming work trying to add them.

After that, I'll move on to getting the Updater to edit saved games.

[archibalduk]

I was going to mention this yesterday (but forgot) - the new version of the Updater is now available: http://www.ehmtheblueline.com/forums/vi ... 10&t=11154

The plan for the next version, as previously mentioned is to add the following:

1) Support for importing and exporting data as CSV rather than XML. [DONE]
2) Fixing the Database Optimisation function. [DONE]
3) Adding adjustment for the Year Founded for teams when using the Year Adjustment function (currently the Updater does not alter this particular year). [DONE]
4) The ability to relax the name matching functions [DONE]
5) Fix the progress dialog window which doesn't resize to its original size
6) Check that Imperial measurements work in attribute_update (I completely forgot to test this).

Then after that, I'll get to work on saved game editing.

Here's an updated version of the task list I posted a little while back:

• Saved game editing (this is potentially complex, but hopefully I'll be able to edit saved games at some basic level using the Updater. Theoretically, it should be possible to use most of the existing Updater functions with the saved games) [DONE]
• League error checking [DONE]
• League-by-league CA distribution stats [DONE]
• Xml template creation [DONE]
• Automatic .cfg file repair (replacing any incorrect apostrophe characters with the correct one) [DONE]
• Global date adjustments (so that all of the years in the database can be adjusted for different start dates) [DONE]
• A function to delete all staff from the database [DONE]
• Automatic strength calculation [DONE]
• The option to automatically remove unused first and second names
• Draft history and/or team rights updating
• Retired numbers updating
• Nations updating
• Ability to change a non-player's job type using the contract updating spreadsheet
• Ability to modify staff's nationalities (perhaps via staff_update.csv) and also add it to the exported xml file.
• CA and/or PA randomisation - for use with the free agent utility as it'll help mix things up
• Mass editing including player reputation editing (similar to the Realism Patch MET Attribute Editor)
• Remove player rights for player/coaches when using the 'Non-Players' option of the Free Agent Utility

[archibalduk]

I've been working on adding csv support to the Updater. So far, the csv export functions are up and running. Next up is to import csv.

I've also added a timer which is displayed in the status bar when using Debug Mode. Exporting roughly 250,000 staff career history entries from the database takes roughly 4.5 seconds when exporting to csv. However, when exporting the same data to xml, it takes just over 17 seconds. It goes to show how much slower using xml is when working with large amounts of data.

[archibalduk]

Csv importing is now working and it makes a big speed/stability difference when importing very large amounts of data. When trying to import approx 250,000 staff career history entries, the Updater really struggled when using xml. Even after leaving it for 15 minutes, it hadn't even managed to load all of the file (let alone the following part of the process which is to match up the names, etc which is much more CPU intensive). However, using csv, it takes 30 seconds to import the same data.

[archibalduk]

I've started work on adding saved game editing to the EHM Updater. So far, the Updater can load saved games and process a number of the sub-files (a saved game is really just a number files joined together). The sub-files that it can process at this stage are all of the DB files (e.g. staff.dat, club.dat, etc). Eventually I'll add the ability to process a few other types of sub-files (such as the sub-file which stores all of the schedules and results). I've noticed whilst doing this that the index.dat sub-file is different to the normal index.dat. It is missing one or two entries and the end of the file appears to be garbage (it repeats the alphabet and some numbers several times). Although this will make it a little harder to modify the index.dat sub-file, it's far from impossible.

The Updater can export all of the same data for saved games as it can for databases. The only exception is that the staff career history, player info/stats and staff awards data is not held in the same place as it is in a database. I really want to figure out where the player career history is located so that I can export this.

Here are a couple of screenshots showing how it is shaping up. The saved game in the screenshots is a 13 year sim I ran with the 1974 DB v2.0 using a 1974 start date. I expect that the Saved Game Index screen will probably only be of use to those wanting to edit saved games via a hex editor because this screen shows little other than the size of each sub-file and it's located with the .sav file (the location is shown both in decimal and hexadecimal format):

[archibalduk]

It has been 7 months since my last post, so I thought it was time I posted an update!

At the beginning of the month I released v2.0.4 of the Updater. This was the first public release for quite some time (in fact, about 6 months). It included a range of bug fixes, rights editing and various small things to make updating a little easier. My plan for v2.0.5 was originally for draft history editing, but I have recently changed my plans somewhat.

Over the summer I collated player career history for players in all playable leagues as well as some non-playable leagues. The Auto Correction function added in v2.0.4 of the Updater was designed to help make this task easier from now on. This is because it will convert the spellings of teams/leagues from online sources into the way they're spelled in EHM. The function works perfectly but the challenge now is to populate the pre-defined lists of team and league names. The Updater can only correct the spellings if I have pre-defined them in the list. So I have been going through the histories and adding the different spellings to the list. I have only done a very small portion of this work so far, but I've come across a further challenge; there are a lot of active junior/minor teams and leagues missing from the game DB. I have been slowly compiling a list (and have only done a fraction of the list so far), but it is clear there are too many to manually enter into the DB using the Pre-Game Editor. And this is where the Updater comes in - i.e. we need to be able to add and rename teams and leagues using the Updater. Hence this is now my priority for v2.0.5.

Updating teams and leagues is a bit more involved than the other updating functions I've added thus far. Whereas it makes sense for adding and editing staff to be split into two different spreadsheets (i.e. staff_new and attribute_update/nonplayer_update), it doesn't really make much sense to do this for teams or leagues. Thus I want to be able to both add and edit teams from the same spreadsheet (club_update) and to be able to do the same for leagues on a single sheet (comp_update). However, before I get to this stage I need to get colour matching working in the Updater and I need to get the Updater to process the DB language files (eng.lng, etc). At present, the Updater doesn't do anything with colours or the language files. Thus there's a bit of groundwork to complete before I can dive into editing teams/leagues.

I have already made a bit of start, with some basic colour support in the Updater. The Updater can now export a list of colours to the usual xml/csv spreadsheet. However, this isn't much use because you can't see the actual shades of colour in the spreadsheet (although I may look into showing the shades in the xml file). I thought something that might be useful would be a colour chart like those colour sheets you can get for paint. So the Updater can now produce an html file showing all of the different shades in the DB. The html file can be opened using any web browser. Here's how it looks right now:



Once I have colour matching and the language files sorted out, I'll next move onto arena adding/editing. The reason for this is because arena editing is simpler than team/league editing. So arena editing will be a good starting point to get this adding/editing on a single spreadsheet function up and running. Once arena editing is done, it'll be reasonably straightforward to get the team/league editing done.

==================================================

On a completely unrelated topic, I recently tried editing the schedule in a saved game using a non-playable league. I still haven't figured out the full structure of the schedule files, but I know enough to successfully edit the schedule for playable leagues. I was interested to see what would happen if I added a schedule for a non-playable league. The answer it seems is that the game crashes. Still, it was worth a try!

[archibalduk]

I'm hoping to get v2.0.5 of the Updater released tomorrow. If not tomorrow then it'll be certainly be within the next few days (it all depends on how much free time I have tomorrow).

The update features a number of bug fixes (one major bug with the DB Optimisation and a number of very minor bugs). It also includes the ability to export a range of new data, such as team colours, currencies and officials. I have also tidied up some of the existing export spreadsheets, such as team competitions, teams and arenas.

Whilst improving the team export spreadsheet, I spotted some new fields I hadn't noticed before. Possibly the most interesting one is that each team has a third set of team colours; presumably these are alternate jersey colours. There are others too: Euro Flag (not sure what this is - I think it's a Yes/No field), Euro Seeding (again, not sure but I think it might also be a Yes/No), Has Linked Club (Yes/No) and Is a PLC (i.e. a company quoted on the stock exchange - Yes/No). There is also a "Scope" setting for the Club Competitions. I have no idea what effect any of these will have or indeed whether these are just left-overs from Championship Manager. It'll be interesting to experiment with these settings once I've got team updating up and running in the Updater.

On a final note, I have these past couple of days been working on a Currency Mini Editor (see the screenshot below). This will be accessible via a new Edit menu in the Updater and it should work with both saved games and databases. The currency data in EHM is very out of date now (at best, it's from January 2007, possibly earlier). I've come up with a way of automatically adjusting the currency exchange rates according to inflation and have the inflation "saved" in the DB (i.e. so inflation can be added/removed at a later date). So this means we can either simply update the exchange rates or we can in addition implement inflation (e.g. the British Pound has risen by 17% since 2007). Once I have finished this feature and updated the readme, I'll be able to release the Updater.

[archibalduk]

I think it's about time I posted another update on my work with the Updater!

A priority for me over the past 12 months has been to get the player career histories updated in the the game. It's easy to get the data in a spreadsheet (albeit very time-consuming), but it's hard to import the data into the DB because there are many junior and lesser-known teams and leagues missing from the DB. There is a further challenge in that online sources often abbreviate or spell team and league names differently to how they are spelled in the DB. All of this must be corrected in order to get the data imported into the DB. There are two things that the Updater can do in order to make this task a little easier:

1) Team/league name auto-correction. This function is already in the Updater and I have been working on expanding the number of team and league names that it'll correct.

2) Adding the missing teams and leagues to the DB via Updater spreadsheets (in a similar manner as the staff_new spreadsheet). Unfortunately I have realised that the way the Updater deals with names/text in the DB is less than optimal.

Due to #2 above, I need to do some serious re-writing of the Updater's code so that text is dealt with in a better way. As it happens, I think a lot of the Updater can be improved.

Over the past 2 months I have been writing a basic Updater for Championship Manager 2001/02 (which has a very similar DB to EHM). This has provided me with an opportunity to write an improved Updater from scratch, including with better text handling. The code files for the CM 01/02 Updater can be shared with the new version of the EHM Updater. So about 80% of the work done on the CM 01/02 Updater will form part of the EHM Updater (the code files are shared which means when I update the code, it benefits both Updaters). The new code has several important benefits, including:

1) Adding new updating spreadsheet functions is now a lot quicker and easier than before. Writing the official/referee attributes update spreadsheet function for the CM Updater took under an hour. The same work with the old EHM Updater code would have taken several hours.

2) Text matching is now significantly faster. As a result, the CM Updater can import a spreadsheet updating all of the attributes for all 82,000 players in under 2 seconds. With the EHM Updater it would take much longer (possibly over 10 minutes).

3) Data/spreadsheet exporting is also faster. I haven't tested the exact speed, but it is definitely quicker.

4) Data can now also be exported as html files and in TBL table forum code (so that it can be pasted directly into forum posts) - this isn't so useful for exporting large volumes of data, but I plan on adding smaller tables such as team reports and the sorts of tables Nino often posts when analysing trends in the DB.

I have also improved the user interface, added a settings screen and a bar chart and table which shows the spread of attributes for a chosen league and attribute. I have posted some screenshots of the CM Updater below (to give an idea of how the new EHM Updater will look). It has been a really useful experience working with the CM community because they have come up with some suggestions that have never been suggested in our community here (such as the attribute chart and the summary of which teams are attached to which stadia/arenas).