A call to make a modding Toolkit for the CB series.

Hop to:

Topic Subject: A call to make a modding Toolkit for the CB series.

« Previous Page 1 2 Next Page »

user3
Pleb

posted 10-16-17 03:45 ET (US)

City Builder executables points to a "phrase" in the ENG file(for english installs). I'm assuming other language installs use different file types(?SPA,FRE,GER...).

The ENG file structure is fairly simple. Using a hex editor; my wife prefered UltraEdit and contributed funds, but I am a cheapskate and use HxD(stable since 2009 and simple enough for me), you can view the simple structure.

At the top of the file(it's header), is the offset value for each "phrase" delimited by the hex value of '00', on either side, as opposed to a 'space' value of 20 that 'Notepad' displays.

Since 2003(Empress mod), people have been editing this file, for Emperor, using a hex editor. I am sure others have done simular ENG editing for the earlier games.
What is needed is a programmed text editor that has some 'Notepad' fuctionality, find and replace but with restrictions so as not replace every instance without user input or nag popup dialogs. The displayed phrases should be stacked(verticle list), rather then word-wrapped. The 'save' function obviously needs to conform to the systems expectation.

Why bother, you ask. With an editor in a modding toolkit, modders can change the "phrases" length, something that cant be done with a hex editor. Modders could even shorten the exteranious phrases never displayed in normal game play. I dont much care if it messes up the 'programmers dash' as they never released it to the public and I doubt Activision maintains a copy that could be released.

Notepad++ looks like a good candidate as it is open sourced. BEOS & Linux enthusists inform me 'Emacs' could do the job with its extensive library, but none have made a custom editor, so it might be over-complicated for our task.

With this one starter tool, we could continue onto Help file corrections, correcting the message concoctions, etc. Even get enough interest that people might start editing the Monuments and Temples, again. Dream a little think how your skillset can help the HG Modding Toolkit.

So, am I beating a horse skeleton here or what?

Pardon any spelling mistakes, new windows install still in progress.

Author

Replies:

Pelagius_II
Pleb

posted 10-27-17 05:25 ET (US) 26 / 52

EDIT: Actually, Pecunia- it looks like a version of the utility is on Github already:
https://github.com/lclarkmichalek/sgreader

There's also a version included with the OpenC3/CaesarIA repos. I'm not as familiar with the file format as others, or C++ in general, but I'd like to help in some fashion if I can.

.

[This message has been edited by Pelagius_II (edited 10-27-2017 @ 10:29 AM).]

Pecunia
CBH Seraph

posted 10-27-17 12:05 ET (US) 27 / 52

@user3: found your weird dialog. It looks like I'm now leaving out too much info in the .xml file. I'll fix it tonight and release 0.3.2.

@Pelagius: over the years I sent the source code of sgreader to several people by request, and it looks like some copies ended up on Github. I have just added it to my citybuilding-tools Github repo. I'd like to make an sgwriter for it, but I'm at a loss how to make it such that it: a) works technically, and b) is easy to work with for users who want to replace their graphics.
The challenge in (a) is that there's a lot more information stored in the .sg2/3 files besides image data: sizes, animation offsets, whether the animation can also run backwards, etc.
The challenge in (b) is that we're talking about 10,000+ images. You don't want to force the user to replace those one by one...

Angel Pecunia | HG Downloads programmer and techie
Caesar IV Heaven | Children of the Nile Heaven | Citadel of Poseidon (Zeus/Poseidon) | The Great Wall (Emperor)

Pelagius_II
Pleb

posted 10-27-17 12:32 ET (US) 28 / 52

@Pecunia: How did you decipher the file format in the first place? Was there some kind of internal documentation you managed to get your hands on, or did you have to figure it out by trial/error and hex-inspection?

I can't really suggest a simple solution to the technical challenges. Maybe you could use the reader to unpack information into .xml format first, so it's more human-legible at least?

Pecunia
CBH Seraph

posted 10-27-17 12:51 ET (US) 29 / 52

Trial and error, and hex inspection indeed. Comparing entries in the sg files, editing a number and seeing in the game what the effect was. Rinse and repeat. Any documentation that may have existed for these games is gone: GOG doesn't even have the source code for the games.
Could you drop me a mail? I'd like to discuss the SG format (and your other thread).

@user3: version 0.3.2 is available

Angel Pecunia | HG Downloads programmer and techie
Caesar IV Heaven | Children of the Nile Heaven | Citadel of Poseidon (Zeus/Poseidon) | The Great Wall (Emperor)

[This message has been edited by Pecunia (edited 10-27-2017 @ 01:23 PM).]

user3
Pleb

posted 10-27-17 13:09 ET (US) 30 / 52

Umm, If a primitive framework would be helpful, to kickstart the brainstorming, you may look here. Link

Thanks Pecunia, I will give it another go with my testing campaign.
-user3

[This message has been edited by user3 (edited 10-27-2017 @ 01:12 PM).]

Pelagius_II
Pleb

posted 10-27-17 13:13 ET (US) 31 / 52

@Pecunia: Sure, I'm happy to chat. I'm amazed that even the source code has disappeared, though.

@user3: I appreciate the tip, but I suspect I'd be happier 'rolling my own' as it were.

alincarpetman
Pleb

posted 10-27-17 19:18 ET (US) 32 / 52

I just found this thread and it's nice to see discussion about modding the CB games.
I ran into problems myself when attempting to find the possibilities and limits with modding these games, you can read my thread at Zen Hax here.

The program Quick BMS is used to extract archived game files and there was limited success with the 4 Impressions CB games: Caesar 3, Pharaoh, Zeus & Emperor.
Some players had emailed me about extracting graphics for modding within the CB games as well as use in other games such as Age of Empires.

Ideally we might need a User Patch/Modding Suite for the CB games and that is a big task.
If you play RTS games such as the Age of Empires series, you can read about those user patches to get an idea of the project and features: AOE1, AOE2, AOE3

It may be helpful to compare and contrast the game architecture and modding potential and limits (Modding Can Dos and Cannot Dos) between the RTS and CB games to help some non-technical/non-programmer players and modders to understand what's going on.

In recent years there is more interest in reviving classic games such at the RTS games but the city building genre has always been a smaller niche and the online presence has shrunk a little (Anno forums went offline, Tilted Mill is a ghost, etc) so hopefully this thread will generate more discussion and I'll try to help where I can.

It looked like OpenCaesar/Caesaria was going to expand the possibility but as I read it has been put on hold for now.
I have imagined in my mind that if the 4 CB games had the same modding options as the Age of Empires series then by now we would have seen many mods and even new civs modded in the games.
Imagine expanded content for the existing games, changed graphics, new features, etc as well as modder-created civs like India or Mesopotamia mod replacing the Egyptians in Pharaoh.

[This message has been edited by alincarpetman (edited 10-27-2017 @ 07:20 PM).]

user3
Pleb

posted 10-28-17 01:45 ET (US) 33 / 52

@Pecunia
That fixed it, I will start editing the EmperorText/EmperorMM files with some ideas I have.

Thanks again for your time and patience on his Toolkit idea.
-user3

Pelagius_II
Pleb

posted 11-03-17 08:20 ET (US) 34 / 52

It looked like OpenCaesar/Caesaria was going to expand the possibility but as I read it has been put on hold for now.
I have imagined in my mind that if the 4 CB games had the same modding options as the Age of Empires series then by now we would have seen many mods and even new civs modded in the games.
Imagine expanded content for the existing games, changed graphics, new features, etc as well as modder-created civs like India or Mesopotamia mod replacing the Egyptians in Pharaoh.

It's funny you should mention that, because another game that's near and dear to my heart- Majesty, the Fantasy Kingdom Sim- recently came out on steam with an updated engine and some very powerful modding tools, including a full-blown internal scripting language that was used for AI-development by the original developers (though for now absent the ability to add new sprites.)

I suspect if those tools had been released 10-15 years ago there would have been a torrent of user-made content, but Cyberlore (the parent firm) is dead and the paradox forums are a ghost-town now. (We don't talk about Majesty 2.)

Ah, to think what might have been...

The Impressions fanbase actually seems to be a little more active- it was certainly much larger back in day- so it's an irony that modding capacity is rather more limited. C'est la vie.

Pelagius_II
Pleb

posted 11-07-17 07:25 ET (US) 35 / 52

Hey folks-

With some help from Pecunia (to whom I have to give some major, major kudos, unpacking this format could not have been easy), I've been able to adapt her image-reading code and have a better idea of what would be required for an SG-writer utility.

The complication, as I see it, is that every .SG file refers to one or more .555 files for storing image data, and in the case of 'external' references, multiple .SG files can refer to the same .555 file. What this entails is that any modifications to a given .555 file will require re-writing any .SG files that reference it, and you can't be certain which .SG files do so until you parse all of them. (The .555 files themselves are just a dumb stack of compressed images- they have no structural metadata themselves to say who they belong to.)

In many cases a particular chunk of image-data is only referenced by a single .SG file, but because of run-length encoding for transparency, many images with the same width/height will compress to different sizes, so you can't just substitute a new image for an old one and rely on squeezing it into the same slot. The .555 files are like a stack of books, where each 'book' is a particular image or sprite or animation-frame. If you want to add/remove/replace a book halfway down, you have to lift or lower all the books on top as well. And because the .SG files all refer to a given book by its height, that necessitates updates to their data as well.

So, that's the conundrum. I think you basically need to construct some supplementary metadata for the .555 (and possibly .SG) files, either cached as .xml or kept in memory, and use that to keep track of the reference graph. Then hopefully the user should only need to nominate a single image for replacement and any needed changes can be propagated with minimal fuss.

Anyway, that's the current situation. The code is here for anyone curious, and I'll let you know once there's progress.

alincarpetman
Pleb

posted 11-07-17 19:48 ET (US) 36 / 52

@ Pelagius

That's good news about Majesty, I haven't tried that game yet but it's better late than never for some update and hopefully over time old and new players and modders will check that out.

And thanks for the 555 file explanation, what do you think of my post #32 above about using Quick BMS to extract the files?
The specific goal was to extract the images for modding inside the existing games as well as porting those same graphics into other games such as AOE2.

I liked the metaphor about the books and 555 files, so I'm wondering how my extraction attempt fits into the bigger picture of modding these games.

[This message has been edited by alincarpetman (edited 11-07-2017 @ 07:50 PM).]

Pecunia
CBH Seraph

posted 11-08-17 00:39 ET (US) 37 / 52

Using QuickBMS for modding is not going to work. To stay with the books analogy: what it does is translate a book in an unknown language into English, by using a dictionary to translate each word. It does that for all languages it knows, until something more or less sensible comes out for one language. But that doesn't take into account that some words have multiple meanings, depending on the context.
And to write a book in a certain language, you need knowledge about grammar, proper punctuation, the meaning of phrases, and more. QuickBMS doesn't have that knowledge.

In any case, we already have the sgreader to extract files. What more do you need?

Angel Pecunia | HG Downloads programmer and techie
Caesar IV Heaven | Children of the Nile Heaven | Citadel of Poseidon (Zeus/Poseidon) | The Great Wall (Emperor)

user3
Pleb

posted 11-08-17 15:30 ET (US) 38 / 52

Quoted from Pecunia:
What more do you need?

Understood, the question was most probably directed at alincarpetman. Personally, I don't need, but rather have a desire for easy to use tools for those interested in modding.

1. sgreader does not preserve the full directory structure of the 555 nor even attempts to unarchive the sg directory structure. First step would be giving the next generation tool these capabilities. Second, the ability to re-archive edited directories for distribution to the community. Don't know, but do the game systems require files to be in an archived state?

2. The development of a plugin for Gimp to edit not only the image data but the sg data, preserving the edits in respective files for re-archival.

De-archive/Re-archive solves Pelagius_II's issues with compressed data sizing and referencing if the plugin or re-archiver does a safety scan of sg files. The algorithm is already part of sgreader. I don't even mind the more recent PNG save format, though BMP might cause less hassle with the plugin.

I'm not asking anyone to devote sparse time to such tools out of 'perceived obligation', just a willingness to work toward a common objective, while keeping the community abreast of their level of participation.

Quoted from alincarpetman:
... as well as porting those same graphics into other games such as AOE2.

Some, including me would have ethical issues with porting graphics between unrelated title owner games. C3 graphics into Cleo/Posd/Emperor might pass my ethics self-check, but others may not be so liberal.

[This message has been edited by user3 (edited 11-08-2017 @ 04:14 PM).]

Pelagius_II
Pleb

posted 11-09-17 07:56 ET (US) 39 / 52

So, good news and bad news: The good news is that I've been able to produce compressed images that match the game's original data byte-for-byte on a range of examples, and I've been able to modify the in-game forum image (meaning the associated .sg2 and .555 files) without causing a crash. The bad news is that the changes I've introduced to the sprite appear badly corrupted on-screen. (The sprite is supposed to look like this.) Not sure why.

EDIT: You can see some of the sprites I was testing with here. Basically, they'd be decompressed, displayed, then recompressed, then checked for identical byte-encoding, then decompressed again and displayed to check whether the final images were identical.

It's quite possible for the final images to be identical without having the same byte-encoding (and in at least one case- the governor's palace- it may be impossible to reproduce exactly the same byte-encoding from a flat image, since there's actually pixel-overlap between the compressed and uncompressed portions of the original image.)

EDIT EDIT: I should also note that, at least for C3, it doesn't look like any .555 files are referenced by more than one .SG file. However, multiple image records within a given .SG file can refer to the same data chunk within a .555 file, so similar complications can arise...

EDIT EDIT EDIT: Huge success! Turns out my compression-routines were being limited by the data-size for the old record. Doy. Now working correctly.

@user3: It certainly appears that the game does expect data in an archived state, which accounts for some of the hassle in editing related files.

More broadly, making any changes aside from 'reskins' of existing game content isn't something I see an obvious way to do without decompiling the core game engine. All the building-sprites seem to be referenced by the index of their entry within a particular .sg2 file (which in turn points at a chunk of data within a .555 file). Is there any other file-format with some metadata on building types, that might allow for new or changed building-lists?

[This message has been edited by Pelagius_II (edited 11-09-2017 @ 09:46 AM).]

user3
Pleb

posted 11-09-17 11:50 ET (US) 40 / 52

Quoted from Pelagius_II:
More broadly, making any changes aside from 'reskins' of existing game content isn't something I see an obvious way to do without decompiling the core game engine.

For Emperor, the building model txt and figures model txt files refer to unused buildings and walker. The 555's contain images/animations from previous games. Poseidon likewise has image artifacts and unused placeholders in the txt files.

I believe there is another option other then decompiling. Similar method as the widescreen development, Hacking the exe building resource list and activating a building for use as an improved/new building and walker behavior changes combining existing game mechanics(heroes preform many walker functions, why not an herbalist+acupuncturist doctor from a hospital. Sentry outpost(crossbow)+guards(spy capture+unrest) that roam the roadways.) Some of this stuff can be done already by modding the guard to use missile fire unfortunately the fighting animation remains as a polearm.

Hacking the exe isn't easy so for the present; I/maybe others would be happy with a reskin for alternate animations to reflect the simple txt modding options or cultural distinctions (Japan/Korean mods).

[This message has been edited by user3 (edited 11-09-2017 @ 12:23 PM).]

Pelagius_II
Pleb

posted 11-09-17 12:02 ET (US) 41 / 52

Sure, I'm happy to help with any reskin-efforts. Do you want me to package up my code as a standalone .jar for now? There's no GUI to speak of, but I could provide some written instructions and a small .XML configuration to select the files you want to modify.

I'll eventually want to rewrite my utility in C++ and distribute it as an .exe (or possibly ask Pecunia to integrate the code with her SGReader utility), but that could be a little while down the road.

I'd love to get in touch with whoever figured out that widescreen mod, by the way.

user3
Pleb

posted 11-09-17 12:15 ET (US) 42 / 52

I can wait for the final product, but maybe another would be interested and post his interest in your progress.

I dont know the specifics of the development but here is a link of Emperor's forum at WSGF. Other games you'll need to lookup. C3 and Pharaoh use different methods, but Posd/Emperor seem simular.
http://www.wsgf.org/forums/viewtopic.php?f=61&t=26165#p146275

[This message has been edited by user3 (edited 11-09-2017 @ 12:20 PM).]

Pelagius_II
Pleb

posted 11-13-17 09:20 ET (US) 43 / 52

Small update- I've more-or-less finished translating my core routines into C++, using the SDL library for visual debugging where required. It seems to be working more-or-less okay for now, notwithstanding some bugs (I can't seem to figure out exporting .bmp files with transparency, and there are rare, unpredictable crashes when reading pixels from SDL surfaces.)

I still need to update some of the libraries I'm using for better compatibility with Pecunia's code (she's using the Qt framework for file IO and image-processing), but I'm probably not gonna sort that out this week. She's agreed to help with packaging an .exe and a proper GUI once she has time, so I might confer with her on that.

@user3: I'm not sure that a GIMP plugin should be an especially high priority, to be honest- the utility should work fine with standard image formats like .bmp or .png. What do you think would be the time-saver here?

user3
Pleb

posted 11-13-17 10:12 ET (US) 44 / 52

@Pelaguis_II
Single frame changes would be OK. Modders can just export from GIMP the animation as single frame BMP/PNG images.

If an animation modder wants something more then he/she can ask for it later, get someone to rewrite the source for them or develop a plugin themself.

Will there be a method to change the animation numbers(frames per set, direction, offsets, focus pixel, sizes) or is just raw image data your going for, like the C3 image updater program for caesaria?
{I have no clue as the proper names in animation and graphics, just guessing from my imagination}

[This message has been edited by user3 (edited 11-13-2017 @ 10:40 AM).]

Pelagius_II
Pleb

posted 11-13-17 10:52 ET (US) 45 / 52

I think there are data fields in the SG files for specifying total animation frames, sure, but I would not be confident about whether any changes to those will be accurately reflected in-game. Given that all the walker-IDs are hard-coded, as it were, I'm not sure that their size/facing/etc. aren't as well. It might work, it might not.

Do you have a sample of a changed animation-sequence that you want to try substituting?

user3
Pleb

posted 11-13-17 12:09 ET (US) 46 / 52

I am not a programmer nor a graphics animator. Whatever is useful for your C3 modding project will do fine.

If someone like member Armel wants to work closely with you then they should speak up.

I have drawn Pelagius_II into this conversation to connect him and Pecunia after she showed an interest in rewriting sgreader and creating sgwriter.

Is 'Social Networking' the correct term you young people use. (shrug)

hoping for the best
-user3

[This message has been edited by user3 (edited 11-13-2017 @ 12:33 PM).]

Pelagius_II
Pleb

posted 11-13-17 17:33 ET (US) 47 / 52

Sure, I'm happy to talk to Armel. A "Shogun" mod for Emperor seems like a logical step, and if they can do pixel art, so much the better.

EDIT:

...Wait a second. There's a Russian website here with screenshots of what looks very much like edited in-game C3 graphics, and a download link to the C3Modder program:

http://caesar-iii.ru/download/mod_gfx.html
http://caesar-iii.ru/download/utility.html

Some nice-looking art, too. Has nobody here talked to these guys? It looks like they've already got this working.

.

[This message has been edited by Pelagius_II (edited 11-13-2017 @ 06:08 PM).]

alincarpetman
Pleb

posted 11-18-17 23:34 ET (US) 48 / 52

@ Pelagius

If you translate the Russian C3 site there's some interesting stuff there:

To modify the game:

SGReader - 1,51 mb - allows you to view graphic files and extract graphics. Also opens the graphic files of Pharaoh, Zeus and the Emperor

C3Modder v2 - 7.78 mb - actually, a modding program that allows you to insert your graphics into the game. Also suitable for Pharaoh

Miscellaneous:

Caesar3 Mapper 1.2 - 1.47 mb - a small program to save the minimap in PNG format from maps and saved games.
For more information about the program, please visit the author's page

C3 Excel (2014.06.07) - 120 kb - excel table, can count: prosperity; population; tax; food 1; meal 2; food 3; dishes; furniture; butter; wine (author: Angel A )

C3 trade - 67,6 kb - allows to calculate expenses and incomes from trade

Deleter 1.2 - 42.5 kb - displays all the saved games and maps in the game folder, with the possibility of their further removal.
The program must be placed in the game folder

C3GameExplorer - 70,3 kb - the program for editing saved games. Allows you to enable / disable the types of resources available on the map, change trading prices, open / close available trade routes and change the cost of opening them, change the current ratings. (sent by: Oldnick )

BuildingsOff - 78,3 kb - disables the ability to build certain buildings in saved games

NewEnemy - 114 kb - the program for changing enemies in the saved games

Clipper - 301 kb - unpretentious program for taking screenshots from the game

A lot of these individual programs would ideally be combined into one User Patch as with AOE 1-3's user patches.

There's a forum here and the links section has a linkback to HG's C3 site, I'd say it's worth sending an email to: vguild (at) yandex (dot) ru to attempt networking on this project.

[This message has been edited by alincarpetman (edited 11-18-2017 @ 11:35 PM).]

Pelagius_II
Pleb

posted 11-19-17 12:25 ET (US) 49 / 52

There's a forum here and the links section has a linkback to HG's C3 site, I'd say it's worth sending an email to: vguild (at) yandex (dot) ru to attempt networking on this project.

I actually sent emails to both that address and the primary author of the C3Modder tool, but haven't heard anything back so far. I didn't realise the site had it's own forums, though, so I might see if I can get response there. (There is the language barrier, of course, but maybe google translate will suffice...)

I like some of the translations there. pottery==dishes, oil==butter. Heh.

EDIT:

Welp, I've posted a new thread here. Have to see if anyone bites...
http://caesar-iii.ru/forum/viewtopic.php?f=8&t=216

.

[This message has been edited by Pelagius_II (edited 11-27-2017 @ 12:13 PM).]

Armel
Pleb

posted 02-06-18 12:23 ET (US) 50 / 52

Sure, I'm happy to talk to Armel. A "Shogun" mod for Emperor seems like a logical step, and if they can do pixel art, so much the better.

I'm sorry, I'm sorry, I'm sorry. I've been busy and I have just entered now in Emperor heaven to check if someone added some clue for retexturing emperor and YESSS!!!! You did it!!! But I cannot answer as the post is closed.

I hope it's not too late, I'm very interested. Maybe I'm going to be slow because I still in exams and in a few weeks the season starts again, But Yes I CAN DO pixelart, And to retexture the emperor for a japan mod is like a dream for me.

I hope to still have time.

Pelagius, I will be very grateful if you have some way to do that.

[This message has been edited by Armel (edited 02-06-2018 @ 12:25 PM).]

Pelagius_II
Pleb

posted 12-19-18 05:41 ET (US) 51 / 52

Hey there Armel-

I'm afraid I've been even slower to check the forums than you were, but if you ever check back at some point, you should be able to access my email from my github page once you sign up for an account yourself. If you're really interested, get in touch.

Mazeppa
Pleb

posted 03-23-20 21:03 ET (US) 52 / 52

It'd be great to have something like this, tbh! Something like what guys did at AoK and AoE 1 (like Trigger Studio and Rock n RoR etc) but for Zeus, Caesar, Pharaoh and Emperor. That'd be really cool to see.

Too bad no-one has really done this.. and modding seemed dedicated only to Caesar3 throughout the last couple of decades... But aside that, C3Modder looks like an interesting tool for Caesar3 and Pharaoh, and I wonder how it really works.. I've also never heard of Clipper before.

[This message has been edited by Mazeppa (edited 03-23-2020 @ 09:04 PM).]

« Previous Page 1 2 Next Page »

Caesar IV Heaven » Forums » The Town Square » A call to make a modding Toolkit for the CB series.

Top

You must be logged in to post messages.
Please login or register

Hop to:

Caesar IV Heaven | HeavenGames