1. Ue4 Build System
2. Ue4 Build Data

The upcoming version of this software-development environment will replace Unreal Engine 4, which has been around since 2014. It’s used in such Mac, iPhone and iPad games as Fortnite. To package an executable that will run on a Mac, you simply need to package for Mac and run the executable on a Mac (it will not run on your PC for testing.) For iOS devices, you will need to obtain a certificate and provision per project, but these are free. As stated before, you only need to pay the fee to publish your project on the appstore. 4 years ago I started to program games as a hobby. I have since then published 4 games on the iOS app store and on google play. I released one game on the mac app store as well. In those 4 years I gained a.

Image Source: Twitter/nightmask3

_{Estimated Read Time: ~16 minutes}

Table of Contents

1- Introduction: Project State & Goals

I have recently joined a group of friends to develop a 3D parkour game in Unreal Engine 4.

The development team was small, it consisted of:

• 1x technical artist / programmer
• 1x sound engineer / producer / designer
• 1x 3D artist

Ue4 Build System

You can check out the team’s Twitter Page here.

The team was using the stock UE4 engine with a couple plugins for art tools such as Houdini, Substance and some utility plugins like AutoSettings.

We wanted to be able to build UE4 from source and make changes to the engine as necessary.Even though I have been working with UE4 source for a while and was familiar with the development workflow, I haven’t really dealt with build and distribution aspects of it. Thankfully, almost all of the game studios I worked with had a dedicated build engineer to handle all this which allowed me to focus on the performance aspect of the titles.Well, up until this time: Now I had to figure out how to setup a UE4 development and distribution workflow and the small team size was making it an achievable goal.

No one in the team dealt with this kind of problem before, and we couldn’t find anyone around our friend circles to ask.Turns out not many build engineers are around, who knew?
There were many unknowns for us to figure out:

• Building UE4 from source is pretty straightforward and well documented, but how to package it for distribution?
• Which files to push to the source control and which files to ignore?
  • We’re talking of processing a few gigs to tens of gigs of binaries, asset files, source/header files, etc.
• How do we go about the plugins with the custom engine?
• What would be the process of upgrading an engine version once we start customizing?
• How to configure the game project so it uses the custom UE4 engine and how to do it seamlessly for the team?
  • Just double clicking a couple files should suffice, no one should be doing any ‘advanced user’ configuration

With all these in mind, I started to think of the next steps considering the current state of things:

The State

• Small team, no dedicated programmer
• Stock UE4.21 - 3-4 major versions behind latest release
• p4 for source control, depot contains only the game project contents
• Packaging the game fails with build-time errors

The Goals

• Engine upgrade to latest UE4
• UE4 build from source, ability to customize as needed
• Engine distribution to team members
• Figure out an efficient p4 workflow for game projects & team members to use the custom UE4 build with
  • Seamless integration
  • Build automation

2- First Steps: Fix errors, upgrade stock UE4, figure out folder hierarchy

The obvioust first step was to fix the packaging errors and carry out an engine upgrade to improve the stability of the project all together and provide a more stable and improved engine version to base the custom engine on.So we did that, fixed the packaging errors and upgraded to UE4.24 - the latest release at that point in time.

Next came some questions:

• Where do we put the engine source directory? - In the game folder? Next to the game folder?
• Where do we put the packaged engine binaries?

At this point, I’ve been looking for some kind of guide on how to setup a p4/UE4 development workflow and I came across some great resources along the way. See References section for the full list.

I want to point out the [5] Wise Engineering - Managing Multiple UE4 Projects resource which was particularly helpful for deciding how to move forward with restructuring the folder hierarchy. The author lays down a folder structure for a customized UE4, supporting multiple game projects while adhering to the needs of the different disciplines (programmers vs artists):

Content developers download the precompiled build for their platform and utilize that build for their workload.

Programmers typically recompile their own local builds of their project.

Build labels are often used to give a finer control over what build a team member uses.

The artists would just get the engine binaries while programmers could utilize both by using the engine built from source for development and the pre-built binaries for testing.

We ended up using the proposed idea and with the following hierarchy in the repository

ZenGarden and ProjectGilgamesh are the game projects

Looking back at this, you could probably also add a Documentation folder here and maybe prepend the game project folders with an underscore to highlight and sort the game project folders. Anyways.

Having the folder hierarchy figured out, next step was to figure out how to configure the perforce server and workspaces.

3- Configuring Perforce: p4ignore & type mappings

The usual workflow for building UE4 from source goes like this:

• Clone the UE4 repo on GitHub_{(Membership to the Epic Games org needed to see the page)} and switch to the specific point release branch you want to build
• Run Setup.bat
• Run GenerateProjectFiles.bat
• Launch UE4.sln

Setup.bat downloads a whole bunch of binaries, media files, patches, header files, etc. If you are running on a budget server, you’ll want to be conservative on what to push to the server in order to avoid filling the available space. The UE4 source folder should take 4.50GB for UE4.25 - this was the size when you synced UE4 from source around the time 4.25 was released. Running Setup.bat will add ~45GB of files to the directory, and if you commit these to p4, you’ll be in for long sync times and will run the risk of running out of space on the p4 server which can also be an unpleasant experience based on your p4 server provider/server.

To avoid all that hassle, I have run ‘Reconcile offline work’ after running Setup.bat on p4v to see which directories get populated by the setup script and added them to the p4ignore.txt as follows.

Adding this ignore block on top of the regular UE4 p4 ignore list, running Reconcile Offline Work after Setup.bat should no longer detect any files.

This helps keeping the repo size in check: ~5GB instead of ~50GB.

So far so good for a stock UE4 setup. If you want to distribute the customized UE4 to your team, you’ll probably want some plugins built & pre-installed. For that, you’ll need to add the plugin source files that can be acquired from the Epic Store, released for the version your custom UE4 is based on. Make sure to check the license for redistribution.

The plugins usually go in one of two places

• EnginePluginsMarketplace
• EnginePluginsRuntime

To properly add the necessary files to build plugins with the custom UE4 source, we add the following to the p4ignore:

Here is the final p4ignore.txt with all the intermediate / cache folders ignored in case you’re interested.

In addition to the ignore list, we also need to configure the file type mappings before we can start using the customized UE4 workflow.

Type mappings tell the p4 server which file permissions to use for certain file types and the UE4 Documentation page on Perforce Setup lays out the details on how to configure this for a UE4 workflow. We ended up using the type mappings mentioned on the website, but with a minor change:

• Add write permission +w to the config files

We did this as the engine packaging process writes into some config files and the build process would fail without it.

Note: If you’ve done this and found out giving all the config files write permission might cause an issue further down the line, please let me know in the comments!

Building the UE4 from source will overwrite some files tracked by the p4 server such as executables, binary files, libraries, and in certain cases config & manifest files. With the type mappings above, we have given write access to these files so the team members cam overwrite them without the need to check them out first with p4v.

Now that the version control is configured, we’re good to start development on the UE4 source drop.

Ue4 Build Data

Next we need to figure out how to package the engine.

4- UE4 Installed Build: Building the Engine for Distribution & Build Automation

Unreal Documentation calls the process Using An Installed Build.

It boils down to the following:

• UE4 uses a scripting system called BuildGraph to generate build tasks and ultimately package the engine
• AutomationTool.exe drives the BuildGraph system, built through UE4.sln
• Build tasks are generated through a configuration file: [UE4Root]/Engine/Build/InstalledEngineBuild.xml

If you start a command line terminal in [UE4Root]EngineBinariesDotNET directory, you can package the engine running the following:

Let’s take a quick look at the parameters:

• We choose [PLATFORM]=Win64 as we’re working on Windows operating system for the -target parameter.
• We want a -clean build.
• Use InstalledEngineBuild.xml for build configuration

The InstalledEngineBuild.xml file contains a bunch of settings. First, we see some default value definitions for various platforms:

Then, these default values are used to determine what platform the engine can package a project with.

Packaging is already a very long process: It takes more than an hour on my 12 core Ryzen 9 CPU while utilizing 100% CPU most of the time. If you’re not planning to build your game project for certain platforms, you can set some of the default values to false to exclude those platforms if you don’t want to wait (a lot) longer.

In our case we didn’t care about Mac, Linux, IOS or Win32 or the XR platforms and wanted only Win64, so we set the following values for defaults:

The output directory for the packaged engine is determined a couple more options later

You can leave it as is or change it to your liking.
If you note the p4ignore earlier, we’ve listed the LocalBuilds/ directory as we’ve decided to leave the output directory as is.

To automate the entire packaging process, I’ve written a batch script with the following steps:

• Check whether UE4.sln file is exists, generate it if it doesn’t exist
• Check if AutomationTool.exe exists, if it doesn’t
  • Find MSBuild.exe using vswhere.exe
  • Build AutomationTool.exe with MSBuild.exe
• Clean up engine output directory if it already exists
• Run AutomationTool.exe to package the engine

PackageEngine.bat

At this point, one can follow the 3 simple steps to package the engine after syncing the repo the first time:

• Run Setup.bat
• Run GenerateSolutions.bat
• Run PackageEngine.bat

Get some tea/coffee, take a walk, practice your favorite musical instrument as this’ll take a while…

And finally…

As you always test your new engine build because everything is assumed broken until tested (right?), you might find out that even the build was successful, runtime errors can still bother you. In our case, Substance was causing trouble with the custom UE4 build.

Using plugins with custom UE4 brings its own problems to the process. Substance was failing loading with a ‘DLL not found’ message when the game project was launched with the custom UE4.

To fix the packaging issue, we’ve extended PackageEngine.bat to copy over the missing DLLs right before exiting:

^ This goes right between popd and exit /b 0 of the PackageEngine.bat source earlier.

5- Engine Distribution & Project Association

Now that we have our installed build ready, we can distribute it to the team.

Again, to automate the process, a batch script is written in the engine source directory (UE4-TK) to copy the packaged engine binaries into its final destination, to UE4-TK-Editor:

PublishEngine.bat

After running this script, ‘Reconcile Offline Work’ on the editor directory should be able to detect the changes made to the engine binaries on consecutive publishes. Remember to delete the final editor folder before reunning the PublishEngine.bat to avoid leftover files from the previous build, in case there are any.

I can possibly extend the PublishEngine.bat to further automate some more work in the future:

• Do some testing
  • Test whether the custom build launches successfully without a crash/error
  • Test whether the project launches successfully
• Push to perforce server

You’ll notice in the script above, we also copy a RegisterEngine.bat script, which doesn’t come with the stock UE4 and wasn’t mentioned before. Before I explain what it is, I want to talk about Engine Association.

If you open a .urpoject file with a text editor, you’ll see which engine version the project is using. Since there could be multiple UE4s installed on a system, there are multiple ways you can specify which UE4 build to use.

• Stock UE4 simply uses a version string
• Source builds use a hash created by UnrealVersionSelector.exe
  • This hash is saved in the registry under ComputerHKEY_CURRENT_USERSoftwareEpic GamesUnreal EngineBuilds
  • This happens when you right click a .uproject file and select Switch Unreal Engine Version…
• You can use a custom string

The last two options utilize the Windows registry to work:

To automate creating this registry entry, a simple batch script is utilized:

RegisterEngine.bat

This script is then distributed with the engine source drop and engine binaries so it’s super easy to update the registry entry for the desired custom UE4 installation directory.

After the .uproject is updated with the custom engine string, double clicking the .uproject file will launch the custom UE4 which concludes the work on publishing the engine.

6- Customizing UE4 & Engine Upgrades from Epic

UE4 source could be considered a large codebase with lots of contributors from different domains:

• Epic Games Devs
• Open Source contributors
• Plugin authors
• Your own contributions for customizing

Working with several UE4 projects scaling from small to AAA in the past couple years, I’ve seen a common pattern with customizing UE4: Mark every modification to the original source with comments

Example: Modifying UE4’s PostProcess stack to add a sharpening pass developed by AMD called Contrast Adaptive Sharpening (CAS)

Notice a few things:

• Each change is encapsulated between *_CHANGES_BEGIN and *_CHANGES_END, clearly marking where the changes have been made
• Each *_CHANGES_BEGIN marker also has an explanation on what it is

These will be specifically useful for:

• Providing a way to separate changes coming from engine upgrade vs changes made for customization.
  This could be particularly handy when:
  • There are merge conflicts during engine upgrade
  • There’s an API-breaking change between major versions and you need to adapt your customization to the new API.
• Narrowing the search results for a specific set of changes

If there were changes other than FidelityFX / CAS, they would show up in this search list, clearly marking which changes modified which file

Finally, we need to figure out how to upgrade the engine after we make some modifications to the original source. As you may have noticed, Epic Games is using GitHub and git to distribute UE4 releases and we are using Perforce to version control the game project.

How do we mix these?

I didn’t get a chance to personally test the workflow, but the [3] Gamasutra: An Unholy Alliance: Unreal Engine, Github & Perforce resource lays out a pretty decent (in my opinion) way to do this:

• Start with a stock UE4 base source
• Modify the engine
• Update perforce with changes
• Create an offline branch of UE4 with git locally
  • Alternatively you can push this to your own server somewhere if you want to keep an online git branch for your modifications
• Commit your changes to git as well

When its time to upgrade the engine

• rebase your customization branch on top of the new UE4 release
• Resolve conflicts
• Update your git branch
• Update p4 server

Last notes

The post turned out a bit longer than I anticipated but I wanted to keep everything under one page and as concise as possible. Even though there are various great resources out there, I couldn’t find everything I needed under one page so I’ve created this one.

If you have a suggestion on how to improve this workflow, let me know in the comments!

7- References