Explaining the game patching / title update process

JD2020 wrote:Hey COD fans –

There seem to be a lot of questions from the online community surrounding the process that developers go through to patch / title update console games, so I want to shed some light and remove some of the mystery by posting a walkthrough of each phase of the process. All in all, it can take anywhere from a minimum of 2 months to several months, depending on the variables involved in a particular game update.

Phase 1: Listen to the community.
* As soon as the game is released, we scour the forums and play online with the fans to listen to feedback and keep an eye out for issues that degrade the experience for gamers.
* We build a list of every issue reported, which goes immediately into our test lab for reproduction.
* Estimated time: minimum of 2 weeks to build a comprehensive list. In actuality, this is an ongoing process that starts as soon as the game is released and goes on forever. It’s given a minimum of 2 weeks here because the first two weeks of a game’s release typically do not yield enough information about the game to be considered comprehensive.

Phase 2: Reproduce the issues and engineer a solution.
* The test team tries to reproduce the issues in a test environment, and enters bugs into our bug-tracking database to assign to engineers, artists, or designers.
* It’s imperative that we reproduce the issues in the test lab. Video games are highly complex pieces of software – every change made to the software impacts something else in the game, often in a way that introduces more bugs. For this reason, it’s very dangerous to “fix” bugs that cannot be reproduced in the test lab because team members don’t know with 100% certainty what the cause of the issue is. I would estimate that 95% of bugs that are reported can be reproduced in the test lab.
* Once the issue is reproduced and the bug is assigned, the team researches the cause of the issue, engineers a solution, compiles it back into the game, and tests their fix.
* Estimated time: minimum of 2 weeks to make the rounds through QA, spend time researching the causes, and implement solutions. This phase is typically the longest part of the process and will generally take more than 2 weeks, depending on the number of issues reported and the complexity of those issues. Once a game is released, every change made to it is a high risk. We never want to make a problem worse, so adequate time is needed here to implement the best possible solution.

Phase 3: Test the updated game and submit to the console manufacturers for approvals.
* Once the team is satisfied that the list of issues has been resolved, the updated game goes through a rigorous internal testing procedure to ensure that all issues have been addressed without introducing any new issues.
* This process is iterative and is mixed in with Phase 2. If, during this phase, new issues are discovered or the original issue is not fixed appropriately, the team has to spend more time researching and implementing a solution, and Phase 3 starts over again. As stated above, every change made to the software has a potential impact somewhere else in the game, so we cannot just test the one issue – we have to test the entire game.
* Estimated time: minimum of 2 weeks to thoroughly test the entire game, iterate on fixing issues, and internally approve the updated game for release. Again, this phase typically takes longer than 2 weeks, but this is a best-case scenario.

Phase 4: Console manufacturers test and approve the game update for release.
* As soon as our QA team has approved the game update internally, it is then submitted to the console manufacturers for their own testing and approval. At this point, it is completely out of our hands as developers.
* If the game update is approved by the console manufacturers, the process is pretty much done. If the game update is rejected, an abbreviated version of the entire process starts again from Phase 2.
* Estimated time: minimum of 2 weeks to get an approval report from the console manufacturers, typically less time to get a rejection report.

Phase 5: Release!
* Once the game update is approved by both internal QA and the console manufacturers, it gets prepped for release. This is typically a very short phase, but it can add time to the process depending on outside factors.
* Estimated time: minimum of 2-3 days.

That is the patching / title update process in a nutshell. Of course, the intricacies of the process are far more detailed than this, but this description provides a general overview. The minimum estimates of time listed above are best-case scenarios, and on average, it takes longer than 2 months. In rare emergency cases, this time can be trimmed down, but this is very rare, and depends on what is being addressed.

You have also seen posts in the forums regarding “hot fixes”. The description above outlines the process for publishing permanent game updates and does not apply in the entirely same way to hot fixes – hot fixes are temporary, network-based fixes that are used to patch very simple issues in the game. These are not permanent fixes, and they only apply to online portions of the game. There are only a few issues that we can address with hot fixes, and they require the same amount of internal testing as full game updates.

It’s also important to note that not all issues can be addressed in a patch / title update. Once the game is printed to disc, certain aspects of it are committed for the life of the game, due to memory and other technical limitations.

There you have it – hopefully this removes some of the mystery of the process and educates the community as to why game updates take as much time as they do. Enjoy the game and keep communicating your feedback to us!

Regards,
-JD