In March 2017, Carhartt reported that their Database Server for Commerce which hosts Carhartt websites was experiencing issues. Users were being prevented from performing interactive processing on the system. The IT Department had started investigations into what was wrong with the help of the support teams at IBM.​

While these investigations were ongoing, Carhartt advised Maxava that they were considering a role swap to the backup server so that the user activity could return to normal. We were noted that the replication was still processing successfully to the backup server and it was only some interactive user activity that was being locked.​

When that decision was made by Carhartt management, Gary Adkins, IBM i System Administrator worked from his “recovery runbook” which described the steps he needed to take. This included the use of several automation scripts that had previously been created with the assistance of Maxava specialists. For example, after acquiescing the system, he actioned the process of rolling to new journal receivers. It was to ensure that there were no further updates arriving on the backup server from the production server, and the system was role-swap ready. “The support provided by the Maxava technical team, along with the clearly defined and documented steps for completing a role swap with Maxava HA, resulted in a smooth and quick transition to our target system,” recalled Adkins.​

The role swap commenced, first with a training configuration, followed by the real configurations. Carhartt’s diligence in their testing for role swap readiness meant they were prepared and well-practiced for these types of eventualities. The runbook instructions were built during these tests but have always been treated as living, breathing documents by Carhartt due to the fact that their environments change regularly.​

Towards the end of the role swap, Carhartt contacted Maxava technicians who helped review the successful completion of the swap tasks to date, and discussed the next steps. “The Maxava technical team was in contact with me throughout the role swap. The timely manner in which the role swap was executed ensured that Carhartt faced minimal impact to our ability to serve our e-commerce customers,” Adkins continued. There were a few items which needed a manual intervention. But the end result was that the backup server was brought online for the users to access.​

While the backup system was performing the role of the primary server, Carhartt was able to analyze and fix the issues on the production server. Once they were satisfied, Carhartt chose to role swap back to the original hardware before the start of the next work day.​

The success of the failover and subsequent recovery shows the strength of partnership between the Carhartt team and Maxava. In addition, by combining the best software and regular role swap tests, whether full or simulated, Carhartt is assured that they can respond to system outages without missing a beat. “Testing will help you develop a solid plan and feel confident in your ability to handle a role swap. But it is real-world successes like this that validate the process,” concluded Gary Adkins.​

Maxava actively promotes regular role swap tests for their customers because of the benefit that organizations gain from knowing they are ready. Combined with the fact that Maxava HA requires minimal care and feeding on a regular basis to be ready for those test exercises, a role swap or failover is not something to be feared.