The Book of Db2 Pacemaker – Chapter 5: Pacemaker Configuration Parameters: Across the Db2-Verse

In Chapter 2 of this series, Gerry addressed the following frequently asked question:

Can I make custom configuration changes to the Db2 Pacemaker resource model?

with a two-staged approach:

• Stage 1: Make public a list of conventions used by Db2 and Pacemaker/Corosync configuration parameters that must not be tempered with.
• Stage 2: Automations and tooling to enforce the above guidelines.

Well … you may view this chapter the grand introduction of Stage 1 with the presentation of THREE major considerations when it comes to tweaking the Db2 base resource model across countless diverse Pacemaker cluster deployments in the immense Db2-Verse!

Consideration 1 – Leave our configuration parameters alone!

First, let’s level set on the expectation.  There are likely hundreds of configuration parameters in Pacemaker and Corosync combined. Db2 only became aware of a fraction of them during our robust test cycle.  It is, therefore, impossible to produce an exhaustive list of what can and what cannot be modified.  Db2, like every other Pacemaker consumer, relies on the defaults for most parameters chosen by Pacemaker and Corosync.

To help facilitate the discussion as to what extent in the resource model you are allowed to tweak, and the guardrails articulated in this blog, I elect to use the peeling of the onion analogy as illustration.

In the picture above (reading bottom up),

1. Layer 1 is the innermost core representing the lowest denominator across the entire Db2-Verse, meaning that every single deployment in all form factors with Db2 Integrated Pacemaker will adopt the Pacemaker/Corosync defaults unless otherwise overridden at a higher layer.
   
   
2. Layer 2 contains Pacemaker/Corosync configuration parameters set by Db2 in the base resource model.  The values are carefully set based on our iterative experimentations on all supported scenarios.  These parameters have profound impact on failure detection timing, node liveliness detection, quorum & fencing behaviour, monitoring level, etc.  Given the impact, none of the these should be overridden by higher layers.
   
   
3. Layer 3 is “special” in the sense that it doesn’t exist (yet).  That’s said, getting popular products such as QRep to have its own resource agents integrated into the base Db2 resource model is a matter of WHEN, not IF.  As a result, this layer can be seen as an “Extension” to the base Db2 resource model where it may add additional configuration parameters setting to the “untouchable” list in the future.
   
   
4. Layers above 3 (labeled as Customized Resource Model 1 through X in the diagram above) refers to all consumers of Db2 base (and extended in the future) Pacemaker resource model who have a need to add to the resource model for additional monitoring or other purposes.

Now, irrespective of the type of changes intended to cater towards your specific use case(s), the following must be adhered to when the Db2 integrated Pacemaker solution is used:

Reason: There might be a chance where configuration parameters not specified in layers 2 and 3 are adjusted from their Layer 1 defaults resulting in a behavioural change that manifest itself externally as a Db2 unexpected behaviour.   It is important to recognize that while Db2 strives to uncover these settings, it is not possible to validate every permutation as previously mentioned.  This is the risk you must assume when modifying Pacemaker/Corosync configuration parameters and the reason why you are advised to always perform standard failover testing in your test environment before putting such changes in production. 

On the other hand, the “untouchable” list is an evolving one, your input and findings can pave the way for Db2 to enhance the resource model for other general purposes.

Let me end this segment with the following golden rule:

This means there can be no “cross-layers” override of configuration parameters.  This is the only way to mitigate risks in your changes.

Consideration 2 – Leave our resources and constraints in base model alone!

Enough said about configuration parameters as “untouchables”.  The reality is that they are not the ONLY untouchables.  The following aspects of the resource model are equally important. 

1. Nodes 
2. Resources
3. Constraints including location, colocation, and order constraints
4. Cluster configuration properties
5. Resource defaults 

Each of the above are set up by either our cluster management utility, db2cm, or the related resource agent scripts.  Their names, associated parameters, and relationship scores must not be altered manually unless it is performed through the db2cm utility, or at the request and under the guidance of Db2 support. 

Note that resources and constraints configured by db2cm are all prefixed with “db2_” except for the fence agent where it has a prefix of “fence_db2_”, you should NOT name your non-Db2 resources with these prefixes. 

Without loss of generality, let’s use a sample HADR resource model and walk through the five aspects mentioned above in detail, highlighting the “untouchables” in each.  The same rules apply to all other High Availability configurations supporting Pacemaker today, namely Mutual Failover, and in the future, DPF and pureScale.

1. Nodes

The untouchables include:

• Node name (i.e. pcmk-srv-1, pcmker-srv-2)
• Node ID (i.e. 1, 2)
• Any listed attributes. These are used in the Db2 engine for various purposes.

• • db2inst-pcmker-srv-1_db2inst1_0_start=0
  • db2hadr-db2inst1_db2inst1_HADRDB_reint=0
  • db2hadr-db2inst1_db2inst1_HADRDB_takeover=0

2. Resources

The untouchables include: 

• Resource name
  • The parameter prefixed by “db2_” immediately after the “primitive” keyword.
    
    
•  Resource agent
  • The parameter immediately after the resource name.
  • From the sample above, db2hadr, IPaddr2, db2inst, db2ethmon are for HADR.  Different HA configurations such as Mutual Failover, DPF, and pureScale will have their own specific resource agents.
    
    
• Resource agent parameters
  • The space delimited list after the keyword “params”.
  • These are the parameters passed into the corresponding resource agent for processing.  Under no circumstances should these be modified.
    
    
• Operations along with their interval, timeouts, and on-fail attributes.
  • All lines begin with keyword “op”.
  • For most resources, there should be at least three such keywords / lines to represent the behaviour for the standard START, STOP, and MONITOR operations for the associated resources.
  • The timeout, monitor interval values are well tested and should not require any adjustments.  In most cases, a need to tune these values is an indication of a problem elsewhere in the system.  E.g. if starting the instance took a while that exceeds the current START timeout of 900s (15 minutes), the wrong solution is to extend the timeout of 900s to higher values.  The starting of an instance should not take that long, perhaps some resources are not available which itself needs to wait for the timeout before it can proceed to the next step in instance start-up.  That should be looked at instead of tuning up the timeout period.
    
    
• Meta-attributes
  • These are resource level metadata representing the state of the resource (in the case of “is-managed”), failure & recovery behaviour.  Altering these will surely impact the well-tested behaviour.  As such, unless instructed by Db2 support, these values should NOT be altered.

In addition to the “No-alter” stand for the above, no new parameters, operations, or meta-attributes should be added unless they are configured through the db2cm utility. 

3. Constraints

(NOTE: the above is only a snippet of all the constraints in the HADR resource model)

The untouchables cover every aspect in location, colocation, and order constraints configured by db2cm utility, namely the following:

• Constraint type
• Constraint name 
• The resource names (and their order) as specified. 
• Constraint score 

In addition to disallowing “cosmetic” changes to Db2 resource definition, it is important to recognize that extreme care must be taken into account if constraints were added to correlate between non-Db2 resources and Db2 resources.  Some may be safe; others may lead to a complete disaster.  For example, in HADR, if a new colocation constraint were to be created between a non-Db2 resource that can only run on one of the two hosts with the primary database, then the primary will only be able to run on that host – completely breaking the HADR model.

4. Cluster Configuration Properties

All parameters under the “property” keyword have cluster-wide scope that should not be tempered with unless the change was done through db2cm utility or under the guidance of Db2 support. 

5. Resource defaults

These represent the defaults to be used for Db2 unless otherwise overridden intentionally at individual resource definition.  It is critical that these remain unchanged, and no additional ones added unless the change is triggered by the db2cm utility or under specific guidance from Db2 support.

Consideration 3 – Other Heads-up!

While the above set the ground rules and perhaps gives you the green light to make modifications to the Db2 base resource model, you should also be aware of the following:

1. Validation of the modified Db2 resource model

It is recommended that you first perform the change in your test cluster prior to production.  It is also paramount that you conduct your own validation (e.g. resource failover, host reboot, and others deemed critical in your use case) to ensure your changes indeed have no impact to the base Db2 behaviour.

2. Fast Redeployment via db2cm -import / -export capability

“db2cm -export” is typically used to perform periodic back up of the resource model for safety measure.  It can also be used to do redeployment on the same or different set of hosts.  The latter requires some manual changes to the exported file.  Please note that there is no guarantee your additional non-Db2 resources will be exported.  By the same token, you may not edit the exported configuration file by adding non-Db2 entities into it.  “db2cm -import” will not recognize them and fail the operation.

3. Resource Model Repair

In future release, there will likely be a new option “-repair” to reconstruct the resource model from scratch.  In this scenario, none of the non-Db2 resources introduced will be retained.  You will need to recreate them manually after the repair action completes.

4. Features in Pacemaker/Corosync not used by Db2

If your resources rely on Pacemaker/Corosync features currently NOT used by Db2, please be aware that:

• Db2 does not support them. You may not open a Db2 support case on Pacemaker features not used by Db2 directly.
• Those features may not be packaged in future Db2 Pacemaker.

5. Db2 Support

Db2 support only handles issues with the Db2 base resource model, meaning that you are responsible for all non-Db2 base model resources behaviour.  You may be requested to revert to the base Db2 resource model for the purpose of narrowing down the problem.

The Million-Dollar Question – “What change is actually allowed then?”

In a way, this is already answered in the above.  But perhaps, let me be give a few more concrete examples here.  Conceptually, most requests I have received to customize the resource model are for the following reasons:

Scenario 1:  Need to introduce one or more entities into the resource model where they need to be monitored and be associated with one of the existing resources so that if the resource fails, the new entities will also flow to the standby (in the case of HADR or Mutual Failover).

This is a reasonable request. It will require, but not limited to, the following actions from you:

• • Introduce your own resource agent(s) with implementation to the START, STOP, and MONITOR actions to manage your new resource(s).
  • Creation of your resource(s)
  • Creation of constraints (be mindful of the side effect of altering Db2 behaviour as described in Consideration #2, under Constraints section)

If you do not alter configuration parameters outlined in this blog, the existing base Db2 resource model entities are not modified, and you are aware of all other considerations pointed out in Consideration #3, this is an acceptable change.   

Scenario 2:  Alter the database failover behaviour from individual to either group of them or ALL at the same time.

This is another reasonable request.  In fact, we have received Aha idea to provide this support.  Note that this requires changes to the base resource model.  So, you should not modify this on your own.

Scenario 3:  Alter the timeout value of monitor action

A reasonable request but it will not be done as part of base model changes.  Typically, this change is a result of Db2 support investigation. If the response from Db2 support is to modify the individual timeout, it will be a supported change.  But this should not be modified without the consent of Db2.

To sum it all up.  Yes, you may customize the base Db2 Pacemaker resource model if you agree to all of the above :D.  I would like to apologize for the rather lengthy blog … but it speaks to the volume of complexity involved in modifying the base Db2 resource model.  Next up is the penultimate chapter in this series – The Book of Db2 Pacemaker Chapter 6 – Pacemaker log: Everything Everywhere All at Once.  Stay tuned…

 Quick reference to this series:

• Chapter 1 - Red pill or Blue pill ?
• Chapter 2 - Pacemaker Cluster … Assemble!
• Chapter 3:  Pacemaker Resource Model: Into the Pacemaker-Verse
• Chapter 4: Quorumania