MSDTC changes in WS2008, Part 3: Cluster features

As I mentioned in parts 1 & 2, WS2008 clusters are designed to have more than one functioning MSDTC instance.  Once the DtcGetTransactionManager[Ex] had been upgraded to connect to multiple MSDTC instances simultaneously, the issues next focused on configuration and management.

MSDTC Configuration

An MSDTC instance is defined by the information to establish an RPC connection, a log file, and some security and accessibility information.  The connection information consists of a NetBIOS name and four GUIDs, also known as connection IDs, or CIDs.  In a non-clustered system, the NetBIOS name is the one used by the system, and the rest of the information is located in the local registry.  The CIDs and log file information can be found under HKLM\Software\Classes\CID, and the pretty much the rest can be found under HKLM\Software\Microsoft\MSDTC.

These settings represent three roles: what does the MSDTC service need to configure itself, what does the system provide, and what does the MSDTC proxy need to contact the MSDTC service.  On a single node system it can be hard to tell these apart.

But then you look at a pre-WS2008 cluster and you'll see that some of the settings, specifically the CIDs, the security settings, and the log location, have been migrated into the cluster registry hive (the name is calculated, so there isn't a simple path to point out).  Information pertaining to the capabilities of an individual system, such as tracing, XA DLL registration, and version information remain in the various local hives.  The MSDTC proxy understands when it is in a cluster, so that information is not a clearly derived.

At one level, the WS2008 changes amounted to supporting multiple entries in the cluster registry, each of which is attached to a different cluster resource.  For backwards compatibility, one of the MSDTC cluster instances is designated the default cluster instance, and uses the cluster alias as the NetBIOS name. 

Add a command line feature to start an MSDTC service for a specific configuration, and we're done.

Picking 'the right MSDTC'

There's a big problem with the solution so far.  If the application or resource manager specifies the MSDTC instance in DtcGetTransactionManager[Ex], then the correct MSDTC instance will be used, and you'll get the benefits of from being able to spread the traffic around, from the instance always being local to your resource, and so forth.

Unfortunately, almost no one specifies the MSDTC instance to use.  This is partially due to the assumption that there was just one, hardened by the MSDTC proxy behavior to bind to the first MSDTC instance specified in the process.  Figuring out how to select a 'good' MSDTC instance was the next big hurdle.

A good MSDTC is one that isn't overly remote from the caller, that has the same availability characteristics, or better, of the caller, and one that doesn't add additional MSDTC instances for no good reason.  Oh, and did I mention that existing programs need to continue to work?

This then produced the following rules:

• Any caller that does not specify whether it is a local application or a cluster resource uses the default cluster instance.
• If the caller is specified as a local application, the local MSDTC instance, if available, is used.  Otherwise the cluster default instance is used.
• If the caller is specified as aligned with a specific cluster resource:
  • If there is an MSDTC instance that is part of that cluster resource, it will be used.
  • If not, and there is a cluster default MSDTC instance available, it will be used.
  • If there are no cluster instances available, then a local MSDTC instance will be used with a notification of a potential loss of availability.

A caller can be identified in two ways: programmatically or via configuration.  For the first, the DtcGetTransactionManagerEx call has been extended.  The call has the signature:

EXTERN_C EXPORTAPI __cdecl DtcGetTransactionManagerEx (

tchar * pszHost,      // NetBIOS name of the MSDTC instance, or null

tchar *pszTmName,     // must be null

REFIID riid,          // IID for the interface to retrieve

DWORD grfOptions,     // Configuration options

void * pvConfigParams,// Configuration parameters

void **ppvObject      // returned object instance

);

The structure passed in pvConfigParams is what has been extended to now look like:

typedef struct _OLE_TM_CONFIG_PARAMS_V2

{

    DWORD           dwVersion;         // Must be set to 2

    DWORD           dwConcurrencyHint; // Set to 0

    APPLICATIONTYPE applicationType;   // Specifies alignment to local or not

    GUID            clusterResourceId; // If not local, align to this resource

} OLE_TM_CONFIG_PARAMS_V2;

typedef enum APPLICATIONTYPE

{

    LOCAL_APPLICATIONTYPE,

    CLUSTERRESOURCE_APPLICATIONTYPE

};

Note that the call to DtcGetTransactionManagerEx can specify the specific MSDTC instance to use.  The new configuration parameter settings allow for a different approach.  Rather than having the caller specify the MSDTC instance to use, by name, these new settings allow a caller to make statements about itself that the MSDTC proxy will use to select a reasonable MSDTC instance.  The assumption is that statements that a caller makes about itself are less likely to change due to a configuration change in the cluster, and leaves more leeway for later optimization algorithms -- such as continuing to use an existing cluster MSDTC instance if it is already in use for that transaction.

It is also important to realize that in the case of the CLUSTERRESOURCE_APPLICATIONTYPE what the caller is specifying is the preferred cluster resource to align to.  In the case of a cluster resource this is most likely to be itself.  In the case of, say, a node-local COM+ component, this could well be the cluster resource that the SQL Server that it uses is in.

The second way that a caller can be identified is via external configuration settings known as TM mappings.  The configuration specifies executables, COM+ application IDs, and cluster resource IDs that are to be mapped to specific MSDTC instances.  These settings are used when the caller to DtcGetTransactionManagerEx supplies neither the name of an MSDTC instance or a OLE_TM_CONFIG_PARAMS_V2 structure.

The configuration settings can be found under HKLM\Cluster\MSDTC\TMMappings.  The values can be set, viewed, and deleted via options on msdtc.exe, as follows:

• To set up a mapping, use
  msdtc.exe -tmMappingSet -name

[-exe <pathname> | -service <full service name> | -complusappID  <app ID>]

[-local|-clusterResourceName <resource name>]

• To view a mapping, use
  msdtc.exe -tmMappingView -name
   
• To view all known mappings, use
  msdtc.exe -tmMappingView *
   
• To delete a mapping, use
  msdtc.exe -tmMappingClear -name
   
• To delete all known mappings, use
  msdtc.exe -tmMappingClear *

For instance, let's say that I have a application "JoesGarageShopApp.exe" that only works against local files and only on a single node.  I'd like that application to only use the local MSDTC instance, whichever that one is.  I'd do that with:

msdtc.exe -tmMappingSet -name JoesGarageShopMapping -exe JoesGarageShopApp.exe -local

What happens to resource managers?

From the point of view of the resource manager, effectively nothing.  They make the calls to the MSDTC proxy, and receive notifications in the way that they always have.  There are two significant changes, but they are done transparently behind the MSDTC proxy interface.

The first is that a transaction passed into a resource manager from an application may be using a different MSDTC instance than the one that the resource manager wishes to use.  If that is the case, the MSDTC proxy will emulate the operations to pull the transaction to the new MSDTC instance (see http://blogs.msdn.com/florinlazar/archive/2004/10/02/236965.aspx for a good description of how this normally works).

The second change is that pPrepInfo presented at IResourceManager::Reenlist may specify a different MSDTC instance than the one that is currently being used.  In that case, the MSDTC proxy, as part of handling the  Reenlist request, will internally issue the correct DtcGetTransactionManagerEx call to connect to that MSDTC instance as well.

Kernel Transaction Support

As something of a detail point, there is some underlying plumbing in place to handle failover of transacted NTFS and transacted registry resource managers.  The overall logic is the same as I've talked about above.  The one difference is that the kernel interfaces are not themselves remotable, so there is an agent service (the KtmRm service) that acts as the intermediary between the MSDTC instance and the kernel transaction manager.

Jun 1, 2008: Updated to fix a bug in the msdtc.exe command line syntax.