- Cloud Environment -

How to properly use AWS instance storage (ephemeral storage/disk)

  • If SIOS AMIs are not being deployed on an AWS server, these disk initialization steps must be considered. This was introduced with Windows 2016 and continues in later releases.
    • AWS ephemeral storage (Instance Store) must be initialized every time a node starts up from a “cold” shutdown (i.e. having been shut down in the AWS Console or from a CLI “shutdown /s” from Windows – not a reboot).
    • Windows 2012 R2 performs this step automatically (Ec2Config service). AWS includes scripts (EC2Launch) to perform this initialization but the script has to be manually scheduled by the user so that it runs at boot time.
    • The SIOS AMIs on AWS have this task set up automatically. If you launch your own Windows image in AWS and fail to set the script up to run at boot, and you are using ephemeral storage for bitmaps (as recommended in Best Practices), the bitmap file/volume will be missing and DataKeeper mirrors will not recreate.

SOLUTION
  • Launch Disk Management and make sure the ephemeral storage is on-line, healthy, and is assigned the volume letter you selected for the ephemeral storage.
  • Ensure that the drive/drive letter assigned to the ephemeral storage is listed in the registry located at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ExtMirr\Parameters . . . BitmapBaseDir

Note: Apply this workaround to all nodes in the cluster.
LONG TERM SOLUTION
Schedule the EC2Launch script to run at boot time on each system in the cluster to ensure that the ephemeral storage is always available.

- Administrative Processes -

How to change the hostname of an existing license

  • From the License Support menu select List License

  • Select the checkbox for the license you want to change (you can only do one license at a time)

  • Click the Action dropdown and select Repair

  • Enter the new hostname

  • Select Repair then select Complete
How to determine what GPOs have been applied to a server

ISSUE:

Group Policy Objects are sometimes deployed or redeployed after a server is rebooted impacting the way the LifeKeeper (GUI) or DataKeeper may or may not perform.



Solution:



There are two methods:



From Start\Run enter rsop.msc
  • The console will provide a list of policies that have been applied to a server
Or from an elevated command prompt, enter gpresult /Scope Computer /v
  • This output will display a list of policies that have been applied to a server

    Note: To determine what GPOs are being enforced per a user, enter gpresult /Scope user /v


- LifeKeeper GUI -

How to deploy OpenJDK on LifeKeeper

  • Download OpenJDK, the Windows x64 Installer, from https://openjdk.java.net/

  • Deploy the software on the Target node first

  • In LifeKeeper, perform a switchover/In-Service

  • Repeat the steps above on other Target(s) after the switchovers are completed



Note: This is a Highly Available deployment as you are never installing the software on the Source/Active node.

From a Command Prompt

type javac -version

Your output should be as follows:

“C:\Windows\system32>javac -version javac 15.0.1”

If not, you may need to add the following to your Systems Variable PATH/environment:

  • Control Panel\System and Security\System\System Properties\
  • Select the Advance Tab
  • Select Startup and Recovery
  • Select Environment Variables…
  • Select Path and then Edit

    Add C:\Program Files\Java\jdk-15.0.1\bin to your string to reflect the following in the example below:

    C:\Program Files (x86)\Common
    Files\Oracle\Java\javapath;SystemRoot%\system32;%SystemRoot;%SystemRoot%\System32\Wbem;%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;C:\Program
    Files\Microsoft SQL Server\Client SDK\ODBC\110\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Server\120\Tools\Binn\;C:\Program Files\Microsoft SQL Server\120\Tools\Binn\;C:\Program Files\Microsoft SQL Server\120\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\120\Tools\Binn\ManagementStudio\;C:\Program
    Files (x86)\Microsoft SQL Server\120\DTS\Binn\;C:\Program Files\Java\jdk-15.0.1\bin

    Perform the aforementioned on all nodes in your cluster(s).

- Mirror Solutions -

How to recreate a mirror for an existing clustered DataKeeper Resource

ISSUE:

A customer may have experienced some issues where a DataKeeper Resource is listed as Offline via Windows Server Failover Cluster.

Note: An existing job in the DataKeeper GUI will most likely have a RED X status



Solution:


  • Remove all of the old mirror remnants from the Source and Target nodes
  • Manually recreate the mirror via the emcmd . createmirror command



To remove the mirror configuration from both nodes:

  • cd %extmirrbase%/support (This is a shortcut to the DataKeeper\Support directory)
  • run “cleanupmirror (drive letter)Note: Run this command on the Source first. This command performs the following:
    • deletes the local mirror only (NO DATA)
    • resets the switchover flag
    • notifies the driver and service that no mirror is present


To verify that the mirror has been properly removed, execute the following:

  • cd %extmirrbase% (This is a shortcut to the DataKeeper directory)
  • run “emcmd . getmirrorvolinfo (drive letter)
    The output should reflect no mirror e.g. 0 servername 0



Now you are in a position to manually recreate the mirror.

The syntax is as follows:


  • emcmd (Source IP) createmirror (drive letter) (Target IP) A or S (Async mirror/Sync Mirror)
    Upon completion, the DataKeeper UI will now reflect a Resyncing status. This can also can be verified from a command line by executing:
  • emcmd . getmirrorvolinfo e
    The output will reflect “E: 1 servername (Target IP) 2, where 2 is a mirror definition, indicating a Resyncing status

    Return to Windows Failover Clustering and online the Failed DataKeeper Resource
Deleting mirrorendpoints

Steps to delete mirrorendpoints:


• Go in the registry for the node
• Go to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ExtMirr\Parameters\Jobs
• Find the job for that endpoint and remove the endpoint from that job
How to unlock a target/passive node disk that does not release locking the volume

ISSUE:

A customer experienced some issues where a volume was NOT unlocking on a Target/Passive Node

SOLUTION:

Within an Elevated Command Prompt on the Target Node, please perform the following actions:

  1. “cd %extmirrbase%” (This is a shortcut to the DataKeeper directory)
  2. Verify status of the mirror for the volume (i.e. EMCMD <system> GETMIRRORVOLINFO <volume letter>)
    1. https://docs.us.sios.com/dkce/8.9.1/en/topic/getmirrorvolinfo (Please see for the varying mirror states)
  3. Pause Mirror for Target Volume (i.e. EMCMD <system> UNLOCKVOLUME <volume letter>)
    1. https://docs.us.sios.com/dkce/8.9.1/en/topic/unlockvolume

The actions above allowed you to:

  • Launch Disk Management
  • Extend Volume…
  • Return to the DataKeeper UI; Continue and Lock Mirror

- Error Messages –

Status = 87 after executing “emcmd . updatevolumeinfo <drive letter>”

PROBLEM/ISSUE:
LifeKeeper is using a mirrored drive created via DataKeeper. Mirror is corrupted after the drive was lost. Mirror needs to be recreated.

The following tasks were attempted to correct the error on the offending node:

  • emcmd . deletelocalmirroronly <drive letter>;
  • emcmd . clearswitchover <drive letter>;
  • emcmd . updatevolumeinfo <drive letter>;

After the last command (updatevolumeinfo) you receive the “status=87” error message.

Notable:
A Status = 87 is due to:

  • The Volume is locked
  • The partition/Volume is no longer present (i.e. RAW or Unknown state)


RECOMMENDATION/SOLUTION:

Remove the mirror/volume dependency in LifeKeeper first.
As opposed to running the aforementioned commands, executing a “cleanupmirror“ will perform the same steps but bodes a much better result.

Then, using the DataKeeper command line execute these commands on the Source of the mirror first otherwise the Target mirror will continue to get created and/or locked:

  • Launch an elevated Administrator command prompt
  • cd %extmirrbase%\support (short to the Support directory)
  • run “cleanupmirror <drive letter>”

Verify the mirror has been removed by executing the following:

  • return to the \DataKeeper directory
  • execute a emcmd . getmirrorvolinfo <drive letter>;

The status should reveal 0 <drive letter>; 0, which would indicate the mirror is no longer present.

Then, repeat these steps again on the Target node(s) .

Verify the mirror has been removed from the Target node(s) by executing, again :

  • emcmd . getmirrorvolinfo <drive letter>;

The status should reveal 0 <drive letter>; 0, which would indicate the mirror is no longer present.

Result: You should now to able to recreate your mirror using the emcmd . createmirror command, thus returning to your LifeKeeper GUI and adding the Volume Resource/Mirror back into your cluster.

When executing a “emcmd . createmirror”, you receive a Status = 317

ISSUE:

Reasons for a Status = 317

  • The Target volume size (in bytes) is smaller than the Source when attempting to use the emcmd . createmirror command (See properties of the Volumes).
  • When attempting to create a mirror using “emcmd . createmirror “ via 2×1, shared buddy configuration, you may receive a Status = 317.


SOLUTION:

There are two methods:

  1. Ensure that the potential Target location is available or eligible for mirroring (make sure that the volume is seen in
    • Windows Disk Management (Properties of the disk) – Source/Target SAME size in bytes or the Target is LARGER
    • File Manager or Disk Properties, in Disk Management) – Source/Target SAME size in bytes or the Target is LARGER.
  1. For a shared buddy
    • On the offending target(s), execute the following command; emcmd . clearblocktarget {drive letter}.
    • Then verify that target(s) are set to FALSE by running emcmd . getblocktarget {drive letter}

Note: The bitmask for a “shared buddy” should be set to 384.

  • This can be verified by executing “emcmd . getconfiguration {drive letter}”.
  • If not, execute a “emcmd . setconfiguration 384” and execute “emcmd . getconfiguration {drive letter}” to ensure it has been applied.

Additional Assistance: Rebuilding a Mirror

If there is ever a need to rebuild a mirror while maintaining the DataKeeper Job and DataKeeper Volume Resource in the cluster/WSFC/Failover Cluster Manager the steps are as follows:

  1. Always begin from the Source:
    1. Launch an elevated Administrator command prompt
    2. “cd %extmirrbase%” (This is a shortcut to the \DataKeeper directory)

  2. Next we’ll need to identify the status of the mirror from this node/Source:
    1. Type “emcmd . getmirrorvolinfo “ (see the link below for the output this command produces)
    2. You can change the “.” to the destination servers, IP, Hostname, FQDN to determine status

  3. Now you need to remove the old mirror remnants:
    1. Execute “cd Support”
    2. The next command will delete the local mirror only (NO DATA), removes a switchover flag and notifies the SIOS DataKeeper Service & Driver that “No Mirror” is no longer present
    3. “cleanupmirror (drive letter)”
    4. “cd ..” (return to the DataKeeper directory)
    5. “emcmd . getmirrorvolinfo (drive letter)” The output should reflect “0 servername 0”
    6. Repeat the aforementioned commands on the Target(s)

  4. Now you are in a position to manually recreate the mirror(s):
    1. Launch Failover Cluster Manager to determine what server is the Current Owner of all resources.
      1. This will indicated the node that will be the Source when creating the mirror from the command prompt
  1. From the Source:
    1. Execute “emcmd . getjobinfo”
    2. Use the output to verify the IP addresses, drive letter(s) and Mirror Type(s) Async (A)/Sync(S) for recreating the mirror


The syntax for creating or recreating a mirror is as follows:

EMCmd <Source IP> CREATEMIRROR <volume letter> <Target IP> <Type> [options]

Example:
# emcmd 10.x.x.x createmirror D 10.x.x.x A

The completed output should say “Status = 0”.
  1. Return to the DataKeeper UI as you will view the mirror you’ve just created.
    1. Any data at this point that needs to be Resynced to the Target will indicate “Resyncing”


Once all Mirror(s) are in a “Mirroring” state, your WSFC roles a eligible for a Move/Switchover.

Commands to Consider:

Click here for access the list of all the emcmd commands

The most frequently use commands are the emcmd . get… commands:

For Mirror Status:
emcmd . getmirrorvolinfo
emcmd . getserviceinfo
emcmd . getjobinfo
emcmd . getcompletevolumelist

Security and Group Policy Criteria (accounts, policies, TCP/IP settings, rights, and roles, etc)
  1. DataKeeper requires that “File and Printer Sharing for Microsoft Networks” be enabled on the network interfaces to make a NAMED PIPE connection and be able to run DataKeeper’s command line tool (EMCMD).

    To test if you can make a Named Pipe connection, try to map a network drive on the TARGET system. If that fails, you have a Named Pipe issue.

  2. DataKeeper also requires that NetBIOS over TCP/IP and SMB protocols be enabled. If the GUI does not operate correctly, make sure the following network configurations are enabled:

  • Enable NetBIOS over TCP/IP and SMB protocols as in the following example:

    Start -> Control Panel -> Network and Internet -> Network and Sharing Center -> Network Adapter Name (Ex; Ethernet 2) -> Properties -> Internet Protocol Version 4 (TCP/IPv4) -> Properties -> Advanced -> WINS -> Enable NetBIOS over TCP/IP

  • The DataKeeper GUI uses the server Login ID and Password; therefore, the User Name and Password used to log in to the servers themselves must be the same on each server and must have administrator privileges.

  • The user must be logged in with domain admin privileges.

Feedback

Was this helpful?

Yes No
You indicated this topic was not helpful to you ...
Could you please leave a comment telling us why? Thank you!
Thanks for your feedback.

Post your comment on this topic.

Post Comment