Solutions

- Cloud Environment -

How to properly use AWS instance storage (ephemeral storage/disk)

If SIOS AMIs are not being deployed on an AWS server, these disk initialization steps must be considered. This was introduced with Windows 2016 and continues in later releases.
- AWS ephemeral storage (Instance Store) must be initialized every time a node starts up from a “cold” shutdown (i.e. having been shut down in the AWS Console or from a CLI “shutdown /s” from Windows – not a reboot).
- Windows 2012 R2 performs this step automatically (Ec2Config service). AWS includes scripts (EC2Launch) to perform this initialization but the script has to be manually scheduled by the user so that it runs at boot time.
- The SIOS AMIs on AWS have this task set up automatically. If you launch your own Windows image in AWS and fail to set the script up to run at boot, and you are using ephemeral storage for bitmaps (as recommended in Best Practices), the bitmap file/volume will be missing and DataKeeper mirrors will not recreate.

SOLUTION

Launch Disk Management and make sure the ephemeral storage is on-line, healthy, and is assigned the volume letter you selected for the ephemeral storage.
Ensure that the drive/drive letter assigned to the ephemeral storage is listed in the registry located at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ExtMirr\Parameters . . . BitmapBaseDir

Note: Apply this workaround to all nodes in the cluster.
LONG TERM SOLUTION
Schedule the EC2Launch script to run at boot time on each system in the cluster to ensure that the ephemeral storage is always available.

- Administrative Processes -

How to change the hostname of an existing license

From the License Support menu select List License
Select the checkbox for the license you want to change (you can only do one license at a time)
Click the Action dropdown and select Repair
Enter the new hostname
Select Repair then select Complete

How to determine what GPOs have been applied to a server

ISSUE:

Group Policy Objects are sometimes deployed or redeployed after a server is rebooted impacting the way the LifeKeeper (GUI) or DataKeeper may or may not perform.

Solution:

There are two methods:

From Start\Run enter rsop.msc

The console will provide a list of policies that have been applied to a server

Or from an elevated command prompt, enter gpresult /Scope Computer /v

This output will display a list of policies that have been applied to a server

Note: To determine what GPOs are being enforced per a user, enter gpresult /Scope user /v

- LifeKeeper GUI -

How to deploy OpenJDK on LifeKeeper

Download OpenJDK, the Windows x64 Installer, from https://openjdk.java.net/
Deploy the software on the Target node first
In LifeKeeper, perform a switchover/In-Service
Repeat the steps above on other Target(s) after the switchovers are completed

Note: This is a Highly Available deployment as you are never installing the software on the Source/Active node.

From a Command Prompt

type javac -version

Your output should be as follows:

“C:\Windows\system32>javac -version javac 15.0.1”

If not, you may need to add the following to your Systems Variable PATH/environment:

Control Panel\System and Security\System\System Properties\
Select the Advance Tab
Select Startup and Recovery
Select Environment Variables…
Select Path and then Edit

Add C:\Program Files\Java\jdk-15.0.1\bin to your string to reflect the following in the example below:

C:\Program Files (x86)\Common
Files\Oracle\Java\javapath;SystemRoot%\system32;%SystemRoot;%SystemRoot%\System32\Wbem;%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;C:\Program
Files\Microsoft SQL Server\Client SDK\ODBC\110\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Server\120\Tools\Binn\;C:\Program Files\Microsoft SQL Server\120\Tools\Binn\;C:\Program Files\Microsoft SQL Server\120\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\120\Tools\Binn\ManagementStudio\;C:\Program
Files (x86)\Microsoft SQL Server\120\DTS\Binn\;C:\Program Files\Java\jdk-15.0.1\bin

Perform the aforementioned on all nodes in your cluster(s).

- DataKeeper -

How to recreate SteelEyeSharedOwner file if it is removed

When using a shared disk, LifeKeeper uses a SteelEyeSharedOwner file to track ownership of a shared device.

To regenerate this file if it gets removed from the system:

Perform one of the following procedures on all nodes that are sharing the volume:

Reboot each node

Take the disk device Offline and then Online on each node, using Disk Management.

- Mirror Solutions -

How to recreate a mirror for an existing clustered DataKeeper Resource

ISSUE:

A customer may have experienced some issues where a DataKeeper Resource is listed as Offline via Windows Server Failover Cluster.

Note: An existing job in the DataKeeper GUI will most likely have a RED X status

Solution:

Remove all of the old mirror remnants from the Source and Target nodes
Manually recreate the mirror via the emcmd . createmirror command

To remove the mirror configuration from both nodes:

cd %extmirrbase%/support (This is a shortcut to the DataKeeper\Support directory)
run “cleanupmirror (drive letter)” Note: Run this command on the Source first. This command performs the following:
- deletes the local mirror only (NO DATA)
- resets the switchover flag
- notifies the driver and service that no mirror is present

To verify that the mirror has been properly removed, execute the following:

cd %extmirrbase% (This is a shortcut to the DataKeeper directory)
run “emcmd . getmirrorvolinfo (drive letter)”
The output should reflect no mirror e.g. 0 servername 0

Now you are in a position to manually recreate the mirror.

The syntax is as follows:

emcmd (Source IP) createmirror (drive letter) (Target IP) A or S (Async mirror/Sync Mirror)
Upon completion, the DataKeeper UI will now reflect a Resyncing status. This can also can be verified from a command line by executing:
emcmd . getmirrorvolinfo e
The output will reflect “E: 1 servername (Target IP) 2, where 2 is a mirror definition, indicating a Resyncing status

Return to Windows Failover Clustering and online the Failed DataKeeper Resource

Deleting mirrorendpoints: Steps to delete mirrorendpoints:

• Go in the registry for the node
• Go to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ExtMirr\Parameters\Jobs
• Find the job for that endpoint and remove the endpoint from that job

Switchover Mirrors, Continue and Lock Mirror, Delete Job (Delete Mirror) Actions Grayed Out In GUI

For Actions that are NOT Available, NOT Bold because the DataKeeper Mirror is now a DataKeeper Volume Resource in Windows Failover Cluster.

When NOT Bold or NOT available as depicted in the screen capture, the end user should be able to “hover” over the Action and it present a description of the Action(s) and alternate choices:

Use the following EMCMD commands via the command line

Switchover Mirrors – to be performed via a “Move” action in Failover Cluster Manager

emcmd . deletejob

Continue and Lock Mirror

Delete Job (Delete Mirror) – Administration must be performed via command line using:

cleanupmirror command or if the Job will be required to be deleted also:
emcmd . deletejob

How to unlock a target/passive node disk that does not release locking the volume

ISSUE:

A customer experienced some issues where a volume was NOT unlocking on a Target/Passive Node

SOLUTION:

Within an Elevated Command Prompt on the Target Node, please perform the following actions:

“cd %extmirrbase%” (This is a shortcut to the DataKeeper directory)
Verify status of the mirror for the volume (i.e. EMCMD <system> GETMIRRORVOLINFO <volume letter>)
1. getmirrorvolinfo (Please see for the varying mirror states)
Pause Mirror for Target Volume (i.e. EMCMD <system> UNLOCKVOLUME <volume letter>)
1. unlockvolume

The actions above allowed you to:

Launch Disk Management
Extend Volume…
Return to the DataKeeper UI; Continue and Lock Mirror

- Error Messages –

Status = 87 after executing “emcmd . updatevolumeinfo <drive letter>”

PROBLEM/ISSUE:
LifeKeeper is using a mirrored drive created via DataKeeper. Mirror is corrupted after the drive was lost. Mirror needs to be recreated.

The following tasks were attempted to correct the error on the offending node:

emcmd . deletelocalmirroronly <drive letter>;
emcmd . clearswitchover <drive letter>;
emcmd . updatevolumeinfo <drive letter>;

After the last command (updatevolumeinfo) you receive the “status=87” error message.

Notable:
A Status = 87 is due to:

The Volume is locked
The partition/Volume is no longer present (i.e. RAW or Unknown state)

RECOMMENDATION/SOLUTION:

Remove the mirror/volume dependency in LifeKeeper first.
As opposed to running the aforementioned commands, executing a “cleanupmirror“ will perform the same steps but bodes a much better result.

Then, using the DataKeeper command line execute these commands on the Source of the mirror first otherwise the Target mirror will continue to get created and/or locked:

Launch an elevated Administrator command prompt
cd %extmirrbase%\support (short to the Support directory)
run “cleanupmirror <drive letter>”

Verify the mirror has been removed by executing the following:

return to the \DataKeeper directory
execute a emcmd . getmirrorvolinfo <drive letter>;

The status should reveal 0 <drive letter>; 0, which would indicate the mirror is no longer present.

Then, repeat these steps again on the Target node(s) .

Verify the mirror has been removed from the Target node(s) by executing, again :

emcmd . getmirrorvolinfo <drive letter>;

The status should reveal 0 <drive letter>; 0, which would indicate the mirror is no longer present.

Result: You should now to able to recreate your mirror using the emcmd . createmirror command, thus returning to your LifeKeeper GUI and adding the Volume Resource/Mirror back into your cluster.

When executing a “emcmd . createmirror”, you receive a Status = 317

ISSUE:

Reasons for a Status = 317

The Target volume size (in bytes) is smaller than the Source when attempting to use the emcmd . createmirror command (See properties of the Volumes).
When attempting to create a mirror using “emcmd . createmirror “ via 2×1, shared buddy configuration, you may receive a Status = 317.

SOLUTION:

There are two methods:

Ensure that the potential Target location is available or eligible for mirroring (make sure that the volume is seen in
- Windows Disk Management (Properties of the disk) – Source/Target SAME size in bytes or the Target is LARGER
- File Manager or Disk Properties, in Disk Management) – Source/Target SAME size in bytes or the Target is LARGER.

For a shared buddy
- On the offending target(s), execute the following command; emcmd . clearblocktarget {drive letter}.
- Then verify that target(s) are set to FALSE by running emcmd . getblocktarget {drive letter}

Note: The bitmask for a “shared buddy” should be set to 384.

This can be verified by executing “emcmd . getconfiguration {drive letter}”.
If not, execute a “emcmd . setconfiguration 384” and execute “emcmd . getconfiguration {drive letter}” to ensure it has been applied.

Additional Assistance: Rebuilding a Mirror

If there is ever a need to rebuild a mirror while maintaining the DataKeeper Job and DataKeeper Volume Resource in the cluster/WSFC/Failover Cluster Manager the steps are as follows:

Always begin from the Source:
1. Launch an elevated Administrator command prompt
2. “cd %extmirrbase%” (This is a shortcut to the \DataKeeper directory)
Next we’ll need to identify the status of the mirror from this node/Source:
1. Type “emcmd . getmirrorvolinfo “ (see the link below for the output this command produces)
2. You can change the “.” to the destination servers, IP, Hostname, FQDN to determine status
Now you need to remove the old mirror remnants:
1. Execute “cd Support”
2. The next command will delete the local mirror only (NO DATA), removes a switchover flag and notifies the SIOS DataKeeper Service & Driver that “No Mirror” is no longer present
3. “cleanupmirror (drive letter)”
4. “cd ..” (return to the DataKeeper directory)
5. “emcmd . getmirrorvolinfo (drive letter)” The output should reflect “0 servername 0”
6. Repeat the aforementioned commands on the Target(s)
Now you are in a position to manually recreate the mirror(s):
1. Launch Failover Cluster Manager to determine what server is the Current Owner of all resources.
  1. This will indicated the node that will be the Source when creating the mirror from the command prompt

From the Source:
1. Execute “emcmd . getjobinfo”
2. Use the output to verify the IP addresses, drive letter(s) and Mirror Type(s) Async (A)/Sync(S) for recreating the mirror

The syntax for creating or recreating a mirror is as follows:

EMCmd <Source IP> CREATEMIRROR <volume letter> <Target IP> <Type> [options]

Example: # emcmd 10.x.x.x createmirror D 10.x.x.x A The completed output should say “Status = 0”.

Return to the DataKeeper UI as you will view the mirror you’ve just created.
1. Any data at this point that needs to be Resynced to the Target will indicate “Resyncing”

Once all Mirror(s) are in a “Mirroring” state, your WSFC roles a eligible for a Move/Switchover.

Commands to Consider:

Click here for access the list of all the emcmd commands

The most frequently use commands are the emcmd . get… commands:

For Mirror Status: emcmd . getmirrorvolinfo emcmd . getserviceinfo emcmd . getjobinfo emcmd . getcompletevolumelist

Additional Information

Security and Group Policy Criteria (accounts, policies, TCP/IP settings, rights, and roles, etc)

DataKeeper requires that “File and Printer Sharing for Microsoft Networks” be enabled on the network interfaces to make a NAMED PIPE connection and be able to run DataKeeper’s command line tool (EMCMD).

To test if you can make a Named Pipe connection, try to map a network drive on the TARGET system. If that fails, you have a Named Pipe issue.
DataKeeper also requires that NetBIOS over TCP/IP and SMB protocols be enabled. If the GUI does not operate correctly, make sure the following network configurations are enabled:

Enable NetBIOS over TCP/IP and SMB protocols as in the following example:

Start -> Control Panel -> Network and Internet -> Network and Sharing Center -> Network Adapter Name (Ex; Ethernet 2) -> Properties -> Internet Protocol Version 4 (TCP/IPv4) -> Properties -> Advanced -> WINS -> Enable NetBIOS over TCP/IP

The DataKeeper GUI uses the server Login ID and Password; therefore, the User Name and Password used to log in to the servers themselves must be the same on each server and must have administrator privileges.
The user must be logged in with domain admin privileges.

DataKeeper Troubleshooting

Video Solutions