Following on from previous article Citrix Cloud – Cloud Connector in-depth review(including Local Host Cache). which touched on LHC i have been keen to test this functionality as it has been a big concern for enterprises who raised the question around “What happens if the internet is not available?” which LHC for Citrix Cloud clearly addresses.

At the time of writing that article the LHC functionality has not yet reached my citrix cloud setup.It appears mid-march my citrix connector updated and the SQL binaries and database appeared which had not been there originally which indicated to me that LHC has been enabled.

 

OutageModeSupport.IsConnectorOutageModeEnabled – Disable LHC because last Nfuse activity expired after: 3.00:00:00
EntryExit Enter: OutageModeSupport.EnterOutageMode – Transitioning component OutageModeForced has reported a failure with reason = Forced Outage Mode Entry
EntryExit Exit: OutageModeSupport.EnterOutageMode – Cannot enter outage mode due to NoNFuseActivity (undefined)

Validating LHC Is Enabled

To validate this i ran the following powershell command using the XenApp and XenDesktop Remote SDK you can confirm the status of your Citrix Cloud

Get-BrokerSite

This confirmed the LocalHostCacheEnabled was set as True.

Local Host Cache Database

SQL will not appear to be installed via the normal add/remove programs but the binaries will be located in %ProgramFiles%\Microsoft SQL Server\120\LocalDB\

The local host database is stored in “C:\Windows\ServiceProfiles\NetworkService” HaDatabaseName.mdf & HaDatabaseName_log.ldf as shown below.

Force an Outage

To test the local host cache functionality add the following registry key which forces an outage ensuring

HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\DesktopServer\LHC
DWORD : OutageModeForced
VALUE : 1

0 – Takes the Cloud Connector out of outage mode

1 – Put the cloud connector in Outage mode

Running the following in command prompt will add the key and kick off the process of forcing an outage/simulating connectivity issue with Citrix Cloud.

REG ADD HKLM\SOFTWARE\Citrix\DesktopServer\LHC /f /v OutageModeForced /t REG_DWORD /d 1

 

Reviewing the Event logs the below is the sequence of events:-

The Citrix remote broker service will  log Event ID 3001 indicating that an outage has been forced:-

The Citrix High Availability service will  log Event ID 3502 indicating service has now become active and all user requests will be brokered until through the high availability service until the issue has been resolved.

The Citrix CofigSync service will  log Event ID 507 indicating any updated config received will be abandoned until system is returned to normal mode.

This event is only likely created when outage mode is forced using the key and when citrix cloud is accessible via an internet connection as configuration changes would not be normally occur during a real outage

The Citrix remote broker service will continuously  log Event ID 3003 indicating that an outage is still operating (extended outage)

The following video cover adding the registry key, validating outage in Application Log and testing resource enumeration using a locally installed storefront within the same resource location:-

During an outage, if a Cloud Connector is restarted:

• If that Cloud Connector is not the elected primary broker, the restart has no impact.
• If that Cloud Connector is the elected primary broker, a different Cloud Connector is elected, causing VDAs to register. After the restarted Cloud Connector powers on, it automatically takes over brokering, which causes VDAs to register again. In this scenario, performance can be affected during the registrations.

Revert back to Normal Mode

To go back to normal mode add the following registry key which forces an outage ensuring

HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\DesktopServer\LHC
DWORD : OutageModeForced
VALUE : 0

Running the following in command prompt will add the key and kick off the process of returning to normal mode/simulating connectivity restored with Citrix Cloud.

REG ADD HKLM\SOFTWARE\Citrix\DesktopServer\LHC /f /v OutageModeForced /t REG_DWORD /d 0

Once returning to normal mode, this can take up to 10-15 minutes, Event ID 3003 will indicate the registry has been detected and it can take a while for the brokering to be switched back to citrix cloud.

The Citrix High Availability service event ID 3503 is recorded indicating the issue with normal brokering has been resolved and that Citrix High Availability services will stop participating in broker requests.

As per previous event, event ID 2007/2008 for the Citrix High Availability service confirms the XML Services is stopped

And finally the Citrix Remote Broker Provider Service indicate Working Normally which indicates the Outage mode has returned to normal mode.

The following video cover adding the registry key, validating outage is returned to normal mode in Application Log:-

Storefront Bypass Duration

An Issue was identified with the StoreFront server marking the cloud connectors as offline when there was a network interruption. Although the 60 minute default applies if there is multiple cloud connectors added, a single cloud connector will be by default excluded for 5 minutes which means requests will be rejected until this has completed.

As per https://support.citrix.com/article/CTX233424 it is recommend to reduce the bypassduration setting in the storefront servers to 0.

1) Select the store on the StoreFront server

2) Manage delivery controllers

3) Edit Delivery controller

4) Select Settings under the Advanced settings

5)Change the bypass duration to 0

 

Considerations/Performance

RAM

The LocalDB service can use approximately 1.2 GB of RAM (up to 1 GB for the database cache, plus 200 MB for running SQL Server Express LocalDB). The High Availability Service can use up to 1 GB of RAM if an outage lasts for an extended interval with many logons occurring (for example, 12 hours with 10K users). This is additional to the minimum Cloud Connector RAM requirements so 2.2GB should be added to your Cloud Connector VM/Allocated a higher spec Azure VM.

CPU

The number of cores available to the SQL Server Express LocalDB, directly affects Local Host Cache performance. This CPU overhead is observed only during the outage period when the database is unreachable and the High Availability service is active.

LocalDB is limited to the use of a Single Socket and the advise is allocate more cores with single socket as additional sockets/cores will not improve the performance.

Storage

The broker must have sufficient space on the drive where the LocalDB is installed to allow for the database growth during an outage. Citrix has provided an  example, during a logon/logoff test running at 10 logons per second, the database grew by one MB every 2-3 minutes.

Performance considerations

During an outage, one broker handles all the connections. In resource locations that load balance among multiple Cloud Connectors during normal operations, the elected broker might need to handle many more requests than normal during an outage. Therefore, CPU demands are higher. Every broker in the resource location must be able to handle the additional load imposed by LocalDB and all of the affected VDAs, because the broker elected during an outage can change.

Antivirus considerations

As the LHC runs on SQL Server 2014 Express, the normal anti-virus exclusions are recommended. The following sql server data files should be excluded

• *.mdf,
• *.ldf

This will ensure optimal performance is achieved on the LHC cache database.

Internet Considerations

Internet connectivity should never have single point of failure, high availability should be adopted to ensure a route is always available from the data centre/alternative data centre.  Alternatively it should be the highest priority to for your IT department to restore internet connectivity via whatever means are possible.

Software Defined WAN (Such as Netscaler SD-WAN) could be utilised to have multiple paths to the internet over a single logical connection ensuring internet is still available even if a network outage occurs on a single path.

 

I hope you found this article useful and any feedback is greatly appreciated.