Lync Server 2010 or 2013 IM (Instant Messaging) conference no longer works with RCC (Remote Call Control) configured with a PBX

I’ve been called a few times in the past to look at the issue where Lync Server 2010 or 2013 IM (Instant Messaging) conference no longer works when RCC (Remote Call Control) is configured with a PBX and as simple as the root cause is for administrators who have configured RCC before, it may not be the same for ones who are doing it the first time so this post serves to list the symptoms and the solution to the issue.

Problem

You attempt to start a IM conference between 3 participants:

The IM window above has 2 participants (both Terence Luk accounts).  You attempt to invite the 3rd person into the IM chat window:

You see the 3rd participant briefly displayed in the chat windows:

… but quickly disappear leaving the IM:

There are times when you are able to bring the 3rd participant in but you receive the following error when you send a message:

When contacting your support team, reference ID 400 (source ID 239).

Troubleshooting information is available online, including best practices for using Lync.

No one receives the instant message.

Reviewing the application logs on the workstation’s event logs show the following event ID 11 warning logged:

The description for Event ID 11 from source Lync cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.
 
If the event originated on another computer, the display information had to be saved with the event.
 
The following information was included with the event:
 
Lync
80ef0190
RequestUri:   sip:tluk@contoso.com;gruu;opaque=app:conf:chat:id:FS500B1G
From:         sip:tluk@contoso.com;tag=582b69a2de
To:           sip:tluk@contoso.com;gruu;opaque=app:conf:chat:id:FS500B1G;tag=(null)
Call-ID:      a5902bd56aa54dbab02f70021f05257f
Content-type: application/sdp;call-type=im
 
v=0
o=- 0 0 IN IP4 10.40.2.10
s=session
c=IN IP4 10.40.2.10
t=0 0
m=message 5060 sip null
a=accept-types:text/plain multipart/alternative image/gif text/rtf text/html application/x-ms-ink application/ms-imdn+xml text/x-msmsgsinvite
 
 
Response Data:
 
400  Bad request: TR87 protocol violation; required application/csta+xml content missing or invalid
ms-diagnostics-public:  1033;reason=”Previous hop server component did not report diagnostic information”;Domain=”contoso.com”;PeerServer=”aes.contoso.internal”;OriginalPresenceState=”0″;CurrentPresenceState=”0″;MeInsideUser=”Yes”;ConversationInitiatedBy=”0″;SourceNetwork=”0″;RemotePartyCanDoIM=”No”

You proceed to log a snooper session and see the following error logged:

TL_INFO(TF_PROTOCOL) [0]12F8.2FB0::02/25/2013-20:17:11.447.0009ecac (SIPStack,SIPAdminLog::ProtocolRecord::Flush:2387.idx(196))[3778766382] $$begin_recordTrace-Correlation-Id: 3778766382
Instance-Id: 58053
Direction: incoming
Peer: aes.contoso.internal:4723
Message-Type: response
SIP/2.0 400 Bad request: TR87 protocol violation; required application/csta+xml content missing or invalid
Start-Line: SIP/2.0 400 Bad request: TR87 protocol violation; required application/csta+xml content missing or invalid
From: “Luk, Terence” <sip:tluk@contoso.com>;tag=8801af7488;epid=78912b9cdb
To: <sip:tluk@contoso.com;gruu;opaque=app:conf:chat:id:LG7B1YNV>
Call-ID: 5445447d57f84feba63147ce47f325bd
CSeq: 1 INVITE
Via: SIP/2.0/TLS 10.1.1.66:53018;branch=z9hG4bKBE43B1AF.7F7B218E744C2407;branched=FALSE;rport=53018,SIP/2.0/TLS 10.40.2.10:10467;ms-received-port=10467;ms-received-cid=158D00
Content-Length: 0

The error is followed by the following entry:

TL_INFO(TF_DIAG) [0]12F8.2FB0::02/25/2013-20:17:11.447.0009fa7a (SIPStack,SIPAdminLog::WriteDiagnosticEvent:1198.idx(778))[581335822] $$begin_recordSeverity: information
Text: Routed a request on behalf of an application
SIP-Start-Line: ACK sip:svrlyncstd02.contoso.internal:5062;transport=tls SIP/2.0
SIP-Call-ID: 5445447d57f84feba63147ce47f325bd
SIP-CSeq: 1 ACK
Peer: aes.contoso.internal:4723
Data: application=”http://www.microsoft.com/LCS/UserServices”

Solution

As shown in the screenshots and logs above, the Lync Server 2013 environment has an Avaya AES (Application Enablement Services) RCC implementation and what you’ll notice is that as soon as you remove the RCC configuration, IM conferencing would work.  The reason why the RCC integration causes this issue is mainly how the instructions are written in the following guide:

Implementation notes on Integration of Avaya Aura Application Enablement ® Services with Microsoft® Lync® 2010 Server.

Note the $TLSRoute variable configuration shown in the guide:

The problem with these instructions is that the integration guide instructions you to use the *.lync201.local for the MatchUri switch:

By using the *, we’re effectively routing all traffic to the AES server which includes non RCC traffic.  What we want to do is to use a more specific match and in the case of this environment, aes.domain.internal because the server URI for the user is:

sip:aes@aes.domain.internal

With the change to the MatchUri setting, your Get-CsStaticRoutingConfiguration cmdlet should display something similar to the following:

I’m not sure if Avaya has a new guide out but I’ve come across quite a few clients who reached out to me about this issue over the past year so I hope this post will help anyone out there encountering the same issue.