I ran into an interesting problem when a company out West purchased emergency service from us regarding their voicemail not being delivered to their inbox. Their environment consists of Exchange 2007 with UM and a Mitel phone system integrated with OCS enterprise voice through and Audiocodes Mediant 1000 gateway. The client had just re-IP-ed the majority of their servers but left the OCS, gateway and Mitel servers untouched. People were able to call and reach their Exchange UM voicemail, leave a message, but the messages are never delivered to their Exchange mailbox. The problem lasted for 3 days before they decided to ask us for help.
Without looking at the environment, I originally thought the problem was DNS not updating but once I logged into the environment after hours, I discovered it was more complicated than that. Issue was resolved within 30 minutes but the report to the customer took a lot longer to type and here it is:
The following outlines what was discovered and the resolution for the issue between Exchange Hub/CAS/Mailbox and Exchange UM.
Problem Description
When calling a into any user’s extension, the caller can leave a voicemail but the voicemail is never delivered to the called user’s mailbox. The issue began after the network readdressing from 172.26.x.x to 10.3.x.x. All DNS records were updated appropriately.
Analysis/Troubleshooting
One of the errors being logged on EXCH-UM (Unified Messaging Server) provides the following information.
Event Type: Error
Event Source: MSExchange Unified Messaging
Event Category: UMService
Event ID: 1082
Date: 6/3/2010
Time: 12:26:25 PM
User: N/A
Computer: EXCH-UM
Description:
The Unified Messaging server was unable to submit messages to a Hub Transport server because there is no Hub Transport server available to process the request with UM header file “C:Program FilesMicrosoftExchange ServerUnifiedMessagingvoicemail 4b3668f-6af8-4da7-aa4f-3272a74051cd.txt”. Make sure that there is a Hub Transport server located in the same Active Directory site as the UM server. In addition, make sure that the Microsoft Exchange Transport service is started on the Hub Transport server.
Another warning being logged on EXCH-02 (Exchange Server) provides the following message:
Event Type: Warning
Event Source: MSExchangeTransport
Event Category: SmtpReceive
Event ID: 1035
Date: 5/31/2010
Time: 12:09:15 AM
User: N/A
Computer: EXCH-02
Description:
Inbound authentication failed with error IllegalMessage for Receive connector Default EXCH-02. The authentication mechanism is ExchangeAuth. The source IP address of the client who tried to authenticate to Microsoft Exchange is [172.26.186.14].
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
The 2 logs provided by The Company appears to suggest that there are communication problems between the hub transport server and the UM server (both separated on different physical servers).
The first error message suggests that there may be an issue with the physical topology of Active Directory. However, a quick review of the physical topology shows that the appropriate subnets (new and old) have been added to the appropriate sites:
I double checked the subnet of 172.26.176.1/20 and it includes a host address range of 172.26.176.1 to 172.26.191.254 which includes the 2 Exchange server’s IP:
|
Server |
Name |
IP |
|
Exchange Hub/CAS/MB |
EXCH-02 |
172.26.186.49 |
|
Exchange UM |
EXCH-UM |
172.26.186.14 |
The second warning entry actually provided a more important clue of what the cause might be since it shows that Exchange UM does indeed start a session with Exchange Hub but the communication between the 2 servers runs into a problem when authenticating.
If we look at the logs on the Exchange UM server, we’ll notice that there’s actually another warning log that has been flooding the event logs since the IP change:
This warning has been logged since 5/30/2010 @ 8:50:44PM.
The details of the error are as follows:
The Unified Messaging server was unable to submit a message to Hub Transport server “EXCH-02” because the following error occurred: Unexpected SMTP server response. Expected: 220, actual: 500, whole response: 500 5.3.3 Unrecognized command.
I did a simple test from Exchange UM to Exchange Hub to see what an EHLO command outputs and here are the results:
From EXCH-UM.domain.local to EXCH-01.domain.local:
220 exch-02.domain.local Microsoft ESMTP MAIL Service ready at Thu, 3 Jun 20
10 16:25:34 -0600
EHLO
250-exch-02.domain.local Hello [172.26.186.14]
250-SIZE
250-PIPELINING
250-DSN
250-ENHANCEDSTATUSCODES
250-AUTH
250-8BITMIME
250-BINARYMIME
250-CHUNKING
250 XEXCH50
An important 250 response required for UM to Hub communication is actually: 250-X-ANONYMOUSTLS.
The next step is to review the receive connectors on the hub transport server (EXCH-02).
There are 3 receive connectors. The only configured connector that will actually allow Exchange UM to connect properly is the “Default EXCH-02” connector. All the other ones do not have the proper settings to allow Exchange UM to establish a connection.
If we review each connector, we will notice that the “Internal EXCH-02” connector actually has specific IP addresses defined while the “Default EXCH-02” connector does not. Since the former has specific IP addresses defined, it will take precedence over the other.
If we look at the authentication and permissions, we’ll notice that while the permissions are set to allow UM connectivity, the Authentication isn’t:
Resolution
Rather than modify The company’s existing connectors, I went ahead and created a new one as shown here that has only one IP defined (EXCH-UM) so the receive connector catches all Exchange UM connections:
After creating new “EXCH-UM to EXCH-02”, a telnet session from Exchange UM to Exchange UM @ port 25 displays the following after an EHLO command:
220 exch-02.domain.local Microsoft ESMTP MAIL Service ready at Thu, 3 Jun 20
10 16:34:20 -0600
EHLO
250-exch-02.domain.local Hello [172.26.186.14]
250-SIZE 10485760
250-PIPELINING
250-DSN
250-ENHANCEDSTATUSCODES
250-STARTTLS
250-X-ANONYMOUSTLS
250-AUTH NTLM
250-X-EXPS GSSAPI NTLM
250-8BITMIME
250-BINARYMIME
250-CHUNKING
250-XEXCH50
250 XRDST
—————————————————————————————————————-
I hope this post helps anyone out there that may experience the same problems as I did.
11 Responses
Thank you very much. This post saved my day. KR, C.A.
Thanks this hlped me immensly.
Thank you! I was having the same issue. I never knew UM used this type of connection.
Solved the problem. Thanks!
Saved my bacon today. you're the man.
Thanks for the tip! In my environment I have a single Exchange 2013 server running all roles. I do have some custom receive connectors so I had to add the local IPs of the server to the "allow remote mail servers". Seems silly but it was necessary for my UM server to send out.
I also meant to indicate in the last post that I did this on the "Default Frontend " receive connector instead of creating a new connector.
One of the good voicemail messages are all well and good but not when you are trying to inject a more personal voicemail message for your clients so they can appreciate the effort you put into helping them out.
Writing is the key to your success in college and at a job. You should not just think of writing as a chore, but a process to convey your thoughts, ideas and visions. Before you even pick up a pen or sit at a keyboard you should try and develop a voice for your paper. professional voicemail messages
Wow, great post.
I've been fighting with this same issue for two days now. I have a similar setup to the Anonymous posting on September 17, 2013 at 12:32 AM, singel EX 2013 multi role server. Kept getting Event ID 142:
The Microsoft Exchange Unified Messaging service on the Mailbox server encountered an error while trying to process the message with header file “C:Program FilesMicrosoftExchange ServerV15UnifiedMessagingvoicemaildd4174e6-d96d-48d2-a0a5-54a2d9805888.txt”. Error details: “Microsoft.Exchange.UM.UMCore.SmtpSubmissionException: Submission to the Hub Transport server failed. The operation will be retried. —> Microsoft.Exchange.Net.ExSmtpClient.UnexpectedSmtpServerResponseException: Unexpected SMTP server response. Expected: 220, actual: 500, whole response: 500 5.3.3 Unrecognized command
Found multiple postings referencing the Default Receive Connectors and the Exchange Server Authentication security needed to get things working. But both the Default and Default From End Receive Connectors had that option checked, allowing all traffic. When the Anonymous posting on September 17, 2013 at 12:32 AM and at at 12:34 AM stated that he had to add the local IP of the Exchange server to the "allow remote mail servers" to the Default Frontend Receive Connector, I figured what the hell. So I added the IP and it allowed the UM voicemails to be processed without issue.
So stupid, but it's sounds like a MIcrosoft thing to do. thanks MS 🙂