[Tolen]

I just upgraded our BES 4.0 to 4.1.4. Since the upgrade, I notice all the BBs are receiving messages in "clumps". It's like the server holds the messages for a few minutes, then you get 4 or 5 at a time. Any idea what this is or how to stop it? I need to receive my tech support tickets in a timely manner.

[hdawg]

Oy, I hate seeing clumping ... most likely messages are coming in every ~20 minutes or so, during mailbox rescans. Are you having any performance problems with Exchange? Network / Disk / Memory / CPU? Any of the task tray popups about Outlook attempting to reconnect to Exchange?

Perhaps BES is being over-taxed also and can't keep up, but generally I see it on the mail server side.

[Tolen]

They're coming every 15 minutes. I haven't noticed any problems with Exchange. Things worked great in 4.0.

Originally Posted by hdawg

Oy, I hate seeing clumping ... most likely messages are coming in every ~20 minutes or so, during mailbox rescans. Are you having any performance problems with Exchange? Network / Disk / Memory / CPU? Any of the task tray popups about Outlook attempting to reconnect to Exchange?

Perhaps BES is being over-taxed also and can't keep up, but generally I see it on the mail server side.

[DarkWater]

Close enough to ~20 minutes to still be rescan.. but they could also be simply delayed.

Logs would be awesome

[hdawg]

Looks like messages are being delivered via rescan and not notification.

I'd recommend firing up the BlackBerry Resource Kit (BRK) tool MessageFlow.exe (if you google BRK BlackBerry you'll get the direct link to download the BRK download) ... make sure you download the 4.1.4 BRK (just released)

Read through the user guide for MessageFlow.exe and how to use / interpret the results. I have a feeling you'll see a significant amount of your messages being delivered via rescan. You can also see where the messages are being held up looking through the timestamps you see in the output.

If messages are being delivered through a discovery method of rescan it means BES isn't getting / processing the UDP notifications that Exchange is supposed to be sending about new mail / items.

let us know the results of this and come back with more questions

[ladydi]

I had that same problem when I upgraded to 4.1. It turned out that my Sysadmin turned on the windows firewall and refused to let me turn it off. So, now I have to look in the firewall log file when I notice mail "clumping" and add the currently being used UDP port to the exceptions list. Yea for security!!

[hdawg]

OMG! Policy impeding business; yuck!

[ladydi]

meh... I like your new monkey, hdawg

[hdawg]

good use of meh; meh. I think I'll go with a new monkey every month ... perhaps keep a history of them and then startup a poll to see which monkey is the best after a year or so

[Tolen]

I went to check the windows firewall on the BES server and the service is actually disabled so I guess that's not it.

Originally Posted by ladydi

I had that same problem when I upgraded to 4.1. It turned out that my Sysadmin turned on the windows firewall and refused to let me turn it off. So, now I have to look in the firewall log file when I notice mail "clumping" and add the currently being used UDP port to the exceptions list. Yea for security!!

[hdawg]

Run these perfmon counters on your Exchange Server(s) polling once a minute for 7 days if you can deal with waiting that long.

For each array you have in the Exchange Server:

PhysicalDisk\Average Disk sec/Read:
PhysicalDisk\Average Disk sec/Write:
PhysicalDisk\Average Disk Queue Length:

For the server as a whole:

Database\Database Page Fault Stalls/sec:
Database\Log Record Stalls/sec:
Database\Log Threads Waiting:
Paging File\% Usage
Memory\Available Mbytes (MB):
Memory\Pages/sec:
Processor\% Processor Time (_Total):
System\Processor Queue Length:
MSExchangeIS\RPC Requests:
MSExchangeIS\RPC Averaged Latency:

and use the attached sheet to see what your server is doing versus what RIM & Microsoft recommend

The attached file is an Excel Spreadsheet; but I renamed it to .doc to attach.

Fire back some questions if you got em.

[Tolen]

I ran the messageflow.exe and the Discovery column only shows rescan.
What should we do?

Originally Posted by hdawg

Looks like messages are being delivered via rescan and not notification.

I'd recommend firing up the BlackBerry Resource Kit (BRK) tool MessageFlow.exe (if you google BRK BlackBerry you'll get the direct link to download the BRK download) ... make sure you download the 4.1.4 BRK (just released)

Read through the user guide for MessageFlow.exe and how to use / interpret the results. I have a feeling you'll see a significant amount of your messages being delivered via rescan. You can also see where the messages are being held up looking through the timestamps you see in the output.

If messages are being delivered through a discovery method of rescan it means BES isn't getting / processing the UDP notifications that Exchange is supposed to be sending about new mail / items.

let us know the results of this and come back with more questions

[Tolen]

Just so I'm on the record with this: It didn't do any of this with 4.0. It delivered messages in a very timely way. Also, Microsoft Exchange engineers went over the server awhile back with some monitoring tools and it seemed fine (Exchange 2003 on a 2003 server).

Originally Posted by hdawg

Looks like messages are being delivered via rescan and not notification.

I'd recommend firing up the BlackBerry Resource Kit (BRK) tool MessageFlow.exe (if you google BRK BlackBerry you'll get the direct link to download the BRK download) ... make sure you download the 4.1.4 BRK (just released)

Read through the user guide for MessageFlow.exe and how to use / interpret the results. I have a feeling you'll see a significant amount of your messages being delivered via rescan. You can also see where the messages are being held up looking through the timestamps you see in the output.

If messages are being delivered through a discovery method of rescan it means BES isn't getting / processing the UDP notifications that Exchange is supposed to be sending about new mail / items.

let us know the results of this and come back with more questions

[hdawg]

You need to look at the performance on your BES and your Exchange server; one of them is the culprit. Try firing up the perfmon counters on your Exchange server for a day or a few hours ... or even fire up the ExBPA and do a performance baseline on your Exchange server to see what is going on.

Also, check to see if there are any hardware errors with any systems; they'll show similar performance symptoms.

[hdawg]

Also, Microsoft Exchange engineers went over the server awhile back with some monitoring tools and it seemed fine (Exchange 2003 on a 2003 server).

Microsoft's analysis was good and valid when they performed it; I'm sure of that ... but I bet mailboxes have grown, you've added some BES users and your environment isn't quite the same.

Take an assessment for what it is worth. A look at the current environment; I always find the best part of any assessment to be the recommendations on what to do going forward. ... well at least thats what I hear back from my customers.

Its entirely possible that something happened within BES during the upgrade, but unfortunately its not a cut and dry thing due to all the components involved.

[hdawg]

Hey, look who just posted a KB article on this sheezy .... Effects of latency on BlackBerry Enterprise Server for Microsoft Exchange

[Tolen]

I just enabled the performance counters using our Network Monitor application and notice one thing immediately, that the BES server could use more memory - only 66MB available and paging file usage at 70%. The Exchange server counters looked alright for now but it's Saturday night.

Originally Posted by hdawg

You need to look at the performance on your BES and your Exchange server; one of them is the culprit. Try firing up the perfmon counters on your Exchange server for a day or a few hours ... or even fire up the ExBPA and do a performance baseline on your Exchange server to see what is going on.

Also, check to see if there are any hardware errors with any systems; they'll show similar performance symptoms.

[hdawg]

How much ram is in the BES as of now? What other applications are running on the BES?

[Tolen]

Ooh, not too much when I look at it. 512MB
That's the only application on this server and we have about 30 users.

Originally Posted by hdawg

How much ram is in the BES as of now? What other applications are running on the BES?

[hdawg]

I would look to add at least another 512 if you can make it happen; I don't recommend less than 2 gig, unless you're not running A/V or any other 3rd party apps on it at all.