[irn-bru]

Six months ago we had 50 blackberry's today we have 200 and the numbers are set to increase. The problem is BES is killing my Exchange system with RPC calls. Were running Exchange 2003 with 600 users. Exchange performance has gone down hill. During the working day users complain Outlook is slow and users with Blackberrys sometimes stop getting mail.

Today (Sunday) I ran the Microsoft Exchange Troubleshooting assistant and Exmon. The Troubleshooting Assistant is still reported High user RPC activity despite no one being in the office and looking at Exmon all activity is coming from my BES server.

I guess I have two questions.

1. Is there anything I can do to decrease the hit BES is making.
2. Are there any sizing guidlines as to how BES impacts Exchange i.e. if I introduce another 100 Blackberry users how will this impact Exchange and what can I do to my Exchange architecture to improve performance. What are the best practices ?

This has now become a major headache so any help very much appreciated.

[noniman]

Maybe you just need a bigger exchange box or a faster network interconnect between the BES and Exchange servers.

I wish I had a clever answer for you but 600 users & 200 blackerries is a serious work load.

The basic issue as I understand it is that the BES is very agresive in looking for messages to forward, clalendars that changed, etc. That activity must be suported in near real time or the users Outlook experience will be affected as you have noted.

[NJBlackBerry]

Are you running BES and Exchange on the same server (probably not, but it's worth a question)..

If not, are the BES and Exchange servers on the same network? What is the latency between the servers?

[irn-bru]

Exchange is in an Active passive Cluster running on HP DL360G3 with two Intel Xeon 2.80Ghz processors and 2Gb RAM. There are no machine bottlenecks according to the Exchange Troubleshooter although I do see the database volume having to deal with a lot of read requests (as expected with BES). What do you upgrade to if all pointers are that the current hardware is okay.

Exchange and BES are on seperate servers with a Gigabyte backbone connecting them. Ping responce times are less than 1 ms.

I could live with the Blackberrys not operating in real time. Can they be set to replicate on a schedule ?

[john45]

Originally Posted by irn-bru

Exchange is in an Active passive Cluster running on HP DL360G3 with two Intel Xeon 2.80Ghz processors and 2Gb RAM. There are no machine bottlenecks according to the Exchange Troubleshooter although I do see the database volume having to deal with a lot of read requests (as expected with BES). What do you upgrade to if all pointers are that the current hardware is okay.

Exchange and BES are on seperate servers with a Gigabyte backbone connecting them. Ping responce times are less than 1 ms.

I could live with the Blackberrys not operating in real time. Can they be set to replicate on a schedule ?

The first thing I would do is upgrade the memory on the Exchange server. You need at least 4 gigs of ram 8 gigs would be better.

I don't think you have enough memory.

[Blackberry_Jam]

Serveral things you need to be aware of when is comes to performance.

1) As a rule, I use the fact that 1 BES v4 user is equivilant to approx 4 Outlook exchange clients (some sources quote 1:8), The fact that you have 200 BES users could equate to 800 exchange users. so you are looking to spec the exchange server to account for approx 1200 users.

2) Network connections also play a big part, you would be looking for low latency times between BES - Exchange and BES - SQL (if running on a separate DB)

3) The fact that you are running an Exchange cluster, you may want to analyse the performance between the exchange server and disk array, this can affect the performace of the cluster, especially if you take into account the client load that the BES puts on the server.

4)There is also a known memory leak issue with 3rd party mapi call apllications and exchnage which causes the BES server to nose dive in memory performance. look on the RIM support site and search on Memory Leak

[jglass38]

Also make sure you are following the Exchange best practices document for partitioning drives and locating your DBs and Logs on separate partitions (preferably separate spindles as well). I ran into the same problem with Exchange although in a smaller environment (single exchange, single BES, 150 Exchange mailboxes, 30 handhelds, 65GB IS). The engineer before me had everything running on a single partition on the Exchange Server. Once I rebuilt it following best practices, the RPC load dropped way down.

[Cushwake]

Originally Posted by Blackberry_Jam

Serveral things you need to be aware of when is comes to performance.

1) As a rule, I use the fact that 1 BES v4 user is equivilant to approx 4 Outlook exchange clients (some sources quote 1:8), The fact that you have 200 BES users could equate to 800 exchange users. so you are looking to spec the exchange server to account for approx 1200 users.

This was a crutial rule we used to size our exchange servers.

Its NOT hearsay!

We worked on the assumption that due to polling by the BES we should expect 1:5 ratio.

1 blackberry user is equivalent to 5 normal exchange users.

Also have a look at this Use of the /3GB switch in Exchange Server 2003 on a Windows Server 2003-based system it may help.

[Logvin]

What is the "Message State Database Size" set to on your BB? Default is 100, max is 1000. This sets the size of the message state databse in the messages per user. The higher the value, the less load on the Exchange server, the more load on the memory on the BES.

At my work, I dont control the Exchange Servers, i control the BES.. I always have mine set at 1000.

I have 3gig of ram and have over 700 users on this server. This could very easily help reduce the workload on exchange.

[Kul]

On BES 4.0 SP4 where or how can I set/view the "Message State Database Size" ?

[mendes9]

I've only seen this issue with high latency times..

[Highfall]

I am the Exchange and BES Admin at my work. Have been doing Exchange for 9 years now and have seen it all. I am running a setup with 9000 Exchange users and around 600 BES users. The best advice some gave and more can be summed up below:

Exchange - 4 Gigs of memory on the server. Exchange 2000/2003 does not recognize PAE which is needed for applications to use more than 4 Gigs in a 32bit Windows OS. Between the 32 bit OS and Exchange not using PAE, anything above 4 is a waste of money and should be put towards disks. Spindles, spindles, spindles. It's all about disk I/O and Virtual Memory Fragmentation with Exchange. Increasing the number of disks is the biggest performance boost you will get after putting 4 Gigs of memory in and is where most people miss the mark. Go to Exchange 2003 SP2, wherever possible as it fixes almost all of the Virtual Memory Fragmentation issues. Seperate the databases from the TLogs on seperate disk groups so that you seperate the different types of writes/reads. Although on a decent SAN box with intelligence, this is not nearly as critical until you start maxing your system out I/O wise. By all means, never put SQL on the same box as Exchange as both love memory and if you again use a SAN, make sure that you do not share it with other applications like SQL, etc. Exchange loves I/Os to the point that it will starve your other application out. After that, you will see that with the right hardware the number of users you can put on a box is what MS recommends, which is 4000. MAPI and memory limitations play a huge part in this. If they are BES users, even less users per box per the 1:4 average ratio (power users could make that worse as someone said). After that point, you are putting in more boxes.

BES - Learning this one still to some point. 4 Gigs again, 2 Gigs if less than 500 users. Keep everything on the same local LAN when possible. If more than 100 users, move it to a SQL database on a seperate box if possible. This also helps you if your BES box goes south due to major hardware issues. With 4.1 (and maybe 4.0?) your database is gold. The rest can be rebuilt. If you must seperate due to a WAN link to a few users on a remote Exchange box, make sure you put them on a seperate Messaging Agent to avoid hung threads from impacting a larger user base. 4.1 has this ability and not sure about other versions. With AV on the BES, make sure you exclude the BES processes. Disable DEP on the box if you are able as I have seen it mess up the Agent service sometimes, although rare. Some make a great point as to DR and recoverability when you take number of users into account, so I would suggest you weight that out based on what your service level agreement is with your business. I was going to put 1500 users per box but may put only 1000 due to some things I have heard. Messaging Agent restarts will also take a hit as the data must be queued up for everyone on an Agent before it delivers the email. Up your Message State Database Size (4.1) when possible to decrease some load on Exchange, although this does not help too much if you are constantly restarting the Agents for some reason. I am starting to see where it is important to build your BES to the point that you rarely have to restart the Agent or box where possible.

Hope putting a lot of info in one spot helps! I love BES, but man is it a Exchange performace tester...

[mendes9]

Listen to HighFall he knows what's talking about. I'm in a 85,000 mailbox global exchange environment with 2500 BB users. We run typically 800users per exchange box, and roughtly 500users per BES, no performance issues on Exchange. There are lots of tweaks for exchange ...

[auslander]

...wondering if this thread could be moved to the BESx forum and "stickied" as it contains some good, basic info and specific gotchas to watch for

Thanks,

aus

[y115261]

HI
I'm running with BES 4.1 with 900 + users on exchange 2003 environment
last week i had weired issue where BES and exchange was working fine
but mails blackberry handheld users not able to send / receive mails
when i tested BES server for network , database and other connectivity
all results are passed !!
i left with no option other then calling vendor for support , vendor asked me
give reboot of server !! i gave restart now users able to send / receive mails on handheld
i have no clue what made it to work
any body have any clue
awaiting reply
Regards
Raj

[PeterJS]

I don't think Raj's question belongs here, but as an answer, last week, Microsoft released a couple of patches which didn't play all that well with BES. Restart fixed the issue. Make sure you have your BES set to download the MS patches, but not to install and restart automatically until you can find out more about what the patches are going to do.

[Highfall]

Do you have the KB article for the patches or do you know the patch numbers?

[PeterJS]

Certainly:

MS07-034: Cumulative security update for Outlook Express and for Windows Mail
MS07-033: Cumulative Security Update for Internet Explorer
MS07-035: Vulnerability in the Win32 API could allow remote code execution
MS07-031: Vulnerability in Schannel could allow remote code execution

I found that most often (but not always) any MS patch that has anything to do with mail-related items as these do, usually requires a BES restart.