[Hatchy]

Here's a scenario I want to run by anyone who's familiar with BES DR planning...

There are 2 BES' running concurrently (BES for Domino version 4.0, HF4, Domino version 6.5.4, Windows 2003). Each BES has its own SRP. BES1 is the Production BES, BES2 is the failover BES. All the users (50) are on BES1, 0 users on BES2. BES2 is pointing to the configuration database on BES1 (MSDE) using SQL Authentication. The MSDE database is being backed-up on schedule. The State databases on BES1 are replicating to BES2 (15 minute replication interval). This environment is all fine, and up and running. User moves between the two servers works fine.

Should BES1 go down, we simply restore the MSDE database on BES2, run the BES Server Configuration and point BES3 to the now LOCAL database (using Windows Trusted authentication). Then, through the BlackBerry Manager we can move users from BES1 (down) to BES2. There would be a very small window of outtage for the users, and redirection would continue without user interaction (new service books sent to the users when they are moved). Because the State databases are already on BES2, users would have the ability to forward and reply-with-history to messages already on their device.

Is this an effective Failover/DR architecture for BES?

Thanks!

[x14]

It sounds good but it depends on your company's acceptable downtime. Our max downtime is 30 minutes.

Instead of using Domino to replicate the state databases we use a data replication product. Data replication at the file level keeps the state databases synchronized to the second.

So our setup is. BES and Domino is installed on Server1 and Server2. BES and Domino is up and running on Server1 while down on Server2. Data is replicated from Server1 to Server2 at the file system level. When Server1 goes down all we have to do is start BES and Domino on Server2 and everything continue when it last left off.

[earth]

we have a similar setup here. our production BES is ghosted and mirrored on a backup server. as soon as the 1st one goes down, the 2nd one will connect and continue as the failover.

[Hatchy]

Thanks for the replies. I'm starting to like the idea of one SRP, and replicating Profile AND state databases between the 2 servers....

- 2 Domino servers running concurrently (Different Domino Server names, obviously)
- BES sw (identical versions) installed on BOTH of them with the same SRP (Obviously the BES Task only running on one server at a time - ServerA)
- Profiles and State databases replicating between the two Domino servers
- ALL BES Services running on both servers concurrently
- MSDE database on ServerA
- Both Domino servers point to the same MSDE database on ServerA
- ALL users on BESA (BES Task running), No users on BESB (BES Task not running)

If BESA goes down:

- restore MSDE database from backup to ServerB
- Point ServerB to local MSDE database
- MOVE users from BESA to BESB using the BlackBerry Manager on BESB
- Start the BES task on BESB

Voila. All users are back up and running, no new service books to send to the devices (same SRP).

[x14]

Originally Posted by Hatchy

Thanks for the replies. I'm starting to like the idea of one SRP, and replicating Profile AND state databases between the 2 servers....

- 2 Domino servers running concurrently (Different Domino Server names, obviously)
- BES sw (identical versions) installed on BOTH of them with the same SRP (Obviously the BES Task only running on one server at a time - ServerA)
- Profiles and State databases replicating between the two Domino servers
- ALL BES Services running on both servers concurrently
- MSDE database on ServerA
- Both Domino servers point to the same MSDE database on ServerA
- ALL users on BESA (BES Task running), No users on BESB (BES Task not running)

If BESA goes down:

- restore MSDE database from backup to ServerB
- Point ServerB to local MSDE database
- MOVE users from BESA to BESB using the BlackBerry Manager on BESB
- Start the BES task on BESB

Voila. All users are back up and running, no new service books to send to the devices (same SRP).

There's 1 issue with this scenario. The problem is have 2 Domino server.

If you move all the users to BESB and the user run's the BlackBerry Desktop Manager it's going to look for BESA. The user would have to change it to BESB and back when BESA is up again. Of course you can tell the user not to use the desktop software when BESA is down.

[jibi]

Just to note, BES 5.0 will include built-in HA/DR/FO capabilities. Not really sure on all of the details, but I've heard it was quite nifty on the implementation. Just food for thought.

[jmanford]

Originally Posted by jibi

built-in HA/DR/FO capabilities.

Not familiar with these abreviations. Please explain..

[jmanford]

We run BES on Microsoft Virtual Server. We have VS running on two separate servers. BES goes down, restore BES VHD and VMC files to other VS and were backup running.

[dev]

[QUOTE=Hatchy]Thanks for the replies. I'm starting to like the idea of one SRP, and replicating Profile AND state databases between the 2 servers....

Ensure that you setup one-way replication for the Profiles db. That means from the active do a "send only" to the standby Profiles db. The active Profiles db gets easily corrupted when there's another agent (replication agent) actively writing to it simultaneously as the BES message agent. Once corrupted it generates errors in the notes log which directly affects mail redirection. In short all mail redirection stops and Pending mail increases for all users. I experienced it when clustering the Profiles db in a DR scenario as well as when setting up full replication.

[Hatchy]

Originally Posted by x14

There's 1 issue with this scenario. The problem is have 2 Domino server.

If you move all the users to BESB and the user run's the BlackBerry Desktop Manager it's going to look for BESA. The user would have to change it to BESB and back when BESA is up again. Of course you can tell the user not to use the desktop software when BESA is down.

Agreed, but it won't be a problem, noone is using the Desktop Manager.

Some of the Carrier support people seem to think that if a BES is pointing to a remote MSDE database, the configuration won't be supported. When I speak with RIM, they say it absolutely is. Very puzzling?

[jibi]

Originally Posted by jmanford

Not familiar with these abreviations. Please explain..

high availability, disaster recovery, failover.

[siemp]

I´m trying to figure out a high availability for blackberry as well.
Is the next solution a workable situation?

BESA with x users SRP 1

BESB with y users SRP 2

all state db´s on both domino servers.

remote sql db

If one server fails I move all users from that server to the working server.
Any comments?
Is it the same as the first post?
Is the move of users an easy step?

[Hatchy]

Siem, I think that that is the ideal way to do it (2 BES' running concurrently with their own SRP, sharing the same remote SQL database).

Similar to the first post, except I was using a local MSDE database on one of the servers, not a good as your architecture.

Moving users is a very simple process - you can drag and drop them. New service books will get sent to the devices (assuming coverage, if not, they are queued) and they'll become active. The State database which already resides on the server will be used (this is a replica of their State db from the other server). A new profile document will get created in Profiles.nsf.

You could easily test a user move (yourself). You'll see **Activation entries in the Notes log which confirm a successful activation on the new server. Because you have replicas of the State dbs, they'll be used. This allows you to reply-with-history and forward existing items on your device. If you didn't have a replica already, you could do the move, but you would lose this functionality (no reference to existing messages).

Regards,

[amukhey]

Also, you are missing a few things:

--When migrating from Production to DR server, turn OFF blackberry services on Production server / turn BB windows svcs to manual and reboot the server/ shut down the server. If you dont, your SRP will be locked out based on the 5-1 rule.

--If using a sql server, ensure your server is in the AD group for accessing the sql database, besmgmt. If you dont, you will see authentication failure's on the Domino console.

--Registry key (Check Admin guide and BB KB) to enter user name and password for SQL Authentication.

--Do a DNS IP Change for Primary to DR BlackBerry server. -This means you have to shut down Primary server to avoid a conflict.

--Do a DNS IP Change for SQL Server.-This means you have to shut down Primary server to avoid a conflict.

There is more detail to this but this fail over scenario also depends on your Cluster Architecture and replication.

We recently did a cut over of all applications and servers to our DR site since our NY building was being shut down for maintenance (every 3 years).

[Sagz]

You can also do this:

Just make a bat file with this, you can also automate this so it checks using monitoring. This is a setup for SQL environment, I did this along time ago and figured I would share it again. As for the State DB's just do data copies over. You can also have the 2 domino servers replicate, and just have an agent create new reps as they get created on the primary. If the data is out of date (restore of stat db for example) the user may receive some messages again.



This handles failure besides your SQL db. So for example you can do maint without effecting the userbase. As for the SQL db the only real way is to restore to a different server and point the BES boxes to the new host, which can be batched..

Enjoy.
(This is 4.0 environment there are some tweaks you need to do for a 4.1 environment)

c:
cd\temp
REM ************************************** Stop Blueberry Services
psservice \\<servername>-u <username> -p <password> stop "Blackberry Alert"
psservice \\<servername>-u <username> -p <password>stop "Blackberry Attachment Service"
psservice \\<servername>-u <username> -p <password> stop "Blackberry Controller"
psservice \\<servername>-u <username> -p <password> stop"Blackberry Mobile Data Service"
psservice \\<servername>-u <username> -p <password> stop "Blackberry Policy Service"
psservice \\<servername>-u <username> -p <password> stop "Blackberry Dispatcher"
psservice \\<servername>-u <username> -p <password> stop "Blackberry Router"
psservice \\<servername>-u <username> -p <password> stop "Blackberry Synchronization Service"

Sleep 5

OSQL -E -i c:\temp\Raspberry.sql

sleep 3


c:
cd\temp
REM ************************************** Start Raspberry Services
psservice psservice \\<servername>-u <username> -p <password> start "Blackberry Alert"
psservice \\<servername>-u <username> -p <password> start "Blackberry Attachment Service"
psservice \\<servername>-u <username> -p <password> start "Blackberry Controller"
psservice \\<servername>-u <username> -p <password> start "Blackberry Mobile Data Service"
psservice \\<servername>-u <username> -p <password> start "Blackberry Policy Service"
psservice \\<servername>-u <username> -p <password> start "Blackberry Dispatcher"
psservice \\<servername>-u <username> -p <password> start "Blackberry Router"
psservice \\<servername>-u <username> -p <password> start "Blackberry Synchronization Service"


The SQL script is as follows.

use BESMgmt
go
update ServerConfig set ServiceName = 'CN=Raspberry/OU=SRV/O=LAB where ID=1
update ServerConfig set MachineName = '<machinename.something.yada>' where ID=1
update MDSConfig set MDSHost = '<machinename.something.yada>' where ID=1
go

[dropscience]

Reading all this information makes me feel a bit out of the league amongst you experts. I don't have a DB background, nor do I really understand the logistics so I hope you guys can bare with me. We are going to add approximately 120 more users onto the BES but before I proceed I wanted to run some ideas with you guys.

Our scenario is this:

BES A (v4.0 w/SRP 1): 20 users
BES B (v4.1 w/SRP 2): ZERO users

The configuration database sits locally on BES A. I want to move it off onto a DBA managed "SQL farm" (is that the correct terminology?). This allows me to point both BES servers to the same remote config database.

I've been given direction to place both BES servers onto a VMWare environment. These are the following steps I think are correct:

1) Copy existing Configuration Database onto remote location.
2) Point BES A to the new remote location.
3) Install BES B (v4.1 w/SRP2) onto VMWare environment and also point it to the new remote location.

At this point, both BES servers are now pointing to the same configuration database which should allow me to move users freely back and forth between the two servers. Correct?

4) Move users from BES A to BES B.
5) Add remaining 120 users onto BES B.
6) Copy BES A onto VMWare environment.
7) Upgrade BES A to BES 4.1 (it's currently 4.0)
8) Use newly upgraded BES A (now 4.1) as my test BES and DR server.

Is this sound methodology? Also, forgive my ignorance but is "state database" the same as "configuration database"?

Thanks in advance for any suggestions and help!

[noname]

Hey dropscience,

You may like to take note that VMWare ESX Server version 2.5.2 or later is supported with BlackBerry Enterprise Server software version 4.1 Service Pack 2 or later only. (Source: KB04405 "Virtual Machine software is not supported for BlackBerry Enterprise Server installation")

For the cutover of the BESMgmt database local to BESA to a dedicated remote SQL server, you may like to refer to RIM's cutover article. (Source: KB-04261 "How To - Perform a cutover to another BlackBerry Enterprise Server ")

When you have point BESB to the BESMgmt database on this remote SQL server during installation, it will upgrade the database schema to that of version 4.1. All installs of BlackBerry Manager used to administer these 2 servers MUST BE version 4.1. (Source: KB-04505 "What Is - BlackBerry Manager does not function in mixed version domain")

One thing to remember is that if you have move users from BESA (4.0) to BESB (4.1), their SQL records will be stored in version 4.1 data structure. You must take care not to move a 4.1 user back to a 4.0 server as there are some known issues that email or PIM sync will stop after this kind of move.

Lastly, state databases are Lotus Notes database found under "\Domino\data\BES\state" folder. It is different from the SQL config database (BESMgmt). State database serves as a storage for the links that describes the state of data between BES and the user's mail database. (Source: KB-01477 "What Is - BlackBerry state database")

Once BESB is completely setup with all the users moved, you may like to setup a one-way replication for the "BES" folder (BESB -> BESA) to keep these data up to date.

[mattlane]

Interesting that BES is not supported on virtual server, As I know that RIM run there bes servers on virtual servers with no issues.

As for DR if you build a second bes server with the exact same SRP key have them point to the remote database. When one server fails. shut it down. physically pull the network cable out of the back of the server plug it back into your exact replica power back up.

****This way is the quickest and you have not got to move any users, but as someone has already said if you have both SRP keys up at the same time. The SRP key would be diabled hence no mail. ***** A

*****Correct me if I am wrong but Just to let you know for those that move users, when you move users from one bes server to another if they have there blackberry turned off. It will only start delivering mails when it gets associated with the new bes server. So if you move a user. The user turns blackberry his off for a week. he turns his device on. It gets associated to the new bes server. New mail is delivered to that device all mail that was delivered during the week the user is off will never be delivered.

[dropscience]

This is why I visit this forum so often... for help and posts like the one by noname. THANK YOU! I read a few of the KB articles lastnight after my post and it's very reassuring to have some of my thoughts backed up with your knowledge!

On another note, the amount of BES users I have on BES A equates to no more than 20. I'm thinking it would be probably be easier to just do a knife-edge cutover to BES B when it's up. They're all aware that the BES connectivity is on a test basis and the amount of users might not justify the backup of the database.

What do you guys think?