June 2010 Archives

The title may be somewhat of a lie, but it's a hell of an accomplishment to have synced that many accounts in a weekend. In total I have 107,000 accounts now synced and syncing future updates live.

I consider this a pretty big milestone, this is a system I have been working on for a very long time. Many 130+ hr/weeks and I had some home issues during the process, but here I am looking at the end of phase one.


MS Dynamics AX: Amazing internal ERP system, I've been programming this application for 5 years and love every minute of it.



Cast Iron Systems: Excellent mechanism for connecting disparent systems and just making complex stuff easy in general.

It's going much better today.

2:1 API calls to records, this is much better and should allow me to stay within safe api call range.

I actually have very little left to do.

1. Finish syncing about 40k records which should take 100k api calls. I currently have about 150k free.

2. Download local db copy and update dynamics ax with the salesforce account id's.

3. Start all orchestrations in castiron to LIVE sync.

4. Start all batches and batch jobs back up

5. Start other AOS'

6. Test

7. Sleep lol

Hopefully the next entry says complete

Wouldn't it be great if things went according to plan. All was running fantastic, better than could be expected or at least it seemed. Then I wake up from a power nap to find 9000 emails confirming what I expected to happen from the beginning, I had used more than the Salesforce api call limit, which they raised to 300k per 24 hours, but was still a fraction of what I needed. So I am dead in the water for 24 hours and will not be able to finish the contact sync. I am going to rescue the account sync by syncing only the remaining delta accounts, around 50k. I made some modifications to cut the api call usage further and now some fun scripting and I know which accounts are the deltas. I will have Dynamics AX dump out the xml files for these accounts shortly and then be ready to sync them as the api calls free up. I will batch them to complete the rest and will still end this process strong.

Assuming I can rescue accounts, I will handle contacts live but syncing batches each day. This will fix all the SFDC contacts, but the ax contacts will need to be synced up and that will be a little more complicated. Both will handle correctly if they are manually edited or used, but I will script the sync of the deltas over a few days.

 

The big question is can it be done in a weekend!!!

I started tinkering and building on this massive monstrosity of a project about 10 months ago. I didn't think it would happen at times, either the company getting sick of waiting, me losing my mind or it just not being feasible. Well here we are ready to release the largest portion of the system, other parts have been running flawlessly for months and one piece is left to be finished. There are about 100,000 accounts and 80,000 contacts to sync in a weekend, friday night to sunday afternoon, leaving enough time to restore backups if there are problems. Fun huh?

I'm going to document the process as much as my insomnia will allow.

1. Servers and systems were backed up and verified.

2. Start a single Microsoft Dynamics AX AOS that normal users do not access to work from and ensure all other AOS's are turned off.

3. Turn off all batch jobs and remove batch servers.

4. To minimize the amount of API calls I am making to salesforce and to increase processing speed I pull a copy of the salesforce tables down to a local MS SQL server and do the mass of the hard processing there, for the initial sync at least.

5. Start the class that processes the Salesforce.com accounts into Dynamics AX, this is a lengthy process and has to be monitiored for random error that occurs when an address is incorrect or the zipcode / counties tables in Dynamics AX are missing a value.

I would receive the same error no matter how the address record is created but in this instance I am creating the address from a webservice I created, that allows some extended functions you do not get with the normal CustCustomerService webservice that comes with Dynamics AX. Since there was no way to update an address of an account in the standard CustCustomerService webservice I had to create a custom one.

AddressService.createList
Line=1, Pos=315, Xpath=/AddressService/DirRel[1]/DirMap[1]/Address[1]
The value 'VENTURA' in field 'County' is not found in relating table 'Counties'.
Error found when validating record.
Creation has been canceled.
Document Address Service could not be created

I don't really do anything at this stage for those records that error. I will catch them after as they will be missing an address record and that is a requirement of the sync process. The process is currently about 1/3rd complete, well the creation of the 38,000 Salesforce.com accounts into Dynamics, it does it direct from the sql db into custom Dynamics AX classes so it is somewhat fast, still takes 4 hours or so.

6. Next I will start on the output of the files from Dynamics AX and syncing to Salesforce.com. This is the full 106,000 accounts and is a very slow process, so maybe I can get some sleep then.

...More updates to come as long as the process goes well. lol