Nasty SharePoint Workflow Bug

First off, thanks to my friend Kirk Allen Evans for his help with this problem on a Saturday! Microsoft is lucky to have him. In the process he created a nice screen cast that you can watch to set the background. He doesn’t encounter the bug, but as it is a threading issue he might have simply been lucky (or me unlucky threading bugs are like that), or (I’m hoping) he has a hotfix that I lack.

In some situations, a workflow will enter into a non-error, but unrecoverable state that looks as if an OnTaskCreated workflow activity did not fire even though the task exists. Sample code that reproduces the problem in my environment on both MOSS SP1 And MOSS SP1 + Infrastructure update is located here. If the problem occurs, the workflow will not continue processing beyond the OnTaskCreated activity. You can see reports of this issue as experienced by others here. Scroll down to the bottom of the post for a workaround.

I have a moderately complex workflow than starts three different tasks. A simplified version is shown below.

Intermittently, the workflow will stop working with history that makes it look like the workflow failed to handle one or more OnTaskCreated activities as shown below.

Notice that there is no entry for Task 1 Created (nor will there ever be). This workflow is toast and will not complete.

It turns out that the problem is actually in the workflow manager. If you step through the workflow, you will see every onTaskCreated activity light up in the designer, but the runtime will not invoke the method. If you look at the error log, you will see two exceptions:

RunWorkflow: System.ArgumentException: Item has already been added. Key in dictionary: ‘c6923528-0908-4d75-86ed-f58342ea507e’ Key being added: ‘c6923528-0908-4d75-86ed-f58342ea507e’ at System.Collections.Hashtable.Insert(Object key, Object nvalue, Boolean add) at System.Collections.Hashtable.Add(Object key, Object value) at System.Collections.Hashtable.SyncHashtable.Add(Object key, Object value) at Microsoft.SharePoint.Workflow.SPWorkflowManager.TrackCreatedInstance(Guid trackingId, SPWorkflow workflow) at Microsoft.SharePoint.Workflow.SPWorkflowManager.RunWorkflowElev(SPWorkflow originalWorkflow, SPWorkflow workflow, Collection`1 events, SPRunWorkflowOptions runOptions)   

RunWorkflow: System.Workflow.Activities.EventDeliveryFailedException: Event "OnTaskCreated" on interface type "Microsoft.SharePoint.Workflow.ITaskService" for instance id "c6923528-0908-4d75-86ed-f58342ea507e" cannot be delivered. —> System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.SharePoint.Workflow.SPWorkflowHostServiceBase.LoadInstanceData(Guid instanceId, Boolean& compressedData) at Microsoft.SharePoint.Workflow.SPWinOePersistenceService.LoadWorkflowInstanceState(Guid instanceId) at System.Workflow.Runtime.WorkflowRuntime.InitializeExecutor(Guid instanceId, CreationContext context, WorkflowExecutor executor, WorkflowInstance workflowInstance) at System.Workflow.Runtime.WorkflowRuntime.Load(Guid key, CreationCont…   

Apparently, there is a threading issue with the Workflow manager and when it happens, the workflow receives notice that the event is firing, but it can’t load the instance data.

I’m hoping that this is a known issue and that there is a hotfix available. I’ll follow up as I learn more.

In the meantime, you can work around this problem by simply avoiding OnTaskCreated. The requirement I have that led me down this path is to set up permissions on the tasks beyond what the SpecialPermissions property allows (more on that some other day). I will meet that requirement by using a list event handler and apply the permissions outside the workflow via the handler.

Good luck!

–Doug Ware

Atlanta based entrepreneur, author of many SharePoint books and videos, leader of Atlanta .NET user group, founder of InstantQuick, and SharePoint MVP.

No Comments

  • Todd Kitta

    October 9, 2008, 8:57 am

    I hit this issue on every WF project I work on and I cannot believe MS has not address this – extremely disappointing. You can actually use a delay activity to force the workflow to go to sleep and have the task be created. Nice huh?

  • Anon

    December 25, 2008, 3:27 am

    Nice post !

    Thanks for the hard work and the valuable information for sharing with us.

  • Adam Drewes

    January 7, 2009, 4:35 pm

    Is this something to worry about only in conjunction with the Parallel Activity or have you seen this on workflows with just one CreateTask + OnTaskCreated?

    Great post and thanks in advance for any additional information.

  • Doug Ware

    January 7, 2009, 5:11 pm

    I’ve only seen this happen with the Parallel Activity. I haven’t tested it yet, but I am hoping that the December update included a fix.

  • Andrej

    January 12, 2009, 3:31 am

    I face the same issue in WF this replicator activity when running threads in parallel. And in case of sequential run everything works perfect.

    I have August and October hotfixes installed. Does anyone know if December update really fixes the bug?

  • dfm

    March 11, 2009, 10:33 am

    Hello,

    I have the same error but with the OnTaskDeleted, although I never delete the task, I just try to complete it.

    Has anyone found a solution for this?

    My problem is described at http://social.technet.microsoft.com/Forums/en-US/sharepointworkflow/thread/3d8a8af9-a62d-4c38-8118-b2c2ac09cf1f

    Regards,
    dfm

  • Pedro Pablo

    May 11, 2009, 2:33 pm

    I found this real sad answer from Microsoft:

    http://support.microsoft.com/kb/970548

    OK, I can do all that, but I’m complicating my solution adding unnecessary components just because MS don’t want (or can’t) fix their bugs…

    clever guys, they didn’t use any of this buggy Activities in their ECM Examples

  • MCox13579

    July 30, 2009, 7:08 pm

    I hit this same problem (parallel activity in a task replicator) and removed my OnTaskCreated activity, noticing that when the task is created, the OnTaskChanged activity gets the event and can figure out what to do. Now, however, the same error is happening with OnTaskChanged – - intermittently. I’m really stuck. I need the events from the task list or else why bother using workflow?

    Engine RunWorkflow: System.Workflow.Activities.EventDeliveryFailedException: Event “OnTaskChanged” on interface type “Microsoft.SharePoint.Workflow.ITaskService” for instance id “c4b16af0-2b5b-4c67-aba5-3a7b0b82d952″ cannot be delivered. —> System.NullReferenceException: Object reference not set to an instance of an object

  • Rodrigo de Castro Reis

    August 19, 2009, 5:20 pm

    Guys,

    I faced the same problem in my workflow and I have an open case in microsoft.
    But they don´t want to assume that´s a bug.
    I encourage you to open a case. That way we have more power and maybe they fix the bug.

    Let´s keep contact.

    Rodrigo de Castro Reis
    rodrigo.reis@invit.com.br

  • Rick

    September 24, 2009, 4:28 pm

    … and when the task gets locked, the workflow is stuck, since the only thing to move it forward is the OnTaskChanged event.

    If MS is able to fix it, please post.

    -Rick

  • Doug Ware

    September 24, 2009, 4:53 pm

    I have never had that issue an I have built many, many workflows. What I have seen happen is an issue that occurs if any changes are made to a workflow task while it is not complete that do not correspond to an OnTaskChangedActivity (or correspond to an OnTaskChangedActivity that is incorrectly configured) causing the task to lock and preventing subsequent handlers from receiving events.

    The difference is pretty subtle, but the fault in these instances has always been a defect of my own creation. If you send me an example of what you are seeing, i’d be happy to look at it when i get a chance.

  • Rick

    September 28, 2009, 12:16 pm

    Could you give an example of a way to make changes to a task “while it is not complete that do not correspond to an OnTaskChangedActivity?” I have not built many workflows, and in fact I inherited the WF that I’m struggling with from another developer in my company (who has since… left.).

    What I have done is double-checked that every CorrelationToken is properly set up and scoped to the correct OwnerActivityName and that every TaskId is generated at runtime in the MethodInvoking event of the CreateTask activity.

    The issue is intermittent, which tells me that it is unlikely to be due to a bad configuration, but of course I haven’t found any other solution so I’m open to anything. When it happens, the Version of the Task gets set to 2.0, and no one can edit the task.

    Any tips would be greatly appreciated.

  • ERNEST

    October 11, 2009, 5:04 pm

    MY WORKFLOW IS VERY LARGE AND COMPLEX, THE ISSUE APPEAR RANDOM IN AND OTHER TASKS, BUT TO SPEND SOME TIME IT IS UNLOCKED.

  • Paulo Pozeti

    November 9, 2009, 7:55 am

    Hi!

    I got the same problem and spend a lot of hours trying to solve it.
    Apparently I solved it using Thread.Sleep in a Code Activity after the CreateTask Activity.

    No errors until now.

    Hope it help you guys.

  • Hamed bagherzadeh

    November 16, 2009, 2:36 am

    Hi

    I have same problem i can solve this problem by removing ontaskcreated Activity from All steps

    when I see log file after that tasks created i find this error in log file

    “Engine RunWorkflow: System.Workflow.Activities.EventDeliveryFailedException: Event “OnTaskCreated” on interface type “Microsoft.SharePoint.Workflow.ITaskService” for instance id “b7d546ce-2665-4828-8b7f-3496f65a7482″ cannot be delivered. —> System.NullReferenceException: Object reference not set to an instance of an object. … ”

    so I remove onTaskCreated Activity from all steps in workflow.
    now my problem solved and workflow is work properly

  • F00z

    January 21, 2010, 3:46 pm

    I ran into the same issue where i had to save the properties of the created task to another list. Basically i had to use TaskItemId which would always be -1 unless i used OnTaskCreated. After days of searching i found out that i can bind ListItemID to a property and check the property for my logic without using OnTaskCreated

  • Poonam Thawani

    February 2, 2010, 12:19 pm

    I am using the same scenario Parallel activity with multiple task creation activities as described in this blog.

    On Task changed event I am checking if workflow version !=1, i.e. if workflow is locked then i update the workflow version =1 so that I don’t run into workflow lock issue.

    My task gets updated correctly but I get “Task Update was not accepted” screen;(

    Any idea?? Why this error?

  • Min

    March 27, 2010, 1:03 pm

    Hi Rick, I got the exactly same problem as you did. the Ontaskcreated event was not fired, the issue is also intermittent. If the workflow stucked, I deleted it and started the workflow again, it then went through fine. But it’s not acceptable at all.I’m pulling my hair out…. Please let me know if you get any idea how to solve the issue.Thanks a lot!!!!!!!

  • Anthony

    July 9, 2012, 4:11 pm

    Hello all, I’m experienceing the same issue.

    The createTask activity fires and the task is created but the onTaskCreated event never fires and I get this error: “Item has already been added. key dictionary: GUID”

    scenario: multiple part approval process all in sequence (no parallel tasks)

    Each list item goes through the following once sumitted:

    1) task is created for approver

    2) onTaskCreated (for after properties)

    3) e-mail with link to approval form is sent out

    4) approvers approve request

    up to here this process works successfully for about 5 approvers.

    4a) just before the last approver is sent their task e-mail, another e-mail is sent updating originator of current status in the process. (Approved / Waiting for purchase)

    5) final approver approves request

    6) final e-mail with purchase approval is sent to originator.

    I did notice that if I disable the e-mail activity 4a) the workflow proceeds perfectly to the last reviewer.

    Steps Taken:

    1) Updated Sharepoint tothe most recent CU

    2) I’ve checked and rechecked my tokens and there is only one workflowToken defined and the e-mail activity is using it. Each task has it’ own token.

    3) Updated Item.Update() with Item.SystemUpdate()

    4) Blew away the sendEmail Activity and re-created.

    Nothing seems to fix this. If anybody has any other suggestions please let me know!

    Thank you

  • Anthony

    July 18, 2012, 3:21 pm

    Hello all, I figured I’d follow up my issue with my workaround so hopefully others can benefit.

    Another collegue came up with the idea to elimiate the sendEmail activity all together and send the e-mail using a Code Activity with System.Net.Mail.

    This seemed to have resolved my issue in all of my workflows.

    Again, I removed ALL sendmail activities and replaced them with pure code activities…

    Hope this helps others…

Comments are closed