Remote Debugging Azure Functions from Visual Studio Stops Working

…and what you can learn from it

“The first thing you must keep in mind when you start using a serverless computing platform like Azure Functions is that everything is different, especially when things are obviously the same.”
–Abraham Lincoln, 1863

If you develop Azure Functions with Visual Studio it is easy to debug a function app running in the cloud. All you have to do is right-click the function app in Server Explorer and you are in business.

Although it is possible (and ultimately easier in many cases) to debug locally, remote debugging works well and is most convenient. If you choose a consumption plan to make it basically free (you totally did, you know you did!), eventually you may see the dreaded message “The breakpoint will not currently be hit. No symbols have been loaded for this document.”

You and I both know that your first reaction to this problem will be to open your favorite search engine which might lead you to a site named Stack Overflow where you might find the following article: visual studio 2017 remote debugging azure api app: “The breakpoint will not currently be hit. No symbols have been loaded for this document.”

You will try the suggestion and it might work. You will also be see that it was a bug, but it’s been fixed until you check and realize you have that or some later update and think “this is a bug and they broke it again!”

It isn’t a bug and it probably won’t be fixed!

A slightly different problem that might happen for the same reason is that the breakpoint will light up in the Visual Studio editor, but execution won’t stop, or it will stop some of the time but not other times. Maybe you’ll restart the function app and it will work for awhile.

“Ah-hah! I have found a bug,” you will think, and head to github to enter an issue such as this one: Remote debugging woes #538.

The Cause and the Lessons Learned

My experience with my first ‘development’ function app came from the fact that I started with one function and kept adding them. Back then the tools were pretty limited and it was (again) an easy thing to do. I had a variety of triggers in that app and the one I was working on was long running and used much more capacity than a short running web service. If during my development testing I put it under load by sending more than a few messages into the queue at a time, Azure Functions would very helpfully scale my function and helpfully create more instances automatically. Once this happens there is no way to attach to the right instance other than by shear luck because there is no way in a default configuration to steer all requests to a single instance. Furthermore, there is isn’t any way for the Azure Functions host to identify that a particular random message you sent should be treated uniquely and routed to the one instance out of however many there for debugging.

There are a few lessons I learned as a result if this experience that I can share. Here are my top 5.

  1. Never forget that in the default state of a consumption plan, there can (and probably will) be multiple instances.
  2. If you have more than one function in a function app, one of them might be causing the app to scale even if the one you want to debug has no or very little traffic.
  3. Good logging plus Application Insights are things use should strongly consider using throughout your application lifecycle. Logging, because you can’t assume you will be able to remotely debug all scenarios in development or even locally. And Application Insights because it makes it possible to understand and visualize complex and heavy load runtime behaviors.
  4. There are a number of configuration ‘knobs’ you can turn that affect scaling behavior and the number of instances for a given function. I will be writing about this in more depth in a future post.
  5. There are a lot of things to consider when combining functions in a single app. I will also be writing about this in more depth in a future post.

Thanks for reading!

–Doug Ware

P.S. Application Insights makes it easy to see how many instances are active. However, if you are not using Application Insights and prefer to bang rocks together, you can use Process Explorer from the Platform Features tab of the function app in the portal. If you click it and it takes a very long time to load, that is a good sign that there are a bunch of instances, but eventually it will usually display a list.

 

 

Some Real Performance Numbers from Some Real Azure Functions

I’ve been using Azure Functions in various production systems since late 2016. If you wonder how real apps perform using a consumption plan with auto-scaling (which means pay as you go based on usage), I have some data for you!

First up
is the most commonly executed function in my busiest production installation of Azure Functions for SharePoint. The EventDispatch function receives event notifications from various SharePoint Online tenants for a few different apps. When something of interest happens in a SharePoint site, Microsoft sends the EventDispatch function an HTTP POST request. EventDispatch process this message and drops the result into the Service Bus queue for whatever app that created the event receiver subscription.

Over the last 8 days or so this function was triggered around 40,000 times a day at an error rate of about 4 per day.

SharePoint waits for a response from the event receiver for some events and a user experiences this as a delay when this kind of event fires. Therefore, EventDispatch needs to run quickly. One thing people new to Azure Functions sometimes notice is with consumption-based function apps is that functions triggered by web requests are slow during requests that cause Azure to load the function. Fortunately, the traffic to this function is steady. This means that the startup operation happens in a tiny percent of requests and performance is excellent.

The following charts show what normal traffic looks like. Scouts honor! I opened the Application Insights for this app and took a screenshot of the point in time sometimes there is much more traffic and other times there is much less.

This function app and its associated storage account costs me less around $10 a month. The storage account is around 60% of the total cost and there is room for some optimization.

Next up is application specific functionality hosted in a different function app in a different Azure tenant. This particular function app is a bit of a grab bag in terms of individual functions and has a lot of room for improvement in a couple of ways, the least of which is the mingling of a long-running process that introduces a few other issues which are a subject for another day and another post.

The long running process fires periodically based on scheduled message delivery and that schedule causes all of the messages to begin delivery at the same time subject. On top of that, the way I wrote it is dumb and the job takes much longer to execute than it should. So, during the periods when it runs it forces the function app to scale and load on a lot of servers. The numbers are pretty grim.

That high server count represents instances of the app, and it is wasted consumption that costs money. One thing I have noticed is that instances are slow to unwind.

Nevertheless, this badly written function inside this grab-bag app still only costs around $45 a month. I haven’t fixed it because the cost is too low to justify fixing it.