Monitoring Django App with Azure Application Insights

Leave a Comment / DevOps, Django, HowTo / By

Monitoring Django Applications with Azure App Insights

Application Insights is an application performance monitoring(APM) service developed by Microsoft Azure.It is quite useful for monitoring both server side and client side applications built with dotnet,nodejs,java and also python.
In this tutorial we will explore how to monitor your django apps and project using App Insights(AI for short).

Application Insights is an extension of Azure Monitor. APM tools are useful to monitor applications from development, through test, and into production by understanding how an application is performing and reactively review data generated by an application to determine the cause of an issue or error.
It is a useful tool to collect and analyse telemetry data.

Now how do you work with AI?. TLDR

  • Create A Resource
  • Get Connection String / Instrumentation Key
  • Instrumentation
  • Config Integrations

First of all you will need a means to collect as much useful data from the application you are monitoring. The main approach is to integrate an SDK or an agent (software that act as a middleman between the application and the APM like how human agents work) to collect data and metrics.This SDK or agent will act like a hook to collect measurements and data. The process of integrating or adding these SDK or agent to an application you want to monitor and is termed Instrumentation. An instrument is simply a device that is used to take measurement or perform a task. Hence the term instrumentation.

There are several SDK used for instrumentation for AI based on the language of choice.
In our case we will be using Python and Javascript SDK. For now the standard way to monitor your django app is via the Opencensus Python SDK provided by Opentelemetry.
Since we want to monitor both we will be instrumenting both sides,i.e.

  • the server side (django backend) and
  • the client-side (frontend web pages built with javascript and html)

Server Side Instrumentation of Django App

We will want to track all outgoing (outbound) requests coming from our app as well as incoming request and dependencies. A dependency is simply any external call to a service or function, such sql query,REST API call,etc. In our case we will want to track all requests,logs as well as SQL queries being made in our django app. hence we have to install the following packages.

Install the following packages
python -m pip install opencensus-ext-azure opencensus-ext-django opencensus-ext-requests opencensus-ext-logging opencensus-ext-postgresql
  • opencensus-ext-django
  • opencensus-ext-postgresql
  • opencensus-ext-requests
  • opencensus-ext-logging

Next add the sdk and include the SDK as a middleware, this is useful to track every request made in your django app

# settings.py
MIDDLEWARE = (
    ...
    'opencensus.ext.django.middleware.OpencensusMiddleware',
    ...
)

Configure the AzureExporter in your settings.py This is useful for tracing/ distributed tracing of every part of the app.

# settings.py
OPENCENSUS = {
    'TRACE': {
        'SAMPLER': 'opencensus.trace.samplers.ProbabilitySampler(rate=1)',
        'EXPORTER': '''opencensus.ext.azure.trace_exporter.AzureExporter(
            connection_string="InstrumentationKey=<your-ikey-here>"
        )''',
        'EXCLUDELIST_PATHS': ['https://example.com'],  <--- These sites will not be traced if a request is sent to it.
    }

Configure Integrations

To be able to track requests,logs and dependencies such as postgresql, you will have to config the integrations.

The requests integration allows you to be able to trace request via adding a trace-id and span-id to your logs.This is useful for distributed tracing and log correlation. Since we have the requests integrated we can now add these to our logging in our django LOGGING settings as below

# settings.py
from opencensus.trace import config_integration
config_integration.trace_integrations(['requests','logging','postgresql'])

LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "simple": {
            "format": "%(asctime)s | %(levelname)s | %(message)s"
        },
        "azure_verbose": {
            "format": "%(asctime)s |  %(name)s | %(levelname)s | [%(funcName)s:%(filename)s:%(lineno)d] |"
                      " [%(threadName)s:%(process)d] | %(message)s"
                      "traceID=%(traceId)s spanId=%(spanId)s"
        },

    },
    "handlers": {
        "stdout": {
            "level": "DEBUG",
            "class": "logging.StreamHandler",
            "formatter": "simple",
        },
        "log_to_file": {
            "level": "DEBUG",
            "class": "logging.FileHandler",
            "formatter": "verbose",
            "filename": os.path.join(BASE_DIR, "logs", "dev.logs") # same loc as blogproject
        },
         "log_to_azure_ai": {
            "level": "DEBUG",
            "class": "opencensus.ext.azure.log_exporter.AzureLogHandler",
            "connection_string": os.environ.get("APPLICATIONINSIGHTS_CONNECTION_STRING"),
            "formatter": "verbose",
        },

    },
    "loggers": {
        'django': {
            'handlers': ['stdout', 'log_to_file','log_to_azure_ai'],
        },
    }
}

Now we can now head back to our azure portal and see the results and the logs.
For working behind corporate firewall, you will have to configure your proxy in the environment variables. This will help fix the error of “Request Trying”.

export HTTPS_PROXY=http://myproxy.com/3128

Now your app will be able to send telemetry to App Insights

Client-Side Monitoring

In Application Insights, there is a section for the server and another for the client-side also called browser in the App Insights Dashboard. This is useful for monitoring page views,funnel and usage analysis, etc. If you add Application Insights to your page script, you get timings of page loads and AJAX calls, counts, and details of browser exceptions and AJAX failures. You also get user and session counts. All this telemetry can be segmented by page, client OS and browser version, geo location, and other dimensions. To be able to enjoy this feature you will have to instrument the front-end via the nodejs SDK or the javascript snippet setup.
Since we are not using nodejs, we will have to stick with the JavaScript snippet.

Copy the JavaScript snippet to the head element of every html page of where you want to monitor. You can simplify this by adding this snippet to the base.html or layout.html in you django app. By this method your code becomes DRY

<script type="text/javascript">
!function(T,l,y){var S=T.location,k="script",D="connectionString",C="ingestionendpoint",I="disableExceptionTracking",E="ai.device.",b="toLowerCase",w="crossOrigin",N="POST",e="appInsightsSDK",t=y.name||"appInsights";(y.name||T[e])&&(T[e]=t);var n=T[t]||function(d){var g=!1,f=!1,m={initialize:!0,queue:[],sv:"5",version:2,config:d};function v(e,t){var n={},a="Browser";return n[E+"id"]=a[b](),n[E+"type"]=a,n["ai.operation.name"]=S&&S.pathname||"_unknown_",n["ai.internal.sdkVersion"]="javascript:snippet_"+(m.sv||m.version),{time:function(){var e=new Date;function t(e){var t=""+e;return 1===t.length&&(t="0"+t),t}return e.getUTCFullYear()+"-"+t(1+e.getUTCMonth())+"-"+t(e.getUTCDate())+"T"+t(e.getUTCHours())+":"+t(e.getUTCMinutes())+":"+t(e.getUTCSeconds())+"."+((e.getUTCMilliseconds()/1e3).toFixed(3)+"").slice(2,5)+"Z"}(),name:"Microsoft.ApplicationInsights."+e.replace(/-/g,"")+"."+t,sampleRate:100,tags:n,data:{baseData:{ver:2}}}}var h=d.url||y.src;if(h){function a(e){var t,n,a,i,r,o,s,c,u,p,l;g=!0,m.queue=[],f||(f=!0,t=h,s=function(){var e={},t=d.connectionString;if(t)for(var n=t.split(";"),a=0;a<n.length;a++){var i=n[a].split("=");2===i.length&&(e[i[0][b]()]=i[1])}if(!e[C]){var r=e.endpointsuffix,o=r?e.location:null;e[C]="https://"+(o?o+".":"")+"dc."+(r||"services.visualstudio.com")}return e}(),c=s[D]||d[D]||"",u=s[C],p=u?u+"/v2/track":d.endpointUrl,(l=[]).push((n="SDK LOAD Failure: Failed to load Application Insights SDK script (See stack for details)",a=t,i=p,(o=(r=v(c,"Exception")).data).baseType="ExceptionData",o.baseData.exceptions=[{typeName:"SDKLoadFailed",message:n.replace(/\./g,"-"),hasFullStack:!1,stack:n+"\nSnippet failed to load ["+a+"] -- Telemetry is disabled\nHelp Link: https://go.microsoft.com/fwlink/?linkid=2128109\nHost: "+(S&&S.pathname||"_unknown_")+"\nEndpoint: "+i,parsedStack:[]}],r)),l.push(function(e,t,n,a){var i=v(c,"Message"),r=i.data;r.baseType="MessageData";var o=r.baseData;return o.message='AI (Internal): 99 message:"'+("SDK LOAD Failure: Failed to load Application Insights SDK script (See stack for details) ("+n+")").replace(/\"/g,"")+'"',o.properties={endpoint:a},i}(0,0,t,p)),function(e,t){if(JSON){var n=T.fetch;if(n&&!y.useXhr)n(t,{method:N,body:JSON.stringify(e),mode:"cors"});else if(XMLHttpRequest){var a=new XMLHttpRequest;a.open(N,t),a.setRequestHeader("Content-type","application/json"),a.send(JSON.stringify(e))}}}(l,p))}function i(e,t){f||setTimeout(function(){!t&&m.core||a()},500)}var e=function(){var n=l.createElement(k);n.src=h;var e=y[w];return!e&&""!==e||"undefined"==n[w]||(n[w]=e),n.onload=i,n.onerror=a,n.onreadystatechange=function(e,t){"loaded"!==n.readyState&&"complete"!==n.readyState||i(0,t)},n}();y.ld<0?l.getElementsByTagName("head")[0].appendChild(e):setTimeout(function(){l.getElementsByTagName(k)[0].parentNode.appendChild(e)},y.ld||0)}try{m.cookie=l.cookie}catch(p){}function t(e){for(;e.length;)!function(t){m[t]=function(){var e=arguments;g||m.queue.push(function(){m[t].apply(m,e)})}}(e.pop())}var n="track",r="TrackPage",o="TrackEvent";t([n+"Event",n+"PageView",n+"Exception",n+"Trace",n+"DependencyData",n+"Metric",n+"PageViewPerformance","start"+r,"stop"+r,"start"+o,"stop"+o,"addTelemetryInitializer","setAuthenticatedUserContext","clearAuthenticatedUserContext","flush"]),m.SeverityLevel={Verbose:0,Information:1,Warning:2,Error:3,Critical:4};var s=(d.extensionConfig||{}).ApplicationInsightsAnalytics||{};if(!0!==d[I]&&!0!==s[I]){var c="onerror";t(["_"+c]);var u=T[c];T[c]=function(e,t,n,a,i){var r=u&&u(e,t,n,a,i);return!0!==r&&m["_"+c]({message:e,url:t,lineNumber:n,columnNumber:a,error:i}),r},d.autoExceptionInstrumented=!0}return m}(y.cfg);function a(){y.onInit&&y.onInit(n)}(T[t]=n).queue&&0===n.queue.length?(n.queue.push(a),n.trackPageView({})):a()}(window,document,{
src: "https://js.monitor.azure.com/scripts/b/ai.2.min.js", // The SDK URL Source
// name: "appInsights", // Global SDK Instance name defaults to "appInsights" when not supplied
// ld: 0, // Defines the load delay (in ms) before attempting to load the sdk. -1 = block page load and add to head. (default) = 0ms load after timeout,
// useXhr: 1, // Use XHR instead of fetch to report failures (if available),
crossOrigin: "anonymous", // When supplied this will add the provided value as the cross origin attribute on the script tag
// onInit: null, // Once the application insights instance has loaded and initialized this callback function will be called with 1 argument -- the sdk instance (DO NOT ADD anything to the sdk.queue -- As they won't get called)
cfg: { // Application Insights Configuration
    connectionString: "Copy connection string from Application Insights Resource Overview"
    /* ...Other Configuration Options... */
}});
</script>

python

Now you will be able to see which pages were viewed the most,etc via the transaction search section or the logs analytics or performance,metric tabs of AI.

Check out the video tutorial for more.

Thanks for your time
Jesus Saves
By Jesse E.Agbe(JCharis)

Leave a Comment

Your email address will not be published. Required fields are marked *