{"status":200,"article":{"url":"https://hackernoon.com/debugging-serverless-applications-with-dashbird-82a2ed86d757","publish_datetime":null,"title":"","text":{"full":"With AWS Lambda, we get scalability and resilience out-of-the-box. What\u2019s more, AWS also provides built-in monitoring, logging and tracing support through CloudWatch and X-Ray. These built-in tools provide a good starting point but many developers eventually outgrow them as their serverless application becomes more complex.\n\nIn this post, let\u2019s take a serverless application and see how Dashbird can help you debug the application.\n\nChallenges with serverless observability\n\nWhen it comes to observability, serverless has introduced some interesting challenges. For so long, we have relied on the use of agents and daemons to collect metrics and logs. They run silently in the background, away from our critical paths where we are concerned with minimizing user-facing latencies. They collect, buffer and publish these observability data in batches to improve efficiency. As a practice, they are so deeply ingrained into how we monitor our applications, until now.\n\nWhen it comes to serverless, specifically with managed platforms such as AWS Lambda, there\u2019s simply nowhere for us to install these agents!\n\nTo collect metrics and logs as part of your function\u2019s invocation would understandably add overhead to its invocation time. Since AWS is collecting logs from your function asynchronously already, and publishing them to CloudWatch Logs. A common workaround is to subscribe to these logs and perform post-processing on them.\n\nIndeed, that is how Dashbird collects data about your function\u2019s execution. It subscribes the CloudWatch Logs log groups to a Kinesis stream and then processes the events from there. You can read about the advantages of this approach in this article.\n\nAs our serverless applications become more complex, it\u2019s important for us to be able to trace executions across multiple functions. As the demo app demonstrates, even a simple user transaction can span across multiple event sources as well as Lambda functions.\n\nThe demo app\n\nImagine you\u2019re building a Twitter clone. One of the core features of the system is to distribute a user\u2019s post to his followers\u2019 feeds. To implement this feature, imagine we have two separate API endpoints:\n\nPOST posts/create : to create a new post for the current user\n\nGET followers/{userId} : to fetch a user\u2019s followers\n\nEach endpoint is handled by a separate Lambda function \u2014 create-post and get-followers respectively.\n\nWhen a user publishes a new post, the create-post function would save the post in the posts DynamoDB table and also publish a post-created event into a Kinesis stream. This event then triggers a distribute-post function. This function would query the GET followers/{userId} endpoint and then add the post to the followers\u2019 feeds. The get-followers function would query the followers DynamoDB table to return the IDs of the user\u2019s followers.\n\nFor brevity sake I have omitted the logic for actually distributing the posts. So the overall architecture for our demo app looks like the following.\n\nTo make things more interesting, each of the Lambda functions are hardwired to error or timeout based on a configurable probability. The source code for the demo app is available on Github, so feel free to try it out yourself.\n\nIntroducing Dashbird\n\nEven with a simple serverless application like the one outlined above, we have quite a few functions to look after. Let\u2019s see how we can use Dashbird to help us monitor this application and debug issues.\n\nAs soon as I log in, I have a high level dashboard for my account.\n\nIn addition to the data I get in the AWS Lambda console (see below), the Dashbird dashboard has two useful data points:\n\nAverage memory utilization for the functions\n\nCost for the Lambda invocations\n\nNext, in Dashbird\u2019s Lambda console, I can see a high level summary of my functions and their activities over the last 24 hours.\n\nWhat I find very useful here is the fact that it highlights functions that have been idle for 10 days as inactive. As your serverless architecture expands and you end up with hundreds of functions, maintained by different teams, it\u2019s very difficult to track which functions are no longer needed. Having redundant functions lying around is a security risk as they remain an attack surface that can be exploited.\n\nWhile this view alone cannot tell you definitively which functions are no longer used. Many functions are not run on a regular basis. Maybe they are part of a cron job that only runs once a month. Or maybe they are only used during disaster recovery scenarios. Nonetheless, being aware of which functions are inactive encourages teams to ask the question \u201cIs this function still needed?\u201d. From here, maybe better practices can emerge. For example, use tags to mark functions that are expected to be used sparingly so they are not flagged by these checks.\n\nIf I navigate to one of the functions, then I have a function-centric view of invocation time, error, cost and memory utilization. In addition, I can also see a list of the recent invocations. What\u2019s really useful here is that cold starts and retries are clearly labelled. When debugging live issues this lets me quickly narrow down the invocations that I need to pay attention to.\n\nStraight away, I can see that 3 of the invocations timed out after 6 seconds. What\u2019s more, the original Kinesis event was retried 3 times and finally succeeded on the third retry.\n\nIf I click on the \u201c+\u201d button next to an invocation then I can drill down to the invocation itself. Here I can see the logs and X-Ray trace for this invocation in one screen. This is great as it saves me from having to constantly jump between different AWS console.\n\nDebugging with Dashbird\n\nAs mentioned before, the demo app is hardwired to error and timeout. And sure enough, when these failure cases happen, Dashbird\u2019s built-in alerting kicks in and I promptly received emails notifying me that something went wrong.\n\nWhile this built-in alerting is great, I couldn\u2019t see any settings to adjust the alert sensitivity.\n\nAs I followed the links in the emails, to the function, then the failed invocation. Dashbird neatly groups the related invocations \u2014 the initial timeout, and the subsequent retries \u2014 together. I can quickly see that the failed Kinesis event was successfully processed on the 3rd retry.\n\nDashbird also tracks the open issues in the Errors console. Now that I know the problem has resolved itself I can go ahead and resolve the error.\n\nTailing function logs\n\nAnother nice feature of Dashbird is that it\u2019s able to tail the logs for multiple functions at the same time. For the demo app, I want to see the logs for both the create-post and distribute-post functions as I curl the POST posts/create endpoint.\n\nThat way, I can see that the event was successfully published into the Kinesis stream, and was subsequently received by the distribute-post function.\n\nConclusion\n\nOverall I was impressed with what Dashbird has to offer, and it\u2019s clear that a lot of thought has gone into the product. It has many nice touches that makes debugging much easier. For example, grouping retries together, and integrating X-Ray traces and logs in one screen. These might seem like trivial niceties, but they can make a big difference under the high-pressure scenarios of dealing with a live issue.\n\nFrom what I have been able to see, I think Dashbird is a really great tool. The main thing missing for me is the ability to trace executions end-to-end. Personally, I\u2019d really like to see the entire execution traced from the API call to create a post, all the way to the get-followers function performing a Query against the followers DynamoDB table.\n\nIf you\u2019re solely relying on the built-in AWS tools (CloudWatch, CloudWatch Logs, and X-Ray) then you should give Dashbird a try. Why not sign up for a free trial, and deploy the demo app to your environment so you can see how Dashbird can help you debug your serverless application?\n\nOriginally published at dashbird.io on January 25, 2019.","keywords":["function","logs","serverless","invocation","lambda","demo","post","aws","functions","dashbird"],"summary":""},"authors":[],"media":{"images":["https://hackernoon.imgix.net/hn-images/0*3wx05u1keXGMJAij.png?auto=format%2Ccompress&w=3840","https://hackernoon.imgix.net/hn-images/0*MU2JAw5c2Pkrb6Bn.png?auto=format%2Ccompress&w=3840","https://hackernoon.imgix.net/computer.png?auto=format%2Ccompress&w=48","https://hackernoon.imgix.net/hn-images/0*unPiVL0E6QC057iS.png?auto=format%2Ccompress&w=3840","https://hackernoon.imgix.net/hn-images/0*8_jid1Y_sG_cbTJn.png?auto=format%2Ccompress&w=3840","https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&w=96","https://hackernoon.imgix.net/hn-images/0*9-VOw-xsrF3mB5nQ.png?auto=format%2Ccompress&w=3840","https://hackernoon.imgix.net/hn-images/0*J7SEZ_EK1aWK6o3A.png?auto=format%2Ccompress&w=3840","https://hackernoon.imgix.net/images/Lite%20Icon%20%4025px.png?auto=format%2Ccompress&w=48","https://hackernoon.imgix.net/hn-images/0*nq_eP-14T_YcL3ps.png?auto=format%2Ccompress&w=3840","https://hackernoon.imgix.net/hn-images/0*OKovmxFYKL4pGPmG.png?auto=format%2Ccompress&w=3840","https://hackernoon.imgix.net/hn-images/0*-RRtkvYI3B7LvFeT.png?auto=format%2Ccompress&w=3840","https://hackernoon.imgix.net/hn-images/0*Tjx1XgwC9lZZgtCP.png?auto=format%2Ccompress&w=3840","https://hackernoon.imgix.net/hn-images/0*z21gtl_jfN-kE-nn.png?auto=format%2Ccompress&w=3840","https://hackernoon.imgix.net/hn-images/0*NoHAE22yc7RxaT9V.png?auto=format%2Ccompress&w=1920","https://hackernoon.imgix.net/hn-images/0*MU2JAw5c2Pkrb6Bn.png","https://hackernoon.imgix.net/images/avatars/YjziSuGeDecNvCZR0j4bivrb6V13.jpg?auto=format%2Ccompress&w=3840","https://hackernoon.imgix.net/images/Print%20Icon%20%4025px.png?auto=format%2Ccompress&w=48","https://hackernoon.imgix.net/hn-images/0*TDIcxxsnG-n_eap0.png?auto=format%2Ccompress&w=3840","https://hackernoon.imgix.net/hn-images/0*P8P9xvKvn_-IdQzv.png?auto=format%2Ccompress&w=3840","https://hackernoon.imgix.net/hn-images/0*CsuMFbXUfo4gUeLy.png?auto=format%2Ccompress&w=3840"],"movies":[]},"html":"
Write
490 reads

Debugging Serverless Applications with Dashbird

by
\"Yan
byYan Cui|@theburningmonk

AWS Serverless Hero. Independent Consultant. Developer Advocate at Lumigo.

January 29th, 2019
\"Read\"Print\"Read
\"Read\"Print\"Read
\"featured
\"Yan
\"Yan

With AWS Lambda, we get scalability and resilience out-of-the-box. What\u2019s more, AWS also provides built-in monitoring, logging and tracing support through CloudWatch and X-Ray. These built-in tools provide a good starting point but many developers eventually outgrow them as their serverless application becomes more complex.

\n

In this post, let\u2019s take a serverless application and see how Dashbird can help you debug the application.

\n

Challenges with serverless observability

\n

When it comes to observability, serverless has introduced some interesting challenges. For so long, we have relied on the use of agents and daemons to collect metrics and logs. They run silently in the background, away from our critical paths where we are concerned with minimizing user-facing latencies. They collect, buffer and publish these observability data in batches to improve efficiency. As a practice, they are so deeply ingrained into how we monitor our applications, until now.

\n

When it comes to serverless, specifically with managed platforms such as AWS Lambda, there\u2019s simply nowhere for us to install these agents!

\n

To collect metrics and logs as part of your function\u2019s invocation would understandably add overhead to its invocation time. Since AWS is collecting logs from your function asynchronously already, and publishing them to CloudWatch Logs. A common workaround is to subscribe to these logs and perform post-processing on them.

\n

Indeed, that is how Dashbird collects data about your function\u2019s execution. It subscribes the CloudWatch Logs log groups to a Kinesis stream and then processes the events from there. You can read about the advantages of this approach in this article.

\n

As our serverless applications become more complex, it\u2019s important for us to be able to trace executions across multiple functions. As the demo app demonstrates, even a simple user transaction can span across multiple event sources as well as Lambda functions.

\n

The demo\u00a0app

\n

Imagine you\u2019re building a Twitter clone. One of the core features of the system is to distribute a user\u2019s post to his followers\u2019 feeds. To implement this feature, imagine we have two separate API endpoints:

\n
    \n
  • POST posts/create\u00a0: to create a new post for the current user
    • \n
  • GET followers/{userId}\u00a0: to fetch a user\u2019s followers
    • \n
\n

Each endpoint is handled by a separate Lambda function\u200a\u2014\u200acreate-post and get-followers respectively.

\n

When a user publishes a new post, the create-post function would save the post in the posts DynamoDB table and also publish a post-created event into a Kinesis stream. This event then triggers a distribute-post function. This function would query the GET followers/{userId} endpoint and then add the post to the followers\u2019 feeds. The get-followers function would query the followers DynamoDB table to return the IDs of the user\u2019s followers.

\n

For brevity sake I have omitted the logic for actually distributing the posts. So the overall architecture for our demo app looks like the following.

\n
\"Yan
\n

To make things more interesting, each of the Lambda functions are hardwired to error or timeout based on a configurable probability. The source code for the demo app is available on Github, so feel free to try it out yourself.

\n

Introducing Dashbird

\n

Even with a simple serverless application like the one outlined above, we have quite a few functions to look after. Let\u2019s see how we can use Dashbird to help us monitor this application and debug issues.

\n

As soon as I log in, I have a high level dashboard for my account.

\n
\"Yan
\n

In addition to the data I get in the AWS Lambda console (see below), the Dashbird dashboard has two useful data points:

\n
    \n
  • Average memory utilization for the functions
    • \n
  • Cost for the Lambda invocations
    • \n
\n
\"Yan
\n

Next, in Dashbird\u2019s Lambda console, I can see a high level summary of my functions and their activities over the last 24 hours.

\n
\"Yan
\n

What I find very useful here is the fact that it highlights functions that have been idle for 10 days as inactive. As your serverless architecture expands and you end up with hundreds of functions, maintained by different teams, it\u2019s very difficult to track which functions are no longer needed. Having redundant functions lying around is a security risk as they remain an attack surface that can be exploited.

\n

While this view alone cannot tell you definitively which functions are no longer used. Many functions are not run on a regular basis. Maybe they are part of a cron job that only runs once a month. Or maybe they are only used during disaster recovery scenarios. Nonetheless, being aware of which functions are inactive encourages teams to ask the question \u201cIs this function still needed?\u201d. From here, maybe better practices can emerge. For example, use tags to mark functions that are expected to be used sparingly so they are not flagged by these checks.

\n
\"Yan
\n

If I navigate to one of the functions, then I have a function-centric view of invocation time, error, cost and memory utilization. In addition, I can also see a list of the recent invocations. What\u2019s really useful here is that cold starts and retries are clearly labelled. When debugging live issues this lets me quickly narrow down the invocations that I need to pay attention to.

\n
\"Yan
\n

Straight away, I can see that 3 of the invocations timed out after 6 seconds. What\u2019s more, the original Kinesis event was retried 3 times and finally succeeded on the third retry.

\n
\"Yan
\n

If I click on the \u201c+\u201d button next to an invocation then I can drill down to the invocation itself. Here I can see the logs and X-Ray trace for this invocation in one screen. This is great as it saves me from having to constantly jump between different AWS console.

\n
\"Yan
\n

Debugging with\u00a0Dashbird

\n

As mentioned before, the demo app is hardwired to error and timeout. And sure enough, when these failure cases happen, Dashbird\u2019s built-in alerting kicks in and I promptly received emails notifying me that something went wrong.

\n
\"Yan
\n
\"Yan
\n

While this built-in alerting is great, I couldn\u2019t see any settings to adjust the alert sensitivity.

\n

As I followed the links in the emails, to the function, then the failed invocation. Dashbird neatly groups the related invocations\u200a\u2014\u200athe initial timeout, and the subsequent retries\u200a\u2014\u200atogether. I can quickly see that the failed Kinesis event was successfully processed on the 3rd retry.

\n
\"Yan
\n

Dashbird also tracks the open issues in the Errors console. Now that I know the problem has resolved itself I can go ahead and resolve the error.

\n
\"Yan
\n

Tailing function\u00a0logs

\n

Another nice feature of Dashbird is that it\u2019s able to tail the logs for multiple functions at the same time. For the demo app, I want to see the logs for both the create-post and distribute-post functions as I curl the POST posts/create endpoint.

\n

That way, I can see that the event was successfully published into the Kinesis stream, and was subsequently received by the distribute-post function.

\n
\"Yan
\n

Conclusion

\n

Overall I was impressed with what Dashbird has to offer, and it\u2019s clear that a lot of thought has gone into the product. It has many nice touches that makes debugging much easier. For example, grouping retries together, and integrating X-Ray traces and logs in one screen. These might seem like trivial niceties, but they can make a big difference under the high-pressure scenarios of dealing with a live issue.

\n

From what I have been able to see, I think Dashbird is a really great tool. The main thing missing for me is the ability to trace executions end-to-end. Personally, I\u2019d really like to see the entire execution traced from the API call to create a post, all the way to the get-followers function performing a Query against the followers DynamoDB table.

\n
\"Yan
\n

If you\u2019re solely relying on the built-in AWS tools (CloudWatch, CloudWatch Logs, and X-Ray) then you should give Dashbird a try. Why not sign up for a free trial, and deploy the demo app to your environment so you can see how Dashbird can help you debug your serverless application?

\n

Originally published at dashbird.io on January 25, 2019.

About Author

\"Yan
byYan Cui|@theburningmonk

AWS Serverless Hero. Independent Consultant. Developer Advocate at Lumigo.

Read my storiesLearn More

TOPICS

cybersecurity
#aws#aws-lambda#serverless#cloud-computing#debugging-serverless

Light-Mode

Classic

Newspaper

Proof of Usefulness

Dark-Mode

Neon Noir

Minty

Startups of the Year

"},"msg":["CREDITS: this demo application is brought to you by courtesy of Dashbird.io. Dashbird makes monitoring logs and performance of serverless applications a breeze. Start a free trial today, no credit card required: https://dashbird.io","COPYRIGHT NOTICE: the data retrieved by the application may be subject to copyright and/or terms of usage. This application is intended ONLY for demonstration purposes. It does not grant you any rights over the content retrieved and you are solely responsible for non-legal usage of the application."]}