If you have ever heard about AWS Lambda or serverless architecture you also probably heard about AWS X-Ray, no?. This is fine. It has been outside of my eyesight for a while as well because it lacks promotion, it has poor documentation (imho) and in general the Internet seems to have very limited amount of information about this powerful tool (at least I was not able to find the same amount of information as for EC2, for example). And I find it very strange because AWS X-Ray can reveal completely new cloud debugging and tracing experience for you. Why? Let’s investigate.
First things first
To start playing around with AWS X-Ray you of course need AWS account and you should create AWS lambda function (“Author from scratch” template proposed by AWS will be just fine). Programming language that I know the best is python so I am going to use it to create examples. But capabilities of AWS X-Ray are not unique to python and a lot of other languages will provide roughly the same capabilities to you. Also consider that this article will cover only essentials of AWS X-Ray (and how it interacts with lambdas) because this is very big and complicated tool, it requires at least a couple of articles (I have plans to write them). And the last thing, I assume you have some experience with AWS so I will not explain what is IAM Role, S3 bucket and etc. Remember that AWS may charge you for everything you do so before following the examples please visit AWS and check prices https://aws.amazon.com/pricing, after you have finished playing around do not forget to clean up after yourself to avoid charges for nothing.
You should have something like this if you create lambda with “Author from scratch” template. And you should be able to execute the lambda by clicking “Test” and providing data for the event. There is no AWS X-Ray here yet. So let’s go and activate it. Just scroll down until you see “Monitoring tools” configuration and then click on “Edit” button.
There is a number of settings and switches that you can tweak but we are interested in AWS X-Ray switch. Turn it on and save.
As you can see lambda function that was created did not have enough permissions to use AWS X-Ray but it can be fixed automatically as the message in the blue box says. Try to execute the lambda now. It seems that nothing has changed but this is because you have to go to AWS X-Ray to see what is going on. After navigating to the service you will see something like this.
Do not worry there is no data expected here, just hit on “Traces”.
This is it the first trace. Let’s click on it and see what information it contains.
There are three things that you should pay attention to:
- Trace map. It shows which services were affected during the execution. Currently the only affected service is lambda. It is shown twice because AWS wants to show us single lambda twice, just live with that.
- Trace information. It shows what each service was doing during the execution and how long it was executing. Each foldable name (XRayArticle AWS::Lambdas::Function, for example) represents a concept called segment.
- Subsegments. Each segment contains some number of subsegments. In the example, segment XRayArticle AWS::Lambdas::Function contains 2 subsegments: “Invocation” and “Overhead”. You can click on each subsegment to get more information. Currently it contains generic and useless information but I encourage you to click on it anyway.
We will get back to trace details and investigate them in more details closer to the end of this article especially concepts of segment and subsegment.
So far this does not look like something super useful but this is because lambda function that was created does nothing and we are about to change that!
It is time to make next logical step and add some “logic” to the example. So you should do the following things:
- Create new S3 bucket and put file sample_data.json there
2. Update lambda code so it looks like on the picture below (you can download source code here https://github.com/DimonLuk/aws-lambda-xray-examples look for tag works-with-s3-and-xray). Do not forget to give your lambda permissions to download files from S3 and to change some values if you have different set up.
On the screenshot you can see:
- Content of directory where lambda function lives. It contains: .vscode folder with my editor’s configuration, venv with python libraries(if you want to follow the example exactly you should create venv and install libraries listed in requirements.txt there because libraries from this folder will be uploaded to AWS), .gitignore, lamda_function.py contains code of lambda function, lambda.zip is archived lambda code with python libraries, LICENSE, requirements.txt which has 2 requirements: aws-xray-sdk and boto3, zip_lambda.sh shell script to create or update lambda.zip with your last changes so that it can be uploaded to AWS (more details about this here https://docs.aws.amazon.com/lambda/latest/dg/python-package.html#python-package-venv, this is not the best way of delivering your code to AWS Lambda but it is fine when you are investigating something).
- Import of strange function patch_all
3. Call to patch_all function.
Also you can see some code which interacts with S3 but the main point of interest here is patch_all function which patches boto3 library so that it starts sending some information to X-Ray service. A lot of other libraries can be patched in the same way, all you have to do is to import them before calling patch_all (more details here https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-python.html).
Now let’s execute the lambda and if everything is properly configured you should get a successful response. Now it is time to check what is going on in AWS X-Ray service.
What you see here is that S3 bucket appeared on trace map (1) and it appeared as a subsegment in the list (2) (if patch_all is not called, then S3 subsegment will not appear in AWS X-Ray). You can click on it but you will not get a lot of useful information.
Earlier I promised to you to discuss concepts of segments and subsegments so finally I am ready to do this. They are very simple but very, very powerful entities.
Segment, well… is a segment of your code. If you use AWS Lambda like in the examples, then segment is defined by AWS as a whole lambda function and you cannot define your segments within lambda or edit existing. But if you use flask application instead and deploy it to EC2 instance then you are free to define segments in any way you want.
Subsegment is a part of code within a segment. And this entity is always at your disposal, you can declare any part of your code as a subsegment no matter if it is in lambda or in your flask application running on EC2. There is a number of ways to tell to AWS X-Ray that some part of code is a subsegment (https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-python-subsegments.html) but the best way, in my opinion, is to use context manager because it handles exceptions without any overhead whereas other ways do not and you have to implement some unnecessary tricks that look weird.
Let’s add some subsegments magic to the lambda (look for tag subsegment-example in the repo).
You can see that now there is a subsegment named “Throwing exception” also I have added some metadata and annotations to this subsegment so we can finally see how it looks in AWS X-Ray (spoiler, it looks amazing and provides unbelievable capabilities in terms of tracing and debugging but more on that in the summary). Now let’s execute the lambda and see how introduced changes will be reflected in AWS X-Ray. The lambda should fail because exception is not handled but this is the point of this example.
In AWS X-Ray you will see something like this:
Looks much more interesting now. Let’s go to the details of “Throwing exception” subsegment and check “Annotations”, “Metadata” and “Exceptions” tabs.
To sum up, let’s think why I have said that this tool “can reveal completely new cloud debugging and tracing experience for you”.
At first, it is very easy to set up and configure not only for AWS Lambda but for a lot of other AWS services, python libraries, frameworks and other programming languages.
Secondly, it shows you complete map of affected services and the results they produced. This could be very helpful when you just joined a new project because you can make a bunch of requests in dev/test environment, then go to AWS X-Ray and see parts of the infrastructure affected by the request. Not to mention, it is very helpful to see where exactly failure happened in the first place.
Thirdly, segments and subsegments… They are so powerful that I am just amazed. You are allowed to split your code into the chunks in any way you could imagine and you will be able to get very detailed information about every chunk. You can add custom annotations and metadata with the information about incoming data, about expected results and with ANY INFORMATION THAT YOU WANT!!! Combination of these two facts, for me, sounds like legalized “debugging in production environment” because with careful design you can achieve experience very close to what you have when debugging code on your machine executing it line by line. I definitely would like to have this tool working in my production environment because when some bugs occur, I can just go to AWS X-Ray and literally see what, where and why happened and this saves so much time… I do not have to go through logs trying to understand where issues happened, I do not have to run everything locally and try to reproduce, I can just go and see… Sounds like magic but this is what you can achieve with AWS X-Ray (AWS did not pay me for promotion :) ).
Conclusion. To my mind, AWS X-Ray is a very powerful tool and I hope this short article convinced you why. I know that I did not cover a looooooot of things because it is impossible to do so in a single article, I have plans to write more articles about AWS X-Ray and related topics, for example, about architecting subsegments to achieve experience that I have described in the paragraph above or about optimizing costs and etc. So many things need to be covered so I would like to ask you to help me. If you like article and have a spare minute please help me to improve quality of articles and decide on the next topics by participating into a tiny survey (https://forms.gle/FVuZTLDFEZ16jdZN9).
Thank you for reading I hope you enjoyed and learnt something new and exciting!