Tom McLaughlin
In the engineering world a lot of our practices, even at times our best practices, are often just common wisdom passed along from one person to another. With Stack Overflow, Slack, and even Twitter, it’s easier today than it ever was for ideas to propagate. However, a lot of what passes for common wisdom is really just widely held opinions. And nothing says common wisdom has to be right. Where I ran into this distinction recently was with Python’s Boto3 modules (boto3 and botocore) and whether or not I should bundle them with my AWS Lambda deployment artifact.
Recently I found out the common wisdom I’ve adhered to was wrong. (Yes, someone on the internet was wrong.) Like many people, I use the Boto3 modules provided by the AWS Lambda runtime. However after talking with several folks at AWS I discovered, you should not be using the AWS Lambda runtime’s boto3 and botocore module. And you shouldn’t use botocore’s vendored version of the requests module whether no matter what instance of botocore you are using. I’ll explain how I found this out and explore why more than just me have probably gotten this best practice wrong.
How I Found Out I Was Wrong
If you’re not familiar with Python and AWS, the boto3 module, which is built on top of and requires the botocore module, is the official Python SDK for working with AWS. In addition, botocore includes it’s own version of the requests module for working with the HTTP protocol, such as making web API requests. There’s two bits of common wisdom I’ve picked up over the past year and a half when it comes to these modules when deploying Lambda functions. It roughly follows as this:
Because the boto3 module is already available in the AWS Lambda Python runtimes, don’t bother including boto3 and its dependency botocore in your Lambda deployment zip file. Similarly, the requests module is available too because botocore comes with its own vendored copy so don’t bother bundling that either.
So why was I forced to reconsider the wisdom I normally adhere to? Recently I was working on building an AWS serverless security workshop and I was covering vulnerable application dependencies. Knowing there was a recent vulnerability in the requests module I figured I had a perfect example. But, since I used the module vendored by botocore Snyk was not picking this up. I figured the fix was easy. I’ll revert the demo app to using a vulnerable version of the standard requests module, have the person update to a newer version to fix the vilnerability issue, and also slip in the common wisdom of just using the requests module from botocore.
Before recommending people use botocore’s requests module I went and took a look at GitHub to make sure requests is patched. That’s when I found botocore issue 1608. This issue acknowledges the vulnerability but it’s not patched seemingly because botocore only makes limited use of its requests module anymore, doesn’t use the affected code, and the module is in the process of deprecation.
After bringing up my usage pattern with AWS employees on both GitHub and Slack I learned AWS employees suggest you always bundle your own boto3 and botocore modules and to not use botocore’s version of requests. After querying the AWS community via twitter I found adherence to what AWS recommends is mixed.
Shipping Boto3 Modules
The practice of not bundling the boto3, botocore, and requests modules is probably some ancient wisdom the came about because of cold starts. Reduce the size of your deployable by removing modules already in the Lambda runtime environment and you’ll reduce your cold start time by a few milliseconds is how the belief goes. It’s a belief rooted in a truth, smaller deployment packages have shorter cold starts, but I’m not sure of the actual measurable benefits.
Knowing I learned this behavior from observing and reading others I put out a Twitter poll to see how what other people in the AWS serverless community do.
I don’t expect Twitter polls to be 100% accurate of real world practice, but these show show some divergence from AWS recommended practices. What’s more important (to me), I appear to not be the only person who got these practices wrong. What appears to have developed is a gap between what AWS expects people to do and what people are actually doing.
The AWS people I spoke with recommend you bundle all your own application dependencies, including ones in the Lambda runtime environment. While Lambda’s boto3 and botocore modules are patched to add new features and handle security updates which is good, there’s always the potential as with all software updates to introduce breaking changes. That means a long untouched Lambda function can mysteriously break on you due to an update you were unaware of. It doesn’t happen often but I am aware of it happening before.
Also, don’t use the vendored requests module in botocore. It was only meant to be used by botocore and that team of developers is working on deprecating its usage within their code base. They also never intended for others to use it and can’t guarantee API stability. The fact that the API has remained stable for so long is not intentional. It’s also not currently patched for a vulnerability and isn’t a high priority since it is unused code to them.
Why Did I Get This Wrong?
But why have people been getting the best practices wrong? Here’s what the AWS Lambda best practices say:
“Control the dependencies in your function's deployment package. The AWS Lambda execution environment contains a number of libraries such as the AWS SDK for the Node.js and Python runtimes (a full list can be found here: Lambda Execution Environment and Available Libraries). To enable the latest set of features and security updates, Lambda will periodically update these libraries. These updates may introduce subtle changes to the behavior of your Lambda function. To have full control of the dependencies your function uses, we recommend packaging all your dependencies with your deployment package.” - https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html
I have read this document before and that line. What’s always stood out to me in that best practice is this sentence however.
“To enable the latest set of features and security updates, Lambda will periodically update these libraries.”
So you’re telling me AWS will do security patching for me. If I trust AWS to manage my cloud infrastructure and manage my application runtime for me, it’s not a big leap for me to trust them with managing their own SDK for me. In the end I read that sentence and felt leaving Boto3 management up to AWS had more up sides and than down sides. I’ll manage my application dependencies EXCEPT for boto3, botocore, and requests!
If there’s any room for interpretation there is no guarantee that an individual will make the same interpretations and value calculations to come up with the same conclusion as you. As a result, a significant number of users will read what you wrote and do the opposite of what you intended.
Another reason we often get best practices wrong is we learn many of them through socialization. If you find enough confirmation from others that a certain practice is correct or fine, you just assume it to be true. Some might call this “cargo culting” but I like to call it “I’m sorry, but I have real work to do instead of having pedantic arguments”. A lot of the best practices are so minor they’re not worth in depth exploration or arguments.
Take a look at Serverless Framework serverless-python-requirements plugin. By default it strips out boto3 and botocore from your dependencies. If you’re a user of the plugin, did you know that? Did you read that far in the documentation to know that? And if you did know it stripped those modules out of your deployment artifact did you make the assumption that the behavior was a best practice because the plugin you see so widely use did it by default? I’m not blaming or intending to pick on the authors, it’s just an example of how we pickup best practices through socialization and sometimes we’re socializing the wrong things. By the way, there’s an open issue for this and the authors will accept a PR to change the behavior.
Closing
Let’s cover what we’ve learned. First, there’s some common practices that many of us AWS Lambda users do which go against AWS best practice advice. Always ship your own boto3 and botocore Python modules. Also, never use the vendored requests module from botocore. Next, we’re all still learning best practices. That means sometimes we’re misinterpreting them or getting them wrong. Just because you see it extensively done doesn’t mean it’s better than another idea or even correct. And finally, AWS people, remember we don’t know everything you know. We don’t live and breath AWS. We just use it in service of our own organizations. That means when we get things wrong instead of writing behavior off it may be worth investigating how prevalent the behavior is. If a significant amount of users are not using your software as intended, that is an issue you should address. And it might take more than just a PR to solve.