BeamlineJS — CI/CD pipeline for AWS Lambda By AWS Lambda
What is Beamline?
From — Wikipedia — “In accelerator physics, a beamline refers to the trajectory of the beam of accelerated particles, including the overall construction of the path segment (vacuum tube, magnets, diagnostic devices) along a specific path of an accelerator facility. This part is either
- the line in a linear accelerator along which a beam of particles travels, or
- the path leading from a cyclic accelerator to the experimental endstation (as in synchrotron light sources or cyclotrons).”
Once upon a time, I asked myself the below question. This question inspired me to embark on an incredible journey of creating a CI/CD pipeline using FaaS (a.k.a Serverless) architecture.
If my application is “Serverless”, then why should the CI/CD pipeline of my application use servers?
Introducing BeamlineJS
BeamlineJS is made up of “3” AWS Lambda functions that build, release, test and deploy AWS Lambda functions across regions.
After AWS Lambda’s launch in 2014, I started building my own applications using AWS Lambda just for academic purpose. In my current job at Capital One, I am using AWS Lambda (node, Java and Python) for delivering multiple business use cases. We make use of Jenkins to create a CI/CD pipeline of these functions (business use cases). I absolutely love all the features offered by tools such as Jenkins or CircleCI. I have used them extensively in various projects. However, for me to use Jenkins I will have to run it on EC2s (that I will have manage and maintain). In addition, if I want to have a fully operational CI/CD pipeline of a simple “Hello World” lambda function, then I will have to:
- Either setup my own Jenkins server with all the plugins/plumbings and setup CI/CD pipeline of my “Hello World” function OR
- Work with a dedicated team of DevOps engineers who provide and maintain a Jenkins environment on which I can setup a CI/CD pipeline of my “Hello World” function.
To build, release, test and deploy merely 50 lines of my function (business functionality) I will end up writing/using 1000+ lines of code.
BeamlineJS offers a plug & play CI/CD solution for your lambda functions. All you need to do is configure pipeline’s configuration in a .yaml file and subscribe your GitHub repository to your BeamlineJS’s SNS topic. You can setup your own BeamlineJS by following a few simple steps.
Why?
Following are some the primary motivations I needed BeamlineJS to achieve:
- How can I accelerate the integration and delivery of my lambda based applications? — hence the name Beamline, it is a facility that can accelerate the movement of particles (lambda functions) from one stage to another and across environments.
- Minimum development effort to setup the CI/CD pipeline so that I can focus more on delivery of customer intent.
- Easy setup, maintainable, and scalable.
- How can I be sure my function code and function configuration is well integrated and thoroughly tested?
- How can I be sure my function code and configuration is able to go through the quality gates (At Capital One we care about our customers and we have 16 quality gates to ensure that we deliver the best user experience)?
- How can I be sure a function’s code as stored in my repository is what I expect and intend it to be when it is deployed? (inspired by this)
- How can I be sure the function configuration I am invoking is the configuration I expect and intend it to be? (inspired by this)
- Establish a governance process for developing AWS Lambda functions.
- Not tied to a specific cloud provider (didn’t want to use AWS CodePipeline) — yes I tested BeamlineJS only on AWS, however the code is in node.js. So if I have to run this code as a function in another cloud environment, all I need to do is write a cloud provider specific function handler and configurations.
BeamlineJS features:
- Auto-scaleable (AWS Lambda) and multi-region availability.
- Ability to setup BeamlineJS in multiple regions.
- Automatically create pull request to merge from fork into develop branch and from develop branch to master (read further to see the Git branching model it uses).
- Development and QA (staging) pipelines are started automatically when merge of PR is successful on the respective branch.
- Manages lambda function versioning and aliases.
- Handles rollbacks in case of failures.
- Verifies the SHA256 of lambda function’s code and configuration.
- Uses NodeJS frameworks to run unit, functional (QA), integration and load tests.
- Sends slack notifications.
- Function code is released in S3 bucket
- Multi region testing at fork, development, staging (qa) and production pipelines.
BeamlineJS Architecture
Function versioning and alias management
Technology stack:
BeamlineJS expects that the lambda functions that it are going to build, test, release, and deploy are production ready. It requires client functions to utilize the following technology stack to ensure that the lambda functions are production ready.
Try out BeamlineJS -
Code of BeamlineJS is 100% open source. Feel free to clone below repositories and make them your own. I am a beginner in node.js, so if you do any improvements to the code in below repositories please feel free to submit a pull request or send me your Git repository URL.
You can refer to this sample lambda function to create your lambda function’s project structure and package.json. Beamline requires use of a specific Git branching model, flow, and other code quality tools.