💁Hosting a Python FlaskAPI developed using OpenApi/Swagger definition in the Cloud

A getting started example for hosting a serverless API in cloud (AWS API Gateway / AWS Lambda) developed using OpenApi/Swagger definition, Python Flask and AWS SAM

This repository can be a good starting point for various usecases like

1. Hosting a serverless AWS lambda function with a simple-medium complexity data processing logic exposed via REST Api
2. Get insights / score / data from backend AWS resources like Elasticache, SageMaker Endpoint, DynamoDB etc.,
3. Integration point with other AWS Services

⚠️Things to Remember

1. AWS Lambda functions should run within HIG VPC
2. AWS API Gateway should only be a Private API deployed within a HIG VPC
3. No PII data allowed in a NonProd AWS account

Key Topics Covered in this Example

1. Python - Flask API development using OpenAPI/Swagger definition
2. Unit testing
3. PreCommit hook for automated unit testing, formatting before even committing the code - "Shift-left" approach
4. Setting up security scanning pipeline using Jenkins - NexusIQ - Sonar
5. Writing Infrastructure as Code for deploying in the cloud
6. Utilizing Service Catalog products
7. Setting up cloud native CICD pipeline for application deployment

🔗 Quick Links

• Solution Architecture
• Development Setup
• Files and Folders
• Run Locally
• Adding a new resource to the API
• Unit Test
• Security Scanning
• Deploying to cloud
• Using the API
• References
• FAQs
• Support

🛠️ Solution Architecture

💻Development Setup

1. Star this{:target="_blank"} repo
2. Clone this repo directly or fork and clone the fork

   git clone hhttps://github.com/vas610/openapi_with_python.git
   cd openapi_with_python

3. Run the following commands

   python3 -m venv venv
   source venv/bin/activate
   pip3 install -r requirements.txt
   pre-commit install
   cd api_v1
   pip3 install -e .

📂Files and Folders

.
├── api_v1  # API Version
│   ├── api  # This folder contains the flask app, open api definition, backend resource
│   │   ├── __init__.py
│   │   └── people.py  # Backend code for the resource `people` in the api
│   ├── coverage.xml
│   ├── flask_serverless.py  # Boilerplate to host the flask app in AWS Lambda
│   ├── __init__.py
│   ├── openApi  # API Definition
│   │   ├── main.yml  # Main Definition of the API
│   │   └── schemas
│   │       ├── people.yml  # Schema for the resource `people`
│   │       └── responses.yml
│   ├── requirements.txt  #  Python dependencies for the flask app to run successfully in AWS Lambda
│   ├── server.py  # Boilerplate flask app
│   ├── setup.py
│   ├── templates
│   │   └── home.html  # Home page HTML
│   └── tests  # Unit test
│       ├── conftest.py
│       └── test_app_test.py
├── buildspec.yml
├── coverage.xml
├── docs
│   ├── images
│   │   └── sol-arch.png
│   └── index.html
├── README.md
├── requirements.txt  #  Python dependencies for successful local development of the API
├── template.yml  # AWS SAM IaC (Infrastructure as Code) template for AWS API Gateway & AWS Lambda

💿Run locally

1. Go to api_v1 directory
2. Type the following and hit enter

flask_server

3. Application should launch in port 8080 by default
4. Change the port if needed by editing the app_run function in the api_v1/server.py file
5. Refer below for screenshots regarding navigating the swagger/OpenAPI UI

🚀Adding a new resource to the API

1. First, add a new resource in the api_v1/openApi/main.yml similar to /people/get/{lname} and/or /people/post/create
2. If the new resource has a post method similar to /people/post/create then create a new schema for the post request body similar to api_v1/openApi/schemas/people.yml and save it under api_v1/openApi/schemas/
3. Create a new backend resource similar to api_v1/api/people.py and store it in [api_v1/api/]
4. Go back to the new resource added in the api_v1/openApi/main.yml and update the x-openapi-router-controller to point to the newly created backend resource and operationId to the corresponding python method / function in the newly created backend resource
5. Update the unit test api_v1/tests/test_app_test.py similar to api_v1/tests/test_app_test.py to test the newly created resource
6. Run locally based on the instruction provided above to ensure everything is working as expected
7. Run unit test based on the instruction provided below and then commit and push the changes to the remote GitHub repo

🧪Unit test

Run unit test before pushing to GitHub repo

cd .. # go to repo root dir - ds_mlops_openapi_on_aws
pytest api_v1/tests/ -W ignore::DeprecationWarning --cov-report term-missing --cov-report html:coverage_html --cov-report xml:coverage.xml --cov=api

☁️Deploying to cloud

6. Once the product is stood up successfully, go to the corresponding AWS CodeBuild and hit start build
7. Once the build is done successfully, CodePipeline should run automatically and deploy this API
8. This an example implementation. People created using the /people/post/create resource may or may not exist in the subsequent /people/get/{lname} GET calls as the API is deployed in an AWS Lambda Function that has ephermeral memory. For sceniors like this in real use cases, a database like Aurora DB would be required in the backend.

✨Using the API:

Once the API and Swagger/OpenAPI UI is up and running in cloud, it can be used for marketing your API with your clients and can be used by client developers to design / write their API calls

1. Get the URL to access the API and the API KEY by navigating to the CloudFormation{:target="_blank"} template's Output section in the target account

   • Here target account refers to the account in which the API is deployed. This can be discover/NonProd or PreProd/QA or Prod
   • Login to the account using HIGDataScientist or HIGDeveloper persona
   • Filter Stack by using the Stack Suffix that you entered while creating the Service Catalog product
   • Do not share API Key in Prod account for adhoc usage
   • Refer to the screenshot [below][#cloudformation-stack-output]

2. Navigate to the API home page using the above url

3. Click on a resource that you want to explore

4. To Authorize, use the key from above

5. Explore various resources in the API

6. Take the curl command that the OpenAPI/Swagger UI provides, code similar to that in the language of your choice and start accessing the API programmatically

7. Refer below for screenshots regarding navigating the swagger/OpenAPI UI

📸References

1. Navigating Swagger/OpenAPI UI

   • There is no difference in how you navigate the UI irrespective where this application is running - local / cloud

   • ℹ️ For local, you can use any random string as Auth key

   • ⚠️For cloud, you should use the API KEY obtained from the step above or contact the development team

   • Home Page

     {:target="_blank"}

   • Choose a resource

     {:target="_blank"}

     {:target="_blank"}

   • Authorize

     {:target="_blank"}

     {:target="_blank"}

   • Explore

     {:target="_blank"}

2. APIGateway VPC Endpoint

   {:target="_blank"}

3. Setup SAM Pipeline using Service Catalog Product

   {:target="_blank"}

4. CloudFormation Stack Output

   {:target="_blank"}

❓FAQs

1. What is a RESTful API?

   What is REST?

2. What is OpenAPI / Swagger Specification?

   OpenAPI Specification

3. How can I learn writing Swagger Specification ?

4. What is AWS SAM?

   AWS Serverless Application Model (AWS SAM)

📧Support