The goal of this project is to deploy a serverless application on AWS Lambda using Terraform. The serverless application will read data from an S3 bucket, process it using a custom script, and store the processed data in a DynamoDB table. The serverless application will be exposed as a RESTful API using API Gateway.

Before starting this project, you will need the following:

• An AWS account with sufficient permissions to create the infrastructure resources needed for the serverless application.
• Terraform installed on your local machine.
• Basic knowledge of AWS services like S3, DynamoDB, IAM, Lambda, and API Gateway.
• Basic knowledge of Python programming language.

The architecture of the serverless application is as follows:

• The S3 bucket holds the data for the serverless application.
• The Lambda function reads the data from the S3 bucket and processes it using a custom script.
• The processed data is stored in the DynamoDB table.
• The Lambda function is exposed as a RESTful API using API Gateway.

The Terraform configuration file main.tf defines the infrastructure resources needed for the serverless application:

# Define the provider
provider "aws" {
  region = "us-east-1"
}

# Define the S3 bucket
resource "aws_s3_bucket" "data_bucket" {
  bucket = "my-data-bucket"
}

# Define the DynamoDB table
resource "aws_dynamodb_table" "processed_data" {
  name         = "processed-data"
  hash_key     = "id"
  read_capacity = 5
  write_capacity = 5

  attribute {
    name = "id"
    type = "S"
  }

  attribute {
    name = "data"
    type = "S"
  }
}

# Define the IAM role for the Lambda function
resource "aws_iam_role" "lambda_role" {
  name = "lambda-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "lambda.amazonaws.com"
        }
      }
    ]
  })
}

# Define the IAM policy for the Lambda function
resource "aws_iam_policy" "lambda_policy" {
  name        = "lambda-policy"
  policy      = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "s3:GetObject",
          "dynamodb:PutItem"
        ]
        Effect   = "Allow"
        Resource = [
          "${aws_s3_bucket.data_bucket.arn}/*",
          "${aws_dynamodb_table.processed_data.arn}"
        ]
      }
    ]
  })
}

# Attach the IAM policy to the IAM role
resource "aws_iam_role_policy_attachment" "lambda_attachment" {
  policy_arn = aws_iam_policy.lambda_policy.arn
  role       = aws_iam_role.lambda_role.name
}

# Define the Lambda function
resource "aws_lambda_function" "process_data" {
  function_name = "process-data"
  role          = aws_iam_role.lambda_role.arn
  handler       = "process_data.lambda_handler"
  runtime       = "python3.8"
  timeout       = 30
  memory_size   = 128

  # Package the Lambda function and dependencies into a zip file
  filename      = "process_data.zip"
  source_code_hash = filebase64sha256("process_data.zip")
}

# Define the API Gateway
resource "aws_api_gateway_rest_api" "serverless_api" {
  name        = "serverless-api"
  description = "RESTful API for the serverless application"
}

# Define the API Gateway resource
resource "aws_api_gateway_resource" "serverless_resource" {
  rest_api_id = aws_api_gateway_rest_api.serverless_api.id
  parent_id   = aws_api_gateway_rest_api.serverless_api.root_resource_id
  path_part   = "process-data"
}

# Define the API Gateway method
resource "aws_api_gateway_method" "serverless_method" {
  rest_api_id   = aws_api_gateway_rest_api.serverless_api.id
  resource_id   = aws_api_gateway_resource.serverless_resource.id
  http_method   = "POST"
  authorization = "NONE"
}

# Define the API Gateway integration for the Lambda function
resource "aws_api_gateway_integration" "serverless_integration" {
  rest_api_id = aws_api_gateway_rest_api.serverless_api.id
  resource_id = aws_api_gateway_resource.serverless_resource.id
  http_method = aws_api_gateway_method.serverless_method.http_method
  type        = "AWS_PROXY"
  uri         = aws_lambda_function.process_data.invoke_arn
}

# Define the API Gateway deployment
resource "aws_api_gateway_deployment" "serverless_deployment" {
  rest_api_id = aws_api_gateway_rest_api.serverless_api.id
  stage_name  = "prod"
}

# Output the API Gateway endpoint URL
output "api_gateway_url" {
  value = aws_api_gateway_deployment.serverless_deployment.invoke_url
}

The configuration file defines the following resources:

• Provider: The AWS provider is defined, with the region set to us-east-1.
• S3 Bucket: An S3 bucket is defined, with the name my-data-bucket.
• DynamoDB Table: A DynamoDB table is defined, with the name processed-data, and a hash key of id.
• IAM Role: An IAM role is defined for the Lambda function, with the name lambda-role.
• IAM Policy: An IAM policy is defined for the Lambda function, with permissions to read from the S3 bucket and write to the DynamoDB table.
• Lambda Function: A Lambda function is defined, with the name process-data, and configured to read data from the S3 bucket, process it using a custom script, and write the processed data to the DynamoDB table.
• API Gateway: An API Gateway is defined, with the name serverless-api, and configured to expose the Lambda function as a RESTful API.
• API Gateway Resource: An API Gateway resource is defined, with the path part process-data.
• API Gateway Method: An API Gateway method is defined, with the HTTP method set to POST, and no authorization required.
• API Gateway Integration: An API Gateway integration is defined, to integrate the Lambda function with the API Gateway.
• API Gateway Deployment: An API Gateway deployment is defined, to deploy the API Gateway to the prod stage.
• Output: An output is defined to display the API Gateway endpoint URL.

To deploy the serverless application, follow these steps:

1. Create a directory for the Terraform configuration file, and navigate to the directory:

mkdir serverless-app
cd serverless-app

2. Create a file named main.tf, and copy the Terraform configuration file contents shown above into the file.

3. Create a Python script named process_data.py, and copy the following code into the script:

import boto3
import json

s3 = boto3.client('s3')
dynamodb = boto3.client('dynamodb')

def lambda_handler(event, context):
    # Get the S3 bucket and key from the event
    bucket_name = event['Records'][0]['s3']['bucket']['name']
    object_key = event['Records'][0]['s3']['object']['key']

    # Read the data from the S3 object
    response = s3.get_object(Bucket=bucket_name, Key=object_key)
    data = response['Body'].read().decode('utf-8')

    # Process the data
    processed_data = data.upper()

    # Write the processed data to the DynamoDB table
    item = {
        'id': {'S': object_key},
        'data': {'S': processed_data}
    }
    dynamodb.put_item(TableName='processed-data', Item=item)

    # Return a success message
    return {
        'statusCode': 200,
        'body': json.dumps('Data processed successfully!')
    }

4. Zip the process_data.py script and its dependencies into a zip file named process_data.zip:

zip -r process_data.zip process_data.py

5. Initialize the Terraform configuration:

terraform init

6. Plan the Terraform deployment:

terraform plan

7. Apply the Terraform deployment:

terraform apply

8. After the deployment is complete, the API Gateway endpoint URL will be displayed as an output. Use this URL to access the serverless application.

In this project, we learned how to deploy a serverless application on AWS Lambda using Terraform. We defined infrastructure resources like S3, DynamoDB, IAM, Lambda, and API Gateway using Terraform configuration files. We also created a custom Python script to process the data and store it in a DynamoDB table. Finally, we deployed the serverless application using Terraform, and exposed it as a RESTful API using API Gateway.