Category: AWS

Building Workflows with AWS Lambda and Step Functions

Step Functions and Lambda are two key services within the AWS ecosystem that work seamlessly together to create complex, stateful workflows.

What are Step Functions?

  • A visual workflow service: Step Functions allows you to define workflows as a series of steps. Each step can be a task, a choice, or a wait.
  • State machines: These workflows are defined as state machines, which can be executed and managed.
  • Integration with other AWS services: Step Functions integrates with a wide range of AWS services, including Lambda, ECS, and DynamoDB.

What is Lambda?

  • Serverless compute service: Lambda lets you run code without provisioning or managing servers.
  • Event-driven: Lambda functions are triggered by events, such as API calls, file uploads, or messages from other AWS services.
  • Scalable: Lambda automatically scales to meet demand, ensuring your applications can handle varying workloads.

How Step Functions and Lambda Work Together

  • Lambda as a task: One of the most common use cases is to use Lambda functions as tasks within a Step Functions state machine. When a state machine is executed, the Lambda function associated with that step is invoked.
  • Input and output: Step Functions can pass data between steps, allowing Lambda functions to process and transform data as it flows through the workflow.
  • Error handling: Step Functions provides built-in error handling mechanisms, such as retries and catch blocks, to ensure that your workflows are resilient.

Benefits of Using Step Functions and Lambda

  • Simplified development: Step Functions provides a visual interface for creating and managing workflows, making it easier to build complex applications.
  • Scalability: Lambda’s serverless architecture ensures that your applications can scale to meet demand without requiring manual provisioning of resources.
  • Cost-effectiveness: Step Functions and Lambda are pay-as-you-go services, meaning you only pay for the resources you use.
  • Integration with other AWS services: Step Functions’ ability to integrate with a wide range of AWS services makes it a versatile tool for building complex applications.

Example use case: A common use case is building a data processing pipeline. The pipeline might involve:

  1. Ingesting data from a source like S3 or a database.
  2. Transforming the data using Lambda functions.
  3. Storing the processed data in a destination like S3 or DynamoDB.

In Step Functions, you define your workflow in JSON format. Here’s a simplified example:

{
  "Comment": "An example of a simple order process",
  "StartAt": "Check Inventory",
  "States": {
    "Check Inventory": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account-id:function:CheckInventory",
      "Next": "Process Payment"
    },
    "Process Payment": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account-id:function:ProcessPayment",
      "Next": "Notify Customer"
    },
    "Notify Customer": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account-id:function:NotifyCustomer",
      "End": true
    }
  }
}

Step Functions can be used to define the workflow, with each step representing a specific task or decision point. Lambda functions can be used to perform the actual data processing.

AWS Doc: https://aws.amazon.com/step-functions/

How to access Load Balancer logs

AWS provides access logs for Elastic Load Balancers (ELB), allowing you to monitor and analyze traffic patterns. Below are general steps to access and view ELB access logs:

Amazon ELB Access Logs:

  1. Navigate to the EC2 Console:
    • Open the AWS EC2 Console.
  2. Select Load Balancers:
    • In the left navigation pane, choose “Load Balancers” under the “Load Balancing” section.
  3. Choose Your Load Balancer:
    • Click on the name of the load balancer for which you want to access logs.
  4. View Access Logs:
    • In the “Description” tab, look for the “Attributes” section.
    • Check if the “Access logs” attribute is set to “Enabled.”
  5. Access Logs Location:
    • If access logs are enabled, you can find them in an S3 bucket specified in the “S3 bucket” field.
  6. Navigate to S3 Bucket:
    • Open the AWS S3 Management Console.
    • Access the S3 bucket mentioned in the “S3 bucket” field.
  7. Access Log Files:
    • Inside the S3 bucket, you should find log files with a naming convention like <load-balancer-name>/<YYYY>/<MM>/<DD>/....
    • Download or view the log files to analyze the access logs.

AWS CLI:

You can also use the AWS Command-Line Interface (CLI) to access ELB access logs:

# Replace <your-load-balancer-name> and <your-s3-bucket-name> with your actual values

aws s3 cp s3://<your-s3-bucket-name>/<your-load-balancer-name>/<path-to-log-file> .

This command downloads the specified log file to the current directory.

Analyzing Access Logs:

Access logs typically include information such as client IP addresses, request timestamps, response status codes, and more. You can use tools like AWS Athena, Amazon CloudWatch Logs Insights, or other log analysis tools to query and visualize the logs.

Remember to adjust the steps based on the specific type of load balancer you are using (Application Load Balancer, Network Load Balancer, or Classic Load Balancer). Always refer to the official AWS documentation for the most accurate and up-to-date information.

These logs are much helpful if you are looking from which instance the request is coming.

Send Email with Attachment File from Lambda (Nodejs)

You can create a lambda (nodejs) with the following code (written in typescript) to send an email to a user with a file attachment.

In the below example, we are first getting the content from S3 bucket, then creating a csv and sending it to a user (SES verified user).

import {
    S3Client,
    GetObjectCommand
  } from "@aws-sdk/client-s3";
  import { SESClient, SendRawEmailCommand } from "@aws-sdk/client-ses";
  import { Readable } from "stream";
  const s3Client = new S3Client({ region: "ap-southeast-2" });
  const sesClient = new SESClient({ region: "ap-southeast-2" });


  export const sendEmail = async () => {
    const senderEmail = process.env.SENDER_EMAIL_ADDRESS;
    const recipientEmail: any = process.env.RECIEVER_EMAIL_ADRESS;
    const subject = "SUBJECT here";
    const bodyText =
      "Hello,\r\n\nPlease see the attached csv file \n\nThanks";
  
    const getObjectCommand = new GetObjectCommand({
      Bucket: process.env.BUCKET,
      Key: process.env.BUCKET_KEY,
    });
  
    const attachmentData = await s3Client.send(getObjectCommand);
    const attachmentBuffer = await streamToBuffer(
      attachmentData.Body as Readable
    );
  
    const attachmentBase64 = attachmentBuffer.toString("base64");
  
    const emailData =
      `From: ${senderEmail}\r\n` +
      `To: ${recipientEmail}\r\n` +
      `Subject: ${subject}\r\n` +
      `MIME-Version: 1.0\r\n` +
      `Content-Type: multipart/mixed; boundary="boundary"\r\n\r\n` +
      `--boundary\r\n` +
      `Content-Type: text/plain; charset=utf-8\r\n` +
      `${bodyText}\r\n\r\n` +
      `--boundary\r\n` +
      `Content-Type: application/octet-stream\r\n` +
      `Content-Disposition: attachment; filename="file.csv"\r\n` +
      `Content-Transfer-Encoding: base64\r\n\r\n` +
      `${attachmentBase64}\r\n\r\n` +
      `--boundary--`;
  
    const sendRawEmailCommand = new SendRawEmailCommand({
      RawMessage: {
        Data: Buffer.from(emailData),
      },
      Source: senderEmail,
      Destinations: [recipientEmail],
    });
  
    const result = await sesClient.send(sendRawEmailCommand);
    return result.MessageId;
  }

  async function streamToBuffer(stream: Readable): Promise<Buffer> {
    return new Promise((resolve, reject) => {
      const chunks: Uint8Array[] = [];
      stream.on("data", (chunk) => chunks.push(chunk));
      stream.on("end", () => resolve(Buffer.concat(chunks)));
      stream.on("error", reject);
    });
  }

How to connect to SQL database from Lambda using Nodejs

To connect to SQL database from lambda, I am using Nuget package ‘mssql’.

For this, I have created a serverless application using Nodejs 16.x.

To connect to SQL database from lambda, I am using Nuget package ‘mssql’.

In your serverless application, install mssql like below –

npm install mssql

Usage –

I am using Typescript and DBConfig used above is just an interface.
Also replace all the config with your database credentials.

import sql from "mssql";

const config : DBConfig = {
 user: ${process.env.DB_USER},
 password: ${process.env.DB_PASSWORD},
 server: ${process.env.DB_SERVER},
 database: ${process.env.DATABASE},
 options: {
  trustServerCertificate: true,
 },
};

export const run = async () => {
 try {
  await sql.connect(config);
  const result = await sql.query`Select * from [TableName]`;
  var result = result.recordset;
}
catch (err) {
 console.error("Error:", err);
 return {
  statusCode: 500,
  body: JSON.stringify({
  message: "Error accessing the database.",
  error: err,
  }),
 };
}
}

Please note that the RDS and lambda should be in same VPC settings. If while running lambda, you get a timeout error, do verify the VPC settings for both RDS and lambda. If they are in different VPC, then you have to do further settings. Please refer to AWS documentation for that.

Also make sure the IAM role in lambda should have permission to access RDS

Refer to this if you are looking to send an email with attachment from lambda.

Export Cognito Users from AWS Using AWS CLI

Export Cognito Users in AWS

As of today, there is no way to directly export users from Cognito in AWS.

But we can use AWS CLI or AWS SDK to get the list of users.

  • First step is to install AWS CLI on your machine

Click here to download and install AWS CLI

  • Next, step is to set up AWS on your machine. To do this, open cmd (command prompt) and do the following –
$ aws configure
AWS Access Key ID [None]: YourAccessKeyId
AWS Secret Access Key [None]: YourSecretAccessKey
Default region name [None]: YourRegion
Default output format [None]: json

Replace the above with your values. For more info, click here.

  • To get the list of all users in Cognito, run the following command
aws cognito-idp list-users --region <region> --user-pool-id <userPoolId> 
--output json > users.json

The above will return the list of users in a JSON format. If you want to get the result in a table format, run the next command

aws cognito-idp list-users --region <region> --user-pool-id <userPoolId>  --output table > users.txt
  • Now if you want to convert the result json to csv. Use the following code snippet.
private static void ConvertJsonToCsv()
{
 using (var csv = new ChoCSVWriter("users.csv").WithFirstLineHeader())
 {
   using (var json = new ChoJSONReader("CognitoUsers.json")
             .WithField("Username")
             .WithField("Email", jsonPath: "$.Attributes[?(@.Name == 
              'email')].Value", isArray: true)
             .WithField("UserStatus")
         )
  {
     csv.Write(json);
  }
 }
}