Ahmed Nuaman, Managing Director at Firemind, followed with a brief intro of Fireminds practices, and guidance available throughout the day.

As the workshop revolved around the attendees data, we began by looking at data ingestion and the pre-prepped data the attendees had placed into Amazon S3 buckets. Once data is ingested into an S3 bucket, it becomes immediately available for storage, retrieval, and further processing. S3 provides high durability, scalability, and availability for data, making it suitable for a wide range of data ingestion scenarios, including data backups, data lakes, log storage, and data archiving.

Once the data was prepped, attendees began the AWS Glue portion of the workshop. AWS Glue is a fully managed extract, transform, and load (ETL) service provided by AWS. It helps automate the process of preparing and transforming data for analysis, making it easier to discover insights and derive value from ingested data stored in Amazon S3 buckets.

When working with ingested data in an S3 bucket, AWS Glue offers the following functionalities:

Data Catalog: AWS Glue creates and maintains a centralised metadata repository called the Data Catalog. It catalogs the structure, schema, and location of your data in S3. The Data Catalog acts as a single source of truth for metadata, making it easier to discover and understand the ingested data.

Crawlers: AWS Glue Crawlers automatically scan the S3 bucket and infer the schema and structure of the data. Crawlers can identify file formats, partitioning schemes, and extract metadata, which are stored in the Data Catalog. This automated discovery process saves time and effort by eliminating the need to manually define schemas.

ETL Jobs: AWS Glue enables attendees to create ETL jobs to transform the ingested data into a format suitable for analysis. You can use a visual interface or code (Python or Scala) to define the transformations. ETL jobs can perform operations such as filtering, aggregating, joining, and cleaning the data. The transformed data can be stored back in S3 or loaded into other data stores or analytics services.

Data Lake Formation: AWS Glue integrates with AWS Lake Formation, which provides additional capabilities for managing and securing data lakes. It helps define and enforce fine-grained access control policies, data transformations, and data retention policies on the ingested data in S3.

The final stage of the workshop was visualising and exploring the data using Amazon QuickSight. With QuickSight, all users can meet varying analytic needs from the same source of truth through modern interactive dashboards, paginated reports, embedded analytics, and natural language queries.