Collecting, archiving and retrieving surveillance data
with AWS
As a company based within the UK, we are no stranger to the presence of CCTV cameras and the importance of them. With a reported 5.2 million cameras in the UK actively filming back in late 2020 (that’s one camera per 13 people), it’s safe to say that surveillance and it’s uses are an everyday part of most people’s lives.
Surveillance and the cloud
So how does surveillance and the endless hours of video data find its place within the cloud? How does AWS provide the best tools and services to collate, archive and retrieve surveillance data with ease? Let’s find out!
In this particular AWS insight, we’ll be looking at how we collect data from closed-circuit television that’s dispersed across a wide geographical area and then streamline the archiving and retrieval process using Amazon S3 Glacier storage classes.
Click here for an overview of Amazon S3 Glacier Storage classes.
Amazon S3 Glacier
The pivotal tool for this process to flow relies on Amazon S3 Glacier storage classes as well as the Deep Archive features. The Amazon S3 Glacier storage classes are purpose-built for data archiving. They provide businesses with the highest performance, most retrieval flexibility and the lowest cost archive storage in the cloud. All S3 Glacier storage classes provide virtually unlimited scalability and are designed for 99.999999999% (11 nines 😂 ) of data durability. S3 Glacier Deep Archive provides secure and durable object storage for long-term retention and digital preservation for surveillance and any other form of data.
In the AWS proposed solution, they address the following source data paths:
• Locations with old cameras sending feeds to a local Network Attached Storage (NAS).
• Locations with new IP-enabled camera installations.
• Remote locations with slow or unreliable networks.
Collecting video feeds from locations with old cameras
Although there are several methods to implement this solution, AWS focus on two approaches. The first is AWS Storage Gateway Hardware Appliance configured as Amazon S3 File Gateway (this is their preferred approach).
S3 File Gateway provides a way to connect to AWS in order to store application data files and backup images as durable objects in Amazon S3. S3 File Gateway offers SMB or NFS-based access to data in Amazon S3.
The second approach is to use a Network Attached Storage (NAS) from the AWS Marketplace that comes integrated with Amazon S3. The NAS vendor options in AWS Marketplace have S3 integration available, but integrating disparate NAS systems can be a daunting task. Integrating these systems can come with additional operational responsibilities like managing, monitoring and administering different NAS technologies. Let’s look at both approaches to bring data from the old (non IP-enabled) cameras to Amazon S3.
Amazon S3 File Gateway
S3 File Gateway at the source data location provides the NFS/SMB compatible network file share. This file share exposes S3 objects to the local server on premises. The CCTV cameras are reconfigured to use the network file share created by S3 File Gateway at the primary storage location. The data stored on the file then syncs to the Amazon S3 bucket via AWS Direct Connect. The lifecycle transition rule configured on Amazon S3 Standard then moves the data to S3 Glacier Deep Archive.
Click here for more information on Data collection using Amazon S3 File Gateway at source.
Collecting video feeds from locations with IP-enabled cameras
You can ingest real-time live CCTV IP camera feeds into Amazon Kinesis Video Streams (KVS) by installing the Kinesis Video Streams SDK on your devices. Using KVS Producer libraries, the video from existing IP cameras can be fed into Amazon Kinesis Video Streams with ease. KVS endpoints are available over Direct Connect/Public Virtual Interface.
Amazon Kinesis Video Streams uses Amazon S3 as the underlying data store, which means your data is stored durably and reliably. Amazon Kinesis Video Streams automatically indexes the data you store in your video streams. This indexing is based on timestamps generated by the device or timestamps generated by Kinesis Video Streams when it receives the video. You can combine stream-level tags with timestamps to search and retrieve specific video fragments for playback, analytics and other processing!
Last, but not least, collecting video feeds from locations with slow networks
To handle data collection when a network is unreliable or typically slow, the process gets a little more intricate with the use of AWS Snow Family.
This family offers a petabyte-scale data transport solution that uses secure, physical devices to move large amounts of data into and out of AWS.
You can create a job in the AWS Snow Family console to send a Snowball Edge. Once a job is created, AWS ships a Snowball Edge to your location. When it arrives at your specified destination, attach it to your local network and transfer the data using the Snowball client or the Amazon S3 Adapter for Snowball.
Once the device is ready to be returned, ship the Snowball Edge back to AWS and they’ll import your data into Amazon S3. A Snowball Edge job can support up to 80 TB of data and each job has exactly one Snowball Edge associated with it.
For multiple terabytes of data, multiple devices can be used in parallel or clustered together to transfer petabytes of data. This provides you with continued availability of data storage at the source location.
Taking the next step
To continue reading the original AWS Insight and look at the more granular steps within the AWS Snow family (such as AWS Snowcone for smaller data transport jobs), visit this link.
If you’re in a position where you have exemplary amounts of data and no idea how to collect, archive and retrieve in a secure, fast and cost efficient manner, get in touch today!
Get in touch
Want to learn more?
Seen a specific case study or insight and want to learn more? Or thinking about your next project? Drop us a message!
