I need to run some python code on aws platform periodically(probably once a day). Program job is to connect to S3, download some files from bucket, do some calculations, upload results back to S3. This program runs for about 1 hour so I cannot make use of Lambda function as it has a maximum execution time of 900s(15mins).
I am considering to use EC2 for this task. I am planning to setup python code into a startup and execute it as soon as the EC2 instance is powered on. It also shuts down the instance once the task is complete. The periodic restart of this EC2 will be handled by lambda function.
Though this a not a best approach, I want to know any alternatives within aws platform(services other than EC2) that can be best of this job.
If you are looking for other solutions other than lambda and EC2 (which depending on the scenario it fits) you could use ECS (Fargate).
It’s a great choice for microservices or small tasks. You build a Docker image with your code (Python, node, etc…), tag it and then you push the image to AWS ECR. Then you build a cluster for that and use the cloudwatch to schedule the task with Cloudwatch or you can call a task directly either using the CLI or another AWS resource.
- You don’t have time limitations like lambda
- You don’t also have to setup the instance, because your dependencies are managed by Dockerfile
- And, if needed, you can take advantage of the EBS volume attached to ECS (20-30GB root) and increase from that, with the possibility of working with EFS for tasks as well.
I could point to other solutions, but they are way too complex for the task that you are planning and the goal is always to use the right service for the job
Hopefully this could help!