Amazon Web Services Simple Storage Service (AWS S3) Volumes
Navigation
- Amazon Web Services Simple Storage Service (AWS S3) Volumes
- Attach an Amazon Web Services (AWS) volume volume using an IAM user
- Attach an Amazon Web Services (AWS) volume using an IAM role:
This page contains information on configuring a volume that is associated with an S3 bucket in Amazon Web Services' (AWS) cloud storage.
Platform authentication and authorization
BioData Catalyst powered by Seven Bridges provides the option of connecting your Amazon Web Services S3 bucket (volume) to be able to read and write files to and from the Platform. After connection is established, your S3 bucket (volume) behaves like your external storage for the Platform. Authentication of the Platform is done through AWS Identity and Access Management (IAM) services and you can choose between two authentication options, IAM user or IAM role which will be explained in detail on this page. Authorization of the Platform, which defines the actions that can be done on your volume, is defined in a custom IAM policy that is attached to an IAM user or IAM role.
About AWS IAM roles
An IAM Role is an entity, similar to an IAM user, that represents an AWS identity for which you set permission policies to define what other identity associated with your role can and cannot do with your AWS resources. For connecting BioData Catalyst powered by Seven Bridges with your S3 bucket, we recommend creating a role only for that purpose. A role does not have standard long-term credentials such as a password or access keys associated with it. Instead, when you assume a role, it provides temporary security credentials for your role session. The Platform automatically renews these IAM role temporary credentials, so authorized Platform users can use the volume without interruption.
We also use an IAM Role on our side to assume the user’s role when a connection with a volume is requested. The Seven Bridges role can be assumed only by volume services that are necessary for connecting with the user's S3 bucket.
Prerequisites for connecting an S3 bucket to the Platform
- An Amazon Web Service (AWS) account.
- One or more S3 buckets within the AWS account.
Procedure:
Create a custom IAM policy
For the IAM role authentication method, a policy should be generated and attached to the role in order to define permissions that the Platform will have when connecting with your AWS S3 bucket (volume). So, the first step is to create a custom IAM policy that you will attach to your IAM role later on. To create the policy, follow these steps:
- Go to the AWS Management Console.
- In the top menu select Services and then choose IAM.
- In the left navigation menu select Policies.
- Click Create policy and select the JSON tab.
- Copy and paste the following custom policy, taking into account the following:
- Make sure to select the right policy depending on whether you're mounting the volume in Read-only or Read-write mode.
- Make sure to replace <BUCKET_NAME> in the policy with the name of your S3 bucket.
- Either replace <ROOT> with the subdirectory in the bucket to which you want to restrict access when browsing the mounted volume from the Platform, or simply exclude the <ROOT> parameter and set
Resourcein the policy toarn:aws:s3:::<BUCKET_NAME>if you want to make the entire content of the bucket available. - If you are using multiple project locations, make sure that all regions where your analyses will be run are included within your Read-write IAM policy.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "GrantReadOnBuckets",
"Action": [
"s3:ListBucket",
"s3:GetBucketCORS",
"s3:GetBucketLocation"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::<BUCKET_NAME>"
]
},
{
"Sid": "GrantReadOnObjects",
"Action": [
"s3:GetObject"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::<BUCKET_NAME>/<ROOT>/*"
]
}
]
}
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "GrantReadOnBuckets",
"Action": [
"s3:ListBucket",
"s3:GetBucketCORS",
"s3:GetBucketLocation"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::<BUCKET_NAME>"
]
},
{
"Sid": "GrantReadOnObjects",
"Action": [
"s3:GetObject"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::<BUCKET_NAME>/<ROOT>/*"
]
},
{
"Sid": "GrantWriteOnObjects",
"Action": [
"s3:PutObject",
"s3:GetObjectAcl",
"s3:PutObjectAcl",
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::<BUCKET_NAME>/<ROOT>/*"
]
},
{
"Sid": "RequestReadOnCopySourceObjects",
"Action": [
"s3:GetObject"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::sbg-main/*",
"arn:aws:s3:::sbg-main-us-west-2/*"
]
}
]
}
- Click Next: Tags.
- (Optional) Add tags that will help you identify the policy.
- Click Next: Review.
- Enter a policy name, e.g.
sb-access-policy(remember this policy name as you will need to attach it later to your IAM role). - (Optional) Enter the policy description.
- Click Create policy to finish process of policy creation.
The custom IAM policy you have created can be attached to the IAM role you will use to connect your AWS S3 bucket with the Platform.
Set up an IAM role
Follow these steps to create an AWS IAM role that you will to connect an S3 bucket (volume) to the Platform:
- Log in to the AWS Management Console.
- In the top menu select Services and then choose IAM.
- In the left navigation menu select Roles, and then chooseCreate role.
- In the Select type of trusted entity section, choose Another AWS account.
- Enter the following values:
- Account ID: 769038186821
- Check Require External ID and enter at least 6 characters (strongly recommended).
- Click Next: Permissions.
- Use the search bar to find and select the policy you created earlier (e.g.
sb-access-policy) - Click Next: Tags.
- (Optional) Add tags to the role. These are key-value pairs that contain additional information about the IAM role and are not necessary for the process of attaching a volume to the Platform.
- Click Next: Review.
- Enter Role name and its optional description.
- Click Create role to complete the process of role creation. List of all available roles opens.
- Click the name of your newly created role to see the necessary details.
- Under the Trust relationships tab, click Edit trust relationship and replace the Policy Document with the policy shown below. Make sure to select the right policy depending on whether you have set an External ID in step 5 above. If you have configured an External ID, make sure to use its value instead of the <EXTERNAL_ID> placeholder in the policy below.
{
"Version": "2012-10-17",
"Statement": {
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::769038186821:role/volume-connection"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": "<EXTERNAL_ID>"
}
}
}
}
{
"Version": "2012-10-17",
"Statement": {
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::769038186821:role/volume-connection"
},
"Action": "sts:AssumeRole"
}
}
- Click Update Trust Policy to save the update.
If you followed all the steps described above you are ready now to connect your S3 bucket as a volume to the Platform using the newly created IAM role. The credentials you will need are:
- Role ARN - click the name of your role in the list of all roles and copy Role ARN.
- External ID - click the Trust relationships tab. In the Conditions section, this is the value of the sts:ExternalId key.
External ID is an additional security parameter that will be used to authenticate the Platform when accessing your AWS bucket. This parameter is not mandatory but is highly recommended.
Additional configuration
The IAM policy is usually sufficient to permit the Platform to access your S3 bucket. In certain situations, however, it may be necessary to set up additional configuration on the bucket itself. Read how to do this by enabling cross-origin resource sharing (CORS).
Next steps
After you have completed all prerequisites, you can attach your volume to the Platform through the visual interface or the API.