Skip to main content

How to setup Cross Account in AWS S3?

Pre-requisites

The following section describes the permissions required for S3 setup.

IAM permission: Attach IAM policy where cloud IDP is installed to get role detail.

{

              "Version": "2012-10-17",

              "Statement": [{

              "Sid": "VisualEditor0",

              "Effect": "Allow",

              "Action":

 [

              "iam:ListPolicies",

              "iam:GetRole",

              "iam:GetPolicyVersion",

              "iam:PassRole",

              "iam:GetPolicy",

              "iam:ListAttachedRolePolicies",

              "iam:ListRoles",

              "iam:GetRolePolicy"

],

"Resource": "*"

}]

}

Minimum Roles for Service: S3

{

              "Version": "2012-10-17",

              "Statement": [{

              "Sid": "VisualEditor0",

              "Effect": "Allow",

              "Action":

[

              "s3:PutObject",

              "s3:GetObject",

              "s3:ListAllMyBuckets",

              "s3:ListBucket",

              "s3:DeleteObject"

],

"Resource": "*"

}]

}

Minimum Roles for Service: EMR

{

"Version": "2012-10-17",

"Statement": [{

"Sid": "VisualEditor0",

"Effect": "Allow",

"Action":

[

"elasticmapreduce:DescribeSecurityConfiguration",

"elasticmapreduce:ListInstances",

"elasticmapreduce:ListSecurityConfigurations",

"elasticmapreduce:ListSteps",

"elasticmapreduce:SetVisibleToAllUsers",

"elasticmapreduce:PutAutoScalingPolicy",

"elasticmapreduce:DescribeCluster",

"elasticmapreduce:RunJobFlow",

"elasticmapreduce:SetTerminationProtection",

"elasticmapreduce:TerminateJobFlows",

"elasticmapreduce:CancelSteps"

],

"Resource": "*"

}]

}

Setup Cross Account in AWS

The following section explains the step-by-step process to setup Cross Account in AWS.

Let us suppose you have two accounts:

  • M - Master Account, where our EMR cluster is provisioned. Account id is ‘12345678901’.

  • O- Second account we need to access bucket. Account id is ‘32345678902’.

  1. The following points will configure policy and trusted entity

    1. Create role (O_Bucket_access_role) in account ‘O’.

    2. Attach policy to access buckets.

    3. Add account ‘M’ as trusted entity with this role:

      Policy to access buckets: Let us suppose we need to access ‘O_bucket_1’ bucket from second account. Here is the policy you may need to create in this account. For now we have given full access to this policy on this bucket.

      {

      "Version": "2012-10-17",

      "Statement": [{

      "Effect": "Allow",

      "Action": "s3:*",

      "Resource": [

      "arn:aws:s3:::O_bucket_1",

      "arn:aws:s3:::O_bucket_1/*"

      ]

      }]

      }
      Policy in AWS: Add account ’M’ as trusted entity with this role:

       Policy to add trusted relationship:

       {

              "Version": "2012-10-17",

               "Statement": [{

                "Effect": "Allow",

                 "Principal": {

                  "AWS": "arn:aws:iam::12345678901:root"

         },

                    "Action": "sts:AssumeRole",

                    "Condition": {}

                    }]

               }
    Go to the role, click on ‘ Trust Relationship’ :
    Click ‘ Edit Trust Relationship ’.
    You can create multiple roles with same configuration to access from account ‘M’.

  2. Account ‘M’ Setup:
    Create a role in this account having access to the roles created in account ‘O’. Here is the policy for this role (‘M_cross_account_role’):

    {

           "Version": "2012-10-17",

            "Statement": [{

             "Sid": "VisualEditor0",

              "Effect": "Allow",

               "Action": "sts:AssumeRole",

               "Resource": [

                "arn:aws:iam::32345678902:role/O_Bucket_access_role"

                  ]

                  }]

         }
    User can add multiple roles in “Resource” with comma separated values as in json array. Create a security configuration or add policy in bucket: Either you can create a security config and attach with EMR cluster or You can add policy and give access.

    EMR ( Pick only one option)

    1. Create a security configuration

      1. In ‘M’ account us AWS CLI as currently we don’t have any option in UI. Follow below steps to configure security configuration

      2. Create a json file named security_config.json :
        {

                                     "AuthorizationConfiguration": {

                                     "EmrFsConfiguration": {

                                     "RoleMappings": [{

                                     "Role":

                                     "arn:aws:iam::32345678902:role/O_Bucket

                                     _access_role",

                                     "IdentifierType": "Prefix",

                                     "Identifiers": [

                                     "s3://O_bucket_1",

                                     "s3://O_bucket_1/*"

                                                   ]

                                     }]

                      }

        }

        }

      3. Run this command in AWS CLI to create security configuration named (EMR_cross_account_security_config ) Set region and create security configuration in that region aws configure set region us-west-1 aws emr create-securityconfiguration --name EMR_cross_account_security_config --security-configuration file://tmp/security_config.json

      4. You can view Security Configuration in AWS console under EMR service view:

    2. Add policy in bucket : add below policy directly in S3 permissions

      1. Go to S3 bucket.

      2. Go to permission tab.

      3. Select bucket policy.

      4. Add this json after replacing tokens

        {

                      "Version": "2012-10-17",

                      "Statement": [{

                      "Sid": "EMRCrossAccount",

                      "Effect": "Allow",

                      "Principal": {

                      "AWS":

                      "arn:aws:iam::MASTERACCOUNTID:role/MASTER

                      ACCOUNTROLENAME"

                      },

                      "Action": [

                                    "s3:GetObject",

                                    "s3:GetObjectVersion",

                                    "s3:GetObjectAcl",

                                    "s3:GetObjectVersionAcl",

                                    "s3:PutObject",

                                    "s3:PutObjectAcl",

                                    "s3:PutObjectVersionAcl",

                                    "s3:DeleteObject",

                                    "s3:DeleteObjectVersion",

                                    "s3:ListBucket",

                                    "s3:GetBucketAcl",

                                    "s3:GetBucketPolicy",

                                    "s3:GetBucketTagging",

                                    "s3:PutBucketTagging",

                                    "s3:GetObjectVersionTagging",

                                    "s3:PutObjectVersionTagging",

                                    "s3:PutObjectTagging"

                      ],

        "Resource": [

                                     "arn:aws:s3:::TARGETACCOUNTBUCKETNAME/*",

                                     "arn:aws:s3:::TARGETACCOUNTBUCKETNAME"

                      ]

        }]

        }

  3. Provision EMR clustera.      

    1. Go into EMR AWS service and click on Create Cluster. Click on Go to advanced options link on above screen in following screen:

    2. Select version of EMR (Make sure version should be > 5.0). I am using emr-5.23.0. Click Next:

    3. Select fields as per your requirement and click next.

    4. Set cluster name and other relevant properties and click next.

    5. Select EC2 key pair, you have created earlier as shown in below screenshot. Important step is to set EC2 instance profile and Security configuration.

      1. Select same role you have created in Step2.1 named ‘M_cross_account_role ’.

      2. Select security configuration you have created in step2.2 named EMR_cross_account_security_config then Click Next.

    6. In final screen after creating EMR cluster. You can verify role and security configuration on next page.

Setup in HDFS S3

The following section provides a step-by-step process to setup Cross Account in HDFS S3.

  1. Upload HDFS S3 IDP in EMR cluster you have provisioned with required cross account setup.

  2. Change value of property cloud.hadoop.api.for.preprocessing to Y in config file at this location in HDFSAgent ‘ /HDFSAgent/expandedArchive/WEBINF/classes/com/dataguise/hadoop/util/HDFSAgentConfig.properties’. After above changes, start HDFS IDP.

Cross Account Setup in PK Protect

The following section elaborates the step-by step process to set up Cross Account in PK Protect.

  1. Add cloud IDP and S3 IDP in Admin.

  2. Click Manage Cluster/Fileshare and create a cluster for S3:

  3. Select a s3 cluster, you have created in step2 and click Cross Account ARN button.

  4. A pop-up appears where you can enter role name and load all Cross Account role name linked to entered role name: You can manually enter role in this popup, if you know other account role ARN and linked role name in master account.

  5. Let us try to load cross account role.

    1. Enter role name and click Load.

    2. If cross account role exist, it will load role names in panel.

    3. Click on Checkbox under Verify column, you need to use for Cross Account access and click save.

    4. We have selected one ARN and Cross Account setup for S3 browser is complete. All this detail will be now saved in dgcontroller db in encrypted form.

  6. Enter role Manually.

    1. Enter values in Add AWS Account panel.

    2. Click on the add button and check Checkbox under Verify column and click save. Your manual entered ARN will be saved in db with selected S3 cluster.

    3. Cross account setup for S3 browser is complete. All this detail will be now saved in

Applying Cross Account Setup in PK Protect

  1. In PK Protect, create a new task in AWS S3 component. Click on Select Bucket button. A popup appears to select buckets.

  2. There is a drop down for Cross account at top row in pop-up. Default account bucket will get automatically listed in left and right div of popup as shown below.

  3. To load bucket from other account, select other account ARN from drop down and you will see listing for other account if you have permission to list buckets from other account.

  4. If user does not have permission to list bucket from other account but have access to view data of some of buckets. User will first see this error message:

  5. To view content of bucket you have access, you must enter bucket name in selected area in the following image:

  6. Enter bucket and click expand. You will be able to view content of entered bucket. Make sure the path entered is correct in this format. You can use any protocol you have supported in s3 (s3, s3a, s3n): s3://bucketname.

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.