Skip to content

Instantly share code, notes, and snippets.

@GZ315200

GZ315200/AWS学习指南.md

Last active October 11, 2018 06:51
Show Gist options
  • Save GZ315200/7971666ed57affb5a47db29f80a875ee to your computer and use it in GitHub Desktop.
Save GZ315200/7971666ed57affb5a47db29f80a875ee to your computer and use it in GitHub Desktop.
Download ZIP
AWS学习指南
Raw
AWS学习指南.md

AWS 初始化私钥配置

准备工作

  1. 安装AWS CLI命令行 ，如果你的主机已经装好Python ，已经有pip 的命令行 可执行 sudo pip install awscli
  2. 拥有自己的AWS账号，向管理员询求账号，密码
  3. Maven构建Java项目，项目名称aws_learning_guide，并引入AWS SDK for Java 其 maven dependency

配置私钥和公钥

  • 首先登陆AWS主页后， 点击右上角点击账户名 --> 我的安全凭证 --> 用户 -->选择需要配置的用户名 --> 进入配置用户信息界面-->选择安全证书项-->点击创建访问密钥
  • 生成accessKeys.csv文件到指定的本机文件夹。
  • 使用命令 aws confiure 进行本地 Access key ID 和 Secret access key 配置
  • 配置完成可访问相应AWS 的组建的相关API 接口

S3学习指南

入门指南

https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/gsg/s3-gsg.pdf

开发人员指南

https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/dev/s3-dg.pdf

从AWS CLI使用双堆栈终端节点

use_dualstack_endpoint = true时 S3 AWS CLI命令发出的所有请求定向到指定区域的双堆栈终端节点。 可以使用 --region选项指定区域 AWS CLI使用双堆栈终端节点时，支持path和virtual寻址类型。其控制着主机名或URL中是否包含存储桶名称。

设置方式如下：

$ aws configure set default.s3.use_dualstack_endpoint true 
$ aws configure set default.s3.addressing_style virtual

使用s3命令的-- enpoint-url 对每条命令使用参数设置为https://s3-ap-northeast-1.amazonaws.com来对每条命令使用双堆栈终端节点。

AWS SDK for Java双堆栈终端节点示例

import com.amazonaws.AmazonServiceException;  
import com.amazonaws.SdkClientException;  
import com.amazonaws.auth.profile.ProfileCredentialsProvider;  
import com.amazonaws.services.s3.AmazonS3;  
import com.amazonaws.services.s3.AmazonS3ClientBuilder;  
import com.amazonaws.services.s3.model.ObjectListing;  
  
public class DualStackEndpoints {  
  
  public static void main(String[] args) {  
  String clientRegion = "ap-northeast-1";  
  String bucketName = "aws-lambda-test-etl";  
  try {  
  AmazonS3 s3Client = AmazonS3ClientBuilder.standard()  
 .withCredentials(new ProfileCredentialsProvider())  
 .withRegion(clientRegion)  
 .withDualstackEnabled(true)  
 .build();  
  ObjectListing objectListing = s3Client.listObjects(bucketName);  
  //           list all objects in this bucket  
  objectListing.getObjectSummaries()  
 .forEach(s3ObjectSummary -> {  
  System.out.println(s3ObjectSummary.getBucketName());  
  System.out.println(s3ObjectSummary.getETag());  
  System.out.println(s3ObjectSummary.getKey());  
  System.out.println(s3ObjectSummary.getLastModified());  
  System.out.println(s3ObjectSummary.getOwner());  
  System.out.println(s3ObjectSummary.getSize());  
  System.out.println(s3ObjectSummary.getStorageClass());  
 });  
 } catch (AmazonServiceException e) {  
  e.printStackTrace();  
 } catch (SdkClientException e) {  
  e.printStackTrace();  
 } }}

使用IAM用户临时凭证创建请求 - AWS SDK for Java

import com.amazonaws.AmazonServiceException;  
import com.amazonaws.SdkClientException;  
import com.amazonaws.auth.AWSStaticCredentialsProvider;  
import com.amazonaws.auth.BasicSessionCredentials;  
import com.amazonaws.auth.profile.ProfileCredentialsProvider;  
import com.amazonaws.services.s3.AmazonS3;  
import com.amazonaws.services.s3.AmazonS3ClientBuilder;  
import com.amazonaws.services.s3.model.ObjectListing;  
import com.amazonaws.services.securitytoken.AWSSecurityTokenService;  
import com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClientBuilder;  
import com.amazonaws.services.securitytoken.model.AssumeRoleRequest;  
import com.amazonaws.services.securitytoken.model.Credentials;  
import com.amazonaws.services.securitytoken.model.GetSessionTokenRequest;  
import com.amazonaws.services.securitytoken.model.GetSessionTokenResult;  
  
public class MakingRequestsWithIAMTempCredentials {  
  
  public static void main(String[] args) {  
  String clientRegion = "ap-northeast-1";  
  String roleARN = "*** ARN for role to be assumed ***";  
  String roleSessionName = "*** Role session name ***";  
  String bucketName = "aws-lambda-test-etl";  
  
  
  try {  
  AWSSecurityTokenService stsClient = AWSSecurityTokenServiceClientBuilder.standard()  
 .withCredentials(new ProfileCredentialsProvider())  
 .withRegion(clientRegion)  
 .build();  
  
  AssumeRoleRequest roleRequest = new AssumeRoleRequest()  
 .withRoleArn(roleARN)  
 .withRoleSessionName(roleSessionName);  
  
  stsClient.assumeRole(roleRequest);  
  
  GetSessionTokenRequest getSessionTokenRequest = new GetSessionTokenRequest();  
  getSessionTokenRequest.setDurationSeconds(7200);  
  
  GetSessionTokenResult sessionTokenResult = stsClient.getSessionToken(getSessionTokenRequest);  
  
  Credentials sessionCredentials = sessionTokenResult.getCredentials();  
  
  BasicSessionCredentials basicSessionCredentials = new BasicSessionCredentials(sessionCredentials.getAccessKeyId(),  
  sessionCredentials.getSecretAccessKey(),  
  sessionCredentials.getSessionToken());  
  
  AmazonS3 s3Client = AmazonS3ClientBuilder.standard()  
 .withCredentials(new AWSStaticCredentialsProvider(basicSessionCredentials))  
 .withRegion(clientRegion)  
 .build();  
  
  ObjectListing objects = s3Client.listObjects(bucketName);  
  System.out.println("No. of Objects: " + objects.getObjectSummaries().size());   
} catch (AmazonServiceException e) {  
  e.printStackTrace();  
 } catch (SdkClientException e) {  
  e.printStackTrace();  
 }     }  
}

使用联合身份用户临时凭证

import com.amazonaws.AmazonServiceException;  
import com.amazonaws.auth.AWSStaticCredentialsProvider;  
import com.amazonaws.auth.BasicSessionCredentials;  
import com.amazonaws.auth.policy.Policy;  
import com.amazonaws.auth.policy.Resource;  
import com.amazonaws.auth.policy.Statement;  
import com.amazonaws.auth.policy.actions.S3Actions;  
import com.amazonaws.auth.profile.ProfileCredentialsProvider;  
import com.amazonaws.services.s3.AmazonS3;  
import com.amazonaws.services.s3.AmazonS3ClientBuilder;  
import com.amazonaws.services.s3.model.ObjectListing;  
import com.amazonaws.services.securitytoken.AWSSecurityTokenService;  
import com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClientBuilder;  
import com.amazonaws.services.securitytoken.model.Credentials;  
import com.amazonaws.services.securitytoken.model.GetFederationTokenRequest;  
import com.amazonaws.services.securitytoken.model.GetFederationTokenResult;  
  
public class MakingRequestsWithFederatedTempCredentials {  
  
  public static void main(String[] args) {  
  String clientRegion = "ap-northeast-1";  
  String bucketName = "aws-lambda-test-etl";  
  String federatedUser = "zean.ma";  
  String resourceARN = "arn:aws:s3:::" + bucketName;  
  try {  
  AWSSecurityTokenService stsClient = AWSSecurityTokenServiceClientBuilder  
  .standard()  
 .withCredentials(new ProfileCredentialsProvider())  
 .withRegion(clientRegion)  
 .build();  
  
  GetFederationTokenRequest getFederationTokenRequest = new  
  GetFederationTokenRequest();  
  getFederationTokenRequest.setDurationSeconds(7200);  
  getFederationTokenRequest.setName(federatedUser);  
  
  // Define the policy and add it to the request.  
  Policy policy = new Policy();  
  policy.withStatements(new Statement(com.amazonaws.auth.policy.Statement.Effect.Allow)  
 .withActions(S3Actions.ListObjects)  
 .withResources(new Resource(resourceARN)));  
  getFederationTokenRequest.setPolicy(policy.toJson());  
  
  // Get the temporary security credentials.  
  GetFederationTokenResult federationTokenResult =  
  stsClient.getFederationToken(getFederationTokenRequest);  
  Credentials sessionCredentials = federationTokenResult.getCredentials();  
  // Package the session credentials as a BasicSessionCredentials  
 // object for an Amazon S3 client object to use.  BasicSessionCredentials basicSessionCredentials = new BasicSessionCredentials(  
  
  sessionCredentials.getAccessKeyId(),  
  sessionCredentials.getSecretAccessKey(),  
  sessionCredentials.getSessionToken());  
  
  AmazonS3 s3Client = AmazonS3ClientBuilder.standard()  
 .withCredentials(new  
  AWSStaticCredentialsProvider(basicSessionCredentials))  
 .withRegion(clientRegion)  
 .build();  
  
  // To verify that the client works, send a listObjects request using  
 // the temporary security credentials.  ObjectListing objects = s3Client.listObjects(bucketName);  
  System.out.println("No. of Objects = " + objects.getObjectSummaries().size());  
  
 } catch (AmazonServiceException e) {  
  
 } }}

创建存储桶

Note

属于您在特定AWS区域中创建的存储桶的对象绝不会离开该区域，除非显式将它们传输到其他区域。例如，在欧洲（爱尔兰）区域存储的对象将一直留在该区域。

例如，指定AWS区域将它映射到特定的区域的终端节点：

s3-<region>.amazonaws.com

创建存储桶的SDK for Java

import com.amazonaws.AmazonServiceException;  
import com.amazonaws.SdkClientException;  
import com.amazonaws.auth.profile.ProfileCredentialsProvider;  
import com.amazonaws.services.s3.AmazonS3;  
import com.amazonaws.services.s3.AmazonS3ClientBuilder;  
import com.amazonaws.services.s3.model.CreateBucketRequest;  
import com.amazonaws.services.s3.model.GetBucketLocationRequest;  
  
public class CreateBucket {  
  
  public static void main(String[] args) {  
  String clientRegion = "ap-northeast-1";  
  String bucketName = "clearlove";  
  
  try {  
  AmazonS3 s3 = AmazonS3ClientBuilder.standard()  
 .withCredentials(new ProfileCredentialsProvider())  
 .withRegion(clientRegion)  
 .build();  
  
  if (! s3.doesBucketExistV2(bucketName)) {  
  
  s3.createBucket(new CreateBucketRequest(bucketName));  
  
  String bucketLocation = s3.getBucketLocation(new GetBucketLocationRequest(bucketName));  
  
  System.out.println("Bucket location:    " + bucketLocation);  
 } } catch (AmazonServiceException e) {  
  e.printStackTrace();  
 } catch (SdkClientException e) {  
  e.printStackTrace();  
 } }}

删除存储桶

  • 使用AWS CLI方式进行对桶的删除

    $ aws s3 rb s3://bucket-name --force

  • 使用生命周期方式删除桶

要删除存储桶中的所有对象，可以将生命周期规则设置为使对象在创建一天后过期 Important: 如果想清空而非删除存储桶，请确保删除为清空存储桶而添加的生命周期，以使在存储桶中创建的任何对象将保留在存储桶中。在清空存储桶时，清先删除生命周期配置。

  • 使用AWS SDK Java 实现

Note: 对于未启用版本控制的存储桶，您可以直接删除所有对象，然后删除存储桶。对于启用了版本控制的存储桶，您必须先删除所有对象版本，然后再删除存储桶。

import com.amazonaws.AmazonServiceException;  
import com.amazonaws.SdkClientException;  
import com.amazonaws.auth.profile.ProfileCredentialsProvider;  
import com.amazonaws.services.s3.AmazonS3;  
import com.amazonaws.services.s3.AmazonS3ClientBuilder;  
import com.amazonaws.services.s3.model.*;  
  
import java.util.Iterator;  
  
public class DeleteBucket {  
  
  public static void main(String[] args) {  
  String clientRegion = "ap-northeast-1";  
  String bucketName = "clearlove";  
  
  try {  
  AmazonS3 s3 = AmazonS3ClientBuilder.standard()  
 .withCredentials(new ProfileCredentialsProvider())  
 .withRegion(clientRegion)  
 .build();  
  
  ObjectListing objectListing = s3.listObjects(bucketName);  
  
  while (true) {  
  Iterator<S3ObjectSummary> objectSummaryIterator = objectListing.getObjectSummaries().iterator();  
  while (objectSummaryIterator.hasNext()) {  
  s3.deleteObject(bucketName, objectSummaryIterator.next().getKey());  
 }  if (objectListing.isTruncated()) {  
  objectListing = s3.listNextBatchOfObjects(objectListing);  
 } else {  
  break;  
 } }  
  VersionListing versionListing = s3.listVersions(new ListVersionsRequest()  
 .withBucketName(bucketName));  
  
  while (true) {  
  Iterator<S3VersionSummary> versionIter = versionListing.getVersionSummaries().iterator();  
  while (versionIter.hasNext()) {  
  S3VersionSummary vs = versionIter.next();  
  s3.deleteVersion(bucketName, vs.getKey(), vs.getVersionId());  
 }  
  if (versionListing.isTruncated()) {  
  versionListing = s3.listNextBatchOfVersions(versionListing);  
 } else {  
  break;  
 } }  s3.deleteBucket(bucketName);  
 } catch (AmazonServiceException e) {  
  e.printStackTrace();  
 } catch (SdkClientException e) {  
  e.printStackTrace();  
 }  
 }}

清空存储桶

  • 使用AWS CLI方式清空存储桶

    $ aws s3 rm s3://bucket-name/doc --recursive //删除以doc前缀的对象
$ aws s3 rm s3://bucket-name  --recursive //删除所有对象

获取桶里的对象

import com.amazonaws.AmazonServiceException;  
import com.amazonaws.SdkClientException;  
import com.amazonaws.auth.profile.ProfileCredentialsProvider;  
import com.amazonaws.services.s3.AmazonS3;  
import com.amazonaws.services.s3.AmazonS3ClientBuilder;  
import com.amazonaws.services.s3.model.GetObjectRequest;  
import com.amazonaws.services.s3.model.ResponseHeaderOverrides;  
import com.amazonaws.services.s3.model.S3Object;  
  
import java.io.BufferedReader;  
import java.io.IOException;  
import java.io.InputStream;  
import java.io.InputStreamReader;  
  
public class GetObject {  
  
  public static void main(String[] args) throws IOException {  
  String clientRegion = "ap-northeast-1";  
  String bucketName = "clearlove";  
  String key = "zeanfile";  
  S3Object fullObject = null, objectPortion = null, headerOverrideObject = null;  
  try {  
  
  AmazonS3 s3 = AmazonS3ClientBuilder.standard()  
 .withRegion(clientRegion)  
 .withCredentials(new ProfileCredentialsProvider())  
 .build();  
  
  System.out.println("Downloading an object.");  
  fullObject = s3.getObject(new GetObjectRequest(bucketName, key));  
  System.out.println("Content-Type: " + fullObject.getObjectMetadata().getContentType());  
  System.out.println("Content: ");  
  
  displayTextInputStream(fullObject.getObjectContent());  
  GetObjectRequest request = new GetObjectRequest(bucketName,key).withRange(0,9);  
  
  objectPortion = s3.getObject(request);  
  
  System.out.println("Printing bytes retrieved.");  
  
  displayTextInputStream(objectPortion.getObjectContent());  
  
  ResponseHeaderOverrides headerOverrides = new ResponseHeaderOverrides()  
 .withCacheControl("No-cache")  
 .withContentDisposition("attachment;  filename=example.txt");  
  GetObjectRequest getObjectRequest = new GetObjectRequest(bucketName,key)  
 .withResponseHeaders(headerOverrides);  
  headerOverrideObject = s3.getObject(getObjectRequest);  
  
  displayTextInputStream(headerOverrideObject.getObjectContent());  
  
 } catch (IOException e) {  
  e.printStackTrace();  
 } catch (AmazonServiceException e) {  
  e.printStackTrace();  
 } catch (SdkClientException e) {  
  e.printStackTrace();  
 } finally {  
  if (fullObject != null) {  
  fullObject.close();  
 }  if (objectPortion != null) {  
  objectPortion.close();  
 }  if (headerOverrideObject != null) {  
  headerOverrideObject.close();  
 } }  
 }  
  
  private static void displayTextInputStream(InputStream input) throws IOException {  
  BufferedReader reader = new BufferedReader(new InputStreamReader(input));  
  String line = null;  
  while ((line = reader.readLine()) != null) {  
  System.out.println(line);  
 }  System.out.println();  
 } 
 }

上传对象

import com.amazonaws.AmazonServiceException;  
import com.amazonaws.SdkClientException;  
import com.amazonaws.auth.profile.ProfileCredentialsProvider;  
import com.amazonaws.services.s3.AmazonS3;  
import com.amazonaws.services.s3.AmazonS3ClientBuilder;  
import com.amazonaws.services.s3.model.ObjectMetadata;  
import com.amazonaws.services.s3.model.PutObjectRequest;  
  
import java.io.File;  
  
public class UploadObject {  
  
  public static void main(String[] args) {  
  String clientRegion = "ap-northeast-1";  
  String bucketName = "clearlove";  
  String stringObjKeyName = "zean";  
  String fileObjectKeyName = "zeanfile";  
  String fileName = "pom.xml";  
  
  try {  
  
  AmazonS3 s3 = AmazonS3ClientBuilder.standard()  
 .withRegion(clientRegion)  
 .withCredentials(new ProfileCredentialsProvider())  
 .build();  
  
  s3.putObject(bucketName,stringObjKeyName,"Uploaded String Object.");  
  
  PutObjectRequest request = new PutObjectRequest(bucketName,fileObjectKeyName,new File(fileName));  
  
  ObjectMetadata metadata = new ObjectMetadata();  
  metadata.setContentType("plain/text");  
  metadata.addUserMetadata("x-amz-meta-title","someTitle");  
  request.setMetadata(metadata);  
  s3.putObject(request);  
 } catch (AmazonServiceException e) {  
  e.printStackTrace();  
 } catch (SdkClientException e) {  
  e.printStackTrace();  
 }  
 }}

VPC 学习指南

Amazon VPC（Amazon Virtual Private Cloud）允许在已定义的虚拟网络内启动AWS资源

VPC对等连接是两个VPC之间的连接，这两个VPC中的实例可以彼此通信，就像它们在同一网络中一样。可以在自己的VPC之间创建VPC对等连接，也可以与其他AWS账户中的VPC之间创建连接。VPC可位于不同区域内（区域间VPC对等连接） VPC

VPC对等基本知识

要建立对等连接，请执行一下步骤：

  1. 请求者 VPC 的所有者向接受者 VPC 的所有者发送创建 VPC 对等连接的请求。接受者 VPC 可以归您或 其他 AWS 账户所有，不能包含与请求者 VPC 的 CIDR 块重叠的 CIDR 块。
  2. 接受者 VPC 的所有者接受 VPC 对等连接请求以激活 VPC 对等连接。
  3. 要使用私有 IP 地址实现 VPC 之间的流量流动，VPC 对等连接中每个 VPC 的所有者必须向一个或多个 VPC 路由表手动添加指向其他 VPC (对等 VPC) 的 IP 地址范围的路由。
  4. 如果两个 VPC 位于相同区域内，则您可以将您的 VPC 连接修改为启用 DNS 主机名解析。默认情况下， 如果 VPC 对等连接任一侧的实例使用公有 DNS 主机名相互进行寻址，则主机名会解析为实例的公有 IP 地址。

VPC对等连接的生命周期

enter image description here

  • Initiating-request (发起请求)：已发起 VPC 对等连接请求。在这一阶段中，对等连接可能失败或可能转到 pending-acceptance。

  • Failed (已失败)：VPC 对等连接请求失败。在处于此状态时，无法接受、拒绝或删除该连接。请求者仍可 在 2 个小时内看到失败的 VPC 对等连接。

  • Pending-acceptance：等待接受者 VPC 的所有者接受 VPC 对等连接请求。在这一阶段中，请求者 VPC 的所有者可以删除此请求，接受者 VPC 的所有者可以接受或拒绝此请求。如果双方均未对此请求执行任何 操作，该请求将在 7 天后过期。

  • Expired (已过期)：VPC 对等连接请求已过期，任一 VPC 所有者都无法再对该请求执行任何操作。两个 VPC 所有者仍可以在 2 天内看到已过期的 VPC 对等连接。

  • Rejected：接受者 VPC 的所有者拒绝了 pending-acceptance VPC 对等连接请求。在这一阶段中，无 法接受请求。请求者 VPC 的所有者仍可以在 2 天内看到已拒绝的 VPC 对等连接，接受者 VPC 的所有者 仍可在 2 个小时内看到此对等连接。如果请求是在同一 AWS 账户内创建的，则已拒绝的请求会继续显示 2 个小时。

  • Provisioning (正在预置)：VPC 对等连接请求已接受，即将处于 active 状态

  • Active：VPC 对等连接处于活动状态，而且流量可以在 VPC 之间流动 (假设您的安全组和路由表允许流量 流动)。在这一阶段中，任一 VPC 所有者都可以删除 VPC 对等连接，但是无法拒绝它。 Note 如果 VPC 所在的区域中的事件阻止了流量流动，则 VPC 对等连接的状态将保持 Active。

  • Deleting：适用于处于删除过程中的区域间 VPC 对等连接。任一 VPC 的所有者已提交删除 active VPC 对等连接的请求，或者请求者 VPC 的所有者已提交删除 pending-acceptance VPC 对等连接请求的请求。

  • Deleted (已删除)：任一 VPC 拥有者已删除了 active 的 VPC 对等连接，或请求者 VPC 的拥有者已删除 了 pending-acceptance 的 VPC 对等连接请求。在这一阶段中，无法接受或拒绝 VPC 对等连接。VPC 对等连接仍会向其删除方继续显示 2 个小时，向另一方显示 2 天。如果 VPC 对等连接是在同一 AWS 账 户内创建的，则已删除的请求仍将继续显示 2 个小时。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment