AWS - EC2

Posted on |

Login EC2

  • for ubuntu it’s ubuntu@hostname instead of ec2-user@hostname

016.mp4 Elastic Compute Cloud

Region: same region same price, same latency , same regulation
Available Zone: same datacentor
Edge Location: Cloud Front

purchase mode,

  1. on-demand instances
  2. reserved instances
  3. spot instances (错峰用机)

Q: How do I select the right instance type?

Amazon EC2 instances are grouped into 5 families: General Purpose, Compute Optimized, Memory Optimized, Storage Optimized and Accelerated Computing instances.

  • General Purpose Instances have memory to CPU ratios suitable for most general purpose applications and come with fixed performance (M5, M4) or burstable performance (T2);
  • Compute Optimized instances (C5, C4) have proportionally more CPU resources than memory (RAM) and are well suited for scale out compute-intensive applications and High Performance Computing (HPC) workloads;
  • Memory Optimized Instances (X1e, X1, R4) offer larger memory sizes for memory-intensive applications, including database and memory caching applications;
  • Accelerating Computing instances (P3, P2, G3, F1) take advantage of the parallel processing capabilities of NVIDIA Tesla GPUs for high performance computing and machine/deep learning; GPU Graphics instances (G3) offer high-performance 3D graphics capabilities for applications using OpenGL and DirectX; F1 instances deliver Xilinx FPGA-based reconfigurable computing;
  • Storage Optimized Instances (H1, I3, D2) that provide very high, low latency, I/O capacity using SSD-based local instance storage for I/O-intensive applications, with D2 or H1, the dense-storage and HDD-storage instances, provide local high storage density and sequential I/O performance for data warehousing, Hadoop and other data-intensive applications.

019 hands-on connect to EC2 widnows instance

Difference with creating Linux instance,

  • RDP (3389) need to be specified at security group (inbound)
  • Windows Administrator password is dynamically randomly generated and as a PEM just like the linux instance (download)
  • need to extract online and get administrator user and password

EC2Config

About the EC2Config service (New name EC2Launch)
http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/UsingConfig_WinAMI.html

  • built in function executed to enable the box to work with aws cloud. (for example the get metadata link)
  • can be used to initialize the windows box
  • can be used to trigger some management command for the box (like mount the EBS, etc)
  • Note. EC2Launch replaces EC2Config on Windows Server 2016 AMIs.

020 hands-on connect to EC2 via MacOS client

For linux

1
2
chmod 400 xxx.pem
ssh -i "/path/to/pem/xxx.pem" remotehostname

For Windows
download CORD RDP client

021 create a custom AMI

iptable rerounting

1
sudo iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080

After everything is setup
select the instance -> action -> create image

AMI --> Images
Select the image being created we can
put it as Public
copy it to other region and create a instance based on this customized instance
AMI image bill？？？（public or not）

022 EBS (Elastic Block Storage) for EC2

EBS(Elastic Block Storage)

  • attached to an EC2 instance
  • persist independently from the life of the EC2 instance
  • pay only for what you use
1
2
3
4
5
6
7
8
9
lsblk
sudo file -s /dev/xvdb
sudo mkfs -t ext4 /dev/xvdb
sudo file -s /dev/xvdb
sudo mkdir /data
sudo mount /dev/xvdb /data
sudo nano /etc/fstab
sudo mount -a
sudo touch /data/test.txt
  • create a snapshot for the EBS
  • Create a new EBS volume based on the snapshot just created
  • Create another EC2 instance, selecting the snapshot just created when adding extra EBS to the instance

Instance Storage
Instance Storage:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html
Not available for low end EC2
must use block device mapping to attach them as the instance is launched.

EBS snapshots

point-in-time snapshot
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSSnapshots.html

  • Asyncronize creation, but status is pending until fully finished
  • When trigger the snapshot command, EBS needs to be unmounted (see below example for /dev/sdh); or stop the instance if EBS used as root device; or user guarantee no file write to the EBS. But once the command is triggered, the volume can be mounted back when snapshot status is “Pending”.
  • We can have multiple pending snapshots pointing to different time. But total number of pending snapshot will have a limit.
  • EBS snapshots are only available through the Amazon EC2 APIs, not S3 APIs.也就是说虽然snapshot是存在S3的，但并不代表可以使用S3 API访问snapshot的内容。仍旧需要通过EC2来访问这些snapshot
  • 每个snapshot都有unique id，可以通过snapshot id来恢复volume
1
umount -d /dev/sdh
  • command used to create EBS snapshot. (Different in aws cli and aws tools for windows Powershell)
1
aws ec2 create-snapshot --volume-id vol-1234567890abcdef0 --description "This is my root volume snapshot."
1
PS C:\> New-EC2Snapshot -VolumeId vol-12345678 -Description "This is a test"

如何使用tag管理ec2的权限

  • ec2创建一个叫username的tag，值设为有权限访问该ec2的用户名
  • apply policy给用户
  • 当判断用户权限的时候，使用用户id和ec2的tag的“username”的value比较，如何等于则赋权

例子如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
      "Version" : "2012-10-17",
           "Statement" :
      [
           {
                "Effect" : "Allow",
                "Action" : "ec2:*",
                "Resource" : "*",
                "Condition" : {
                     "StringEquals" : {
                          "ec2:ResourceTag/UserName" : "${aws:username}"
                     }
                }
           }
      ]
}

Placement groups

一种配置，创建了之后让aws知道启动EC2的时候如何分配资源

  • cluster placement group： 一旦创建，加入同group的ec2 VM会被分配到同一个AZ，并优化之间的网络环境。
  • Spread placement group： 一旦创建，加入同group的ec2 VM会被分配到不同的硬件上，减少硬件失败造成的单点故障。
  • EC2运行的时候无法换placement group，停下来后可以换placegroup的配置；placement group在有EC2组员的时候不能删除。
  • 不支持tag: can’t create tag for the placement group

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html

Related Reading

SSD-backed EBS and HDD-backed EBS

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html

  • SSD-backed volumes optimized for transactional workloads involving frequent read/write operations with small I/O size, where the dominant performance attribute is IOPS

    • GP2 (General Purpose): 默认通用类型的，IOPS可以在100-1w之间伸缩 ； 最大吞吐量160Mib/s
    • Provisioned IOPS SSD (io1)：最大能到32000； 最大吞吐量500Mib/s
    • 适合： transactional, IOPS-intensive database workloads, boot volumes, and workloads that require high IOPS.

  • HDD-backed volumes optimized for large streaming workloads where throughput (measured in MiB/s) is a better performance measure than IOPS

    • Throughput Optimized HDD (st1)：IOPS最大500；最大吞吐量最大500
      • 适合： MapReduce, Kafka, log processing, data warehouse, and ETL workloads.
    • Cold HDD (sc1)： IOPS最大250；最大吞吐量250
      • 适合： ideal for less frequently accessed workloads with large, cold datasets.

  • IOPS (Input/Output Operations Per Second)，即每秒进行读写（I/O）操作的次数，多用于数据库等场合，衡量随机访问的性能。

  • IOPS的limitation是按照volumn来的，就是一块格式化好的存储mount上来，根据类型不同具备不同的IOPS上限。可以通过mount多块硬盘提高总体的IOPS，对于一台虚拟机来说，不管是什么volumn，最终的IOPS上限是一样的。只是HDD的硬盘要达到上限可能需要mount更多的volumn来实现reach iops的上限。IOPS上限8w/instance

  • Throughput类似，也是一台虚拟机有一个上限。不同的存储可以通过多mount来提高throughput但是无法突破单机上限。吞吐上限1750/instance

  • 如何检查：Linux check current IOPS， r/s每秒读取次数；w/s每秒写入次数

1
iostat -x
  • GP2类型的SSD具备伸缩性。这种伸缩性有复杂的机制。
    • GP2 SSD size从1Gib-16Tib； size越大，baseline performance越好： 1Gib的IOPS=100，16Tib的IOPS=10000，正常情况3*volumn
    • 每块GP2 SSD的初始I/O credit都是5.4m
    • BurstDuration=CurrentCredit/(BurstIOPS-3*VolumnInGB)
    • Initial Credit == Max Credit == 5.4 million
    • BurstIOPS是指这个当前的IOPS比原本assign的baseline performance部分高，但是最高不能超过3k，所以1000G以上的volumn本身baseline就高于3k，因此没有burst一说。
    • 当磁盘IOPS没有达到base performance标准的时候，就可以累计credit
      • CurrentCredit += baseIOPS*Duration;

Pre-warming = Initialization

1
# dd if=/dev/zero of=/dev/xvdX bs=1M

Encryption standards

industry-standard AES-256 algorithm.

Get Instance’s meta data

http://169.254.169.254/latest/meta-data/

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html
https://www.ibm.com/developerworks/cn/cloud/library/1620-openstack-metadata-service/

Launch more like this feature:
哪些配置会从当前机器copy哪些不会
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/launch-more-like-this.html

  • 会： RAM Disk ID； EBS optimization设置； public ip address （setting）
  • 不会： 网卡数目；storage配置。

不同region的ec2之间通信收费问题

  • EC2的data in 和out 是分别收费的。所以ec2 A 到 ec2 B收费，如果数据方向是A-》B，那就收A region的 data out， B region的data in

EC2 local store 的persist问题

ec2实例自带的storage, ec2只有running时候才bill，其它状态都不bill（EBS有单独的收费）

The data in an instance store persists only during the lifetime of its associated instance. If an instance reboots (intentionally or unintentionally), data in the instance store persists. However, data in the instance store is lost under the following circumstances:

  • The underlying disk drive fails

  • The instance stops

  • The instance terminates

Therefore, do not rely on instance store for valuable, long-term data. Instead, use more durable data storage, such as Amazon S3, Amazon EBS, or Amazon EFS.

  • EC2 Compute Unit
    Q: What is an “EC2 Compute Unit” and why did you introduce it?

Transitioning to a utility computing model fundamentally changes how developers have been trained to think about CPU resources. Instead of purchasing or leasing a particular processor to use for several months or years, you are renting capacity by the hour. Because Amazon EC2 is built on commodity hardware, over time there may be several different types of physical hardware underlying EC2 instances. Our goal is to provide a consistent amount of CPU capacity no matter what the actual underlying hardware.

  • Elastic IP address 不用也收费

    • Elastic IP address 是静态ip；pubic ip address是动态的，每次instance重启都会变

  • reverse DNS
    Reverse DNS: From IP to Domain
    A special PTR-record type is used to store reverse DNS entries. The name of the PTR-record is the IP address with the segments reversed + “.in-addr.arpa”.
    For example the reverse DNS entry for IP 1.2.3.4 would be stored as a PTR-record for “4.3.2.1.in-addr.arpa”.
    可以单独申请。

  • Nitro Hypervisor

  • 未来的新虚拟技术，新的aws虚拟机都用这个技术搭建。一个虚拟机可以加载的ebs+网卡（vpc eni）个数加起来=pci device个数,max=27

  • the NVMe device names used by Linux based operating systems will be different than the parameters for EBS volume attachment requests and block device mapping entries such as /dev/xvda and /dev/xvdf. NVMe devices are enumerated by the operating system as /dev/nvme0n1, /dev/nvme1n1

  • Enhanced network

    • 必须启用HVM虚拟机
    • 需要特定的driver，某些aws的机型内置了，如果没有需要手工安装配置。
    • 必须启动在vpc中

  • CloudWatch

    • 一旦启用，默认每分钟收取metrics；侦听信息默认保存两周（即使侦听对象terminate或者删除）；提供命令行mon-get-stats来保存侦听信息到s3或者“Amazon SimpleDB”

      • about SimpleDB Vs DynamoDB ： 简而言之，DynamoDB很可能是用来替代simpledb的

      https://stackoverflow.com/questions/8961333/amazon-simpledb-vs-amazon-dynamodb

    • 收费无关instance类型

    • gragh按照五分钟间隔显示曲线的时候，曲线点是按照五分钟内平均值拟合出来的，所以可能会跟1分钟间隔显示的曲线不同。

    References

    https://youtu.be/agQMFIWr2h4

    Deep Dive EC2 Performance

    EC2 History

Reward Makes Perfect
0%