Posted on |

058.mp4 – SQS Overview

  • up to 10 attributes can be add to a message

  • Size 1 - 256 K

  • Standard usage : provide cloudwatch metric (queue depth) to help auto Scaling

  • Queue type

    • Standard queue
    • FIFO (Max 300TPS, exact once) ; not available in all regions

  • Message Lifecycle (Visibility Timeout)

  • Dead Letter Queue : must be in same region under same account with the source queue

  • Delay Queue

    • Define “DelaySeconds”
    • max inflight msg = 120000

  • Message Timers : individual message being available with a delayed manner. set “DelaySeconds” for individual message

  • Two type of polling

    • short polling : 有消息吗？服务器答： 有/没有
    • Long polling: 有消息吗， 20秒内有消息回我，我等着。 服务器：好的。
      • long polling减少无效的空返回 and false empty response (subset of servers)
      • set “WaitTimeSeconds” 1~20 second

How the “DelaySeconds” change affect existing Message (different behavior for Standard and FIFO)
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-delay-queues.html

Delete the Queue : what happens to existing Message
https://docs.aws.amazon.com/cli/latest/reference/sqs/delete-queue.html

Message Retention & Visiblity timeout
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-limits.html

Short Pooling definition : (Subset of servers)
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html

Queue identifier (format)

https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue

https://sqs.regionname.amazonaws.com/accountnumber/queuename.uniqueforuser[.fifo]
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-general-identifiers.html

Message ID vs Receipt Handle
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-general-identifiers.html

Read more »

Posted on |

OpsWorks Overview

  • Use Chef Recipes
  • Better and fine-controlled way of define infrastructure (Compare to Elastic Beanstalk)

CM Model

  • CM Model (configuration management)

    • Stack: a set of intances and applications
    • Layers: reusable subcomponent of stack
    • Instances: can participate multiple layer
    • Apps: codes running on server

  • Scaling

    • manuall Scaling
    • Automatic scaling : time based; load based
    • can be combined together

  • Chef Recipes — infrastructure as code

DeepDive

  • Difference with Chef Server, (there’s no chef server)

    • Can be agentless (push model)

  • Push Json format event to define to target status for each of the lifecycle of the server.

    • Setup event
    • Config event
    • Deploy event
    • Undeploy event
    • Shutdown event

OpsWorks hands on

  • Create first stack
  • Create sample stack
  • Check the git repo for application and git repo for infrastructure
  • quicker than ElasticBeanStalk or container — high end options

improvements

Separated env for AWS chef recipe and customer recipe to avoid conflicts

Read more »

Posted on |

CloudFormation Overview

JSON format definition about what service needs to be deployed
Infra as code

049.mp4 050.mp4 051.mp4 - hands on cloudformation

  • Create cloudformation template using JSON editor

    • manually add “AWSTemplateFormatVersion”,“Description”,“Parameters”,“Resources”,“Outputs”
    • pre-defined copied: define a DynamoDB with
      • Parameters: ReadCapacityUnit and WriteCapacityUnit
      • Resources: DynamoDB Table with attributes definitions
      • Outputs: print out the table name
    • from AWS console , from “CloudFormation” service main page, select “create a new stack”
      • template can from sample / upload one to S3 / from a URL (hosted in S3)
      • Roll back on failure is “On” by default
    • swith to DynamoDB console, and check the table is created as expected.

  • Create Cloudformation using CloudFormer

    • Create a stack and select “template from sample” and choose the “Cloudformer” template
      • give user and password to login CloudFormer (took 8 min to create from video)
      • output contains the url of the CloudFormer webpage
    • Login CloudFormer
      • create template
      • go through the wizard to select from existing resources as template to form the new template
      • note: it won’t define paramteters (directly copy from the existing resources)
    • Launch the stack using the template created by CloudFormer

CloudFormation Desinger

  • Visualize the Design
  • Can be used to update existing stack

Extend with custom resources

Capability to create resource that implement aws defined create/update/rollback/delete and metadata

Standard Senario: use lambda

Read more »

Posted on |

045.mp4 – Elastic File System : is a NAS

  • EFS a Network Attached Storage
  • EFS is a NAS (Network Attached Storage; a File system) ; S3 & Glacier is a webstore ; EBS is a block
  • EFS can be shared by multi EC2 instances
  • EFS can grow and can shink ; Throughput scales automatically
  • Pay as you go (no minimum)
  • As a NAS, it support thousands of connections
  • Multi AZ replication

After being mounted as Mount Target, it will be shown as a network storage resource.

Security Control

046.mp4 – Handson on EFS

  • mount EFS into certain subnet in VPC (has to be a subnet)
    • Create EFS
    • Launch an EC2 instance
    • Security setting

From Q&A

  • To load data from on-premise to EFS, you can use SCP (Secure Copy)
  • For old classic EC2 not inside VPC, you can use ClassicLink to connect to EFS
  • On-premise server can connect to EFS via AWS Direct Connect
Read more »

Posted on |

DynamoDB Deepdive

  • RDB optimized for storage; nonsql optimized for compute
  • DynamoDB supports both Key-value and document data models

Terminology

Keys:

  • Partition Key; Hash Key;

  • Sort Key;

  • Range Key ;

    • Local secondary index (LSI)
    • Global secondary index (GSI) (max 5 GSI per table)
    • If data size >10G use GSI
    • Primary Key = Partition key + Sort Key

  • Attribute;

  • composite attributes ; composite key : a way of construct partition key

  • Partition Key used to decide which partition it belongs to (uidling unordered hash index)

  • A Partition has 10G limit, if total storage with same partition key exceeded the limit, then sort key is used.

  • Table --> Items --> Attributes --> Partition Keys (Mandatory attribute); Sort Key (Optional)

  • DynamoDB each partition will have totally 3 copies (including itself); when write, you will get success response when 2 writes succeed.

  • Hash Range Table : a table where Hashkey+RangeKey to identify an item.

Scaling

  • Scaling on throughput : WCU and RCU (CU is Capacity Unit)
    • Partition needed = Roundup((total RCU/3k)+(total WCU/1k))
  • Scaling on size (maxsizeperitem=400kb, maxsizeperpartition=10G)
  • Final partition = ceiling( ScalingByThrougput, ScalingBySize)
  • heat map – showing by time and partition dimention about which data being requested. If all data access is focused from a specific partition, then we got Hot Keys which we should avoid by re-design the paritition key.
  • DynamoDB Burst capacity is built in

Data Modeling

Store the data how you will access it.

  • New feature since 2015: Support documents (JSON)
Read more »

Posted on |

VPC deepdive 2016

https://youtu.be/Qep11X1r1QA
very deep session about BGP, VPN, Direct connect. Loads of technical details

Difference between IPSec VPN and DirectConnect

Hardware VPN

  • 2 tunnels/VPC; each tunnel will connect to one AZ
  • 0.05/hours/VPN (that includes 2 tunnels); ( EC2 medium is 0.1/hour)
  • support static VPN and dynamic VPN (BGP)

Static VPN vs Dynamic VPN

  • Static VPN
    • IP address is Static
    • each tunnel need 2 pairs of Security Association (inbound and outbound); that means 1 VPC connection needs 4 SA pairs
  • Dynamic VPN
    • BGP IP address is dynamically generated
    • Use ASN as registry to talk with each other (for AWS, 1 ASN/Region; for customer side, also need to configure ASN)
    • In PROD, you can setup 2 tunnels per VPC from customer site to connect to each VPC owned by you.

Common maintain FAQ for VPN

  • How to change pre-shared key?

Create a new VPN connection and delete current. — (IP config might change)

Read more »

Posted on |

RDS

Max Storage limites

  • MS SQL Server 4TB
  • My SQL, Oracle, PostgreSQL, MariaDB 6TB
  • Aurora 64TB

Supported Relational DB types

  • Support 6 Relational DB types, MS sql;mysql;Oracle; AWS Aurora; PostgreSQL; MariaDB
  • Backup - to S3 (can be encrypted for db or snapshot at rest)
  • Failover - Multi-AZ ; when master fails , standby is promoted , then CName is updated to poiting to standby, then new instance is created to replace the master
  • Read Replica (don’t support ms sql and oracle); one DB can have multiple read replicas
    • If you have multiple read replica, then routing from single url using Route53 or customed HAProxy , not support AWS ELB

Security

  • Network firewall control
    Like EC2 security group, RDS has its own security group settings
  • Access Control
    IAM : IAM can’t control who can log in database (database user and groups); IAM only controls who can have what level of access to the RDS service.
  • Compliance and Transport SSL
  • At Rest Encryption: using KMS(key management service) & Envolope Encryption
    • Limit risk of compromise key
    • Centralized access and audit of key activity
    • just click “enable encryption”
  • At Rest Encryption Limitations
    • only available when creation a new database; and once enabled , can’t be removed.
    • Unencrpyted snapshot can be changed to encrypted snapshot
    • Encrypted db across region (aws is working on that)

config to enable encryption

Metrics and monitoring

  • 1 min interval by default
  • Enhanced Monitoring – more detailed , minimum 1 second
  • AWS Performance Insights for RDS : designed for RDS, with top SQL and help with identify bottlenecks
Read more »

Posted on |

Terminology

Canaries: when deploy limite the new version into a contained env until fully tested then roll out.

CloudFront = Content Delivery Network by AWS

CloudFrontPop: Point Of Presence, Edge Location

  • Located in DataCenter (Major metropolitan) with direct connection with multi ISPs
  • Terminating Viewer connections
  • Request routing to CloudFront is mainly done by DNS Layer

CloudFront deployed on Edge locations (number of Edge locations>Available zones> Regions)

  • Source can be S3 , HTTP Server on AWS or outside AWS
  • CloudFront only cache GET and HEAD requests, for POST,PUT,DELETE cloudfront will only proxy
  • Different combinations to cache for static / dynamic websites
    • For static, we can select only cache documents , exclude HTML/css/code; or set TTL and cache all
    • For dynamic, we can select only cache static content , exclude php/code ; or set TTL and cache all

How it works

Cache Key : generated using request URL (remove query string, protocol and add encoding)

Specify ExpireTime: sent from original header
Specify Max Age: for example max-age=300 means 5 min

Read more »

Posted on |

030.mp4 – CLI

2 ways of interact with AWS via CLI

  • install locally, access AWS services via https
  • install on EC2, access AWS services via SSH
1
2
3
4
5
6
7
8
9
aws --version
aws s3 mb s3://newbucket
aws s3 ls
aws s3 rb s3://newbucket
aws ec2 help
aws ec2 describe-instances


aws iam create-user --user-name newuser

TODO,
setup on mac and handson aws

Read more »
0%