Basic Steps

Cloudformation

• VPC with only private subnet; route table declared
• VGW created and attached to VPC;
• Propagation allowed via vgw to route table
• CGW information declared;

Create Site2SiteVPN

• Pay attention to IPSec Tunnel Interconnection IP CIDR
  https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-vpnconnection-vpntunneloptionsspecification.html

• Download configuration and run from client side

  • Pay attention to propagation CIDR

Client Side

1. Confirm the Client Gateway support BGP
2. Allocate the IpSec tunnel interconnection ip cidr
3. Allocate AWS VPC IP range
4. Confirm Data Centre Propagating IP Rages (default will be 0.0.0.0)

Preparation

Flash the machine
https://sites.google.com/site/ev3devpython/setting-up-vs-code

Connecting with mac
https://www.ev3dev.org/docs/tutorials/connecting-to-ev3dev-with-ssh/

issues
https://github.com/ev3dev/ev3dev/issues/1220

Wireless

Update – Single Azure to SSO to multiple AWS

• Identifier must be unique, it can be string

Config Azure AD SSO to AWS Console via SMAL

Azure Official Doc

https://docs.microsoft.com/en-us/azure/active-directory/saas-apps/amazon-web-service-tutorial

Aditional Notes

The config not align with above doc but needed when doing the config,

Example of claim key/values:

• name: emailaddress
• Namespace: http://schemas.xmlsoap.org/ws/2005/05/identity/claims
• Source: Attribute
• Source attribute: user.mail

Full config as below

Building Your Application with an Amazon Aurora Database (DEM113)

https://youtu.be/-ychuATbqPY

Key New Feature

• Serverless: Auto provision the computing power you need; scale up and down automatically.
• Aurora parallel query
  • An option when provision your DB, suitable for DB used for both transaction and analysis
  • https://aws.amazon.com/blogs/aws/new-parallel-query-for-amazon-aurora/
  • No extra cost but will be more expensive on IO
• Enable Backtrack (select the backup window)
  • Be able to backtrack , extra cost 10USD/month
• Performance Insight
  * by SQL by user(session)

Running a High-Performance Kubernetes Cluster with Amazon EKS (CON318-R1)

https://youtu.be/YQWt6wdAZMU

Optimize pod placement

1. limit the resource
2. Density vs. Size of pods
3. Anti-affinity : keep the CPU heavy pods onto different hosts

Use diagram to balance the design

1. Anti-affinity
2. Secretes
3. Number of Nodes
4. Active Namespaces
5. Pod Churn
6. Pod Density
7. Networking

Kinesis Deepdive

• No 1 popular scenario : moving small and fast moving data into persistent layer
• No 2 popular scenario : Steaming data , NRT notification systems

Kinesis:

• managed services
• streaming data ingestion
• continously processing

Small , fast moving data, being captured quickly , then being consumed concurrently by multi different consumers for different analytics Purpose.

• You can split / merge shards via console

best practises

partition key strategy

• Avoid hot shard
  • use random partition key
  • use high cardinality key
  • use business key : per billing customer or per device id or per stock symbol

provision shards

Redshift Archi overview

• Bottom Layer: Ingestion Backup & Restore layer
• Leader Node & Compute Node
  • Leader node :
• Share Nothing MPP (Massive Parellel Processing) Architecture
• Reduce IO
  • Columnar Storage
  • Compress data ( By Column)
  • Zone Maps : in memory map about min and max value for given column in current block, to prune the query and reduce IO
• Slices
  • depending on cpu cores, each node support different number of slices
  • unit of data partitioning / parallel processing
  • table rows are distributed into different slices
• Data Distribution :
  • ALL; Key; Even(Round robin)
• Two types of hardwares as storage
  • HDD is slower but can scale to petabytes (2PB); SSD is faster but can only support to 300+ TB

Storage Deep Dive

• Advertised (pricing) storage is 1/3 of the true utilized storage, because 2/3 used to data copies.
• Blocks : column data persisted as 1MB immutable blocks.
  • With zone map metadata
  • location of next block
  • can be compressed
• Small write has similiar cost with larger write(1~10 rows = 100k rows)
• Update & Delete will only trigger soft delete, use VACUUM or DEEP COPY to delete ghost rows

References

https://youtu.be/iuQgZDs-W7A

Overview

• <1k/TB/Year

Data Ingestion

Security

Symmetric vs Asymmetric encryption

Blockchain

• cryptographically verifiable

Security

• BlastRadius

Try to create a docker image – used to build ear.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

# use same version of ubuntu
# simulate the prod environment
FROM ubuntu:latest
COPY ./TIB_BW_6.4.2_linux26gl23_x86_64.zip /installtb
COPY ./TIB_bwpluginftl_6.4.1_linux26gl23_x86_64.zip /installplugin

CMD cd /installtb
CMD unzip TIB_BW_6.4.2_linux26gl23_x86_64.zip
CMD ./TIBCOUniversalInstaller-lnx-x86-64.bin -silent -V responseFile='TIBCOUniversalInstaller_BW_6.4.2.silent'

CMD cd /installplugin
CMD unzip TIB_bwpluginftl_6.4.1_linux26gl23_x86_64.zip
CMD ./TIBCOUniversalInstaller-lnx-x86-64.bin -silent -V responseFile='TIBCOUniversalInstaller_bwpluginftl_6.4.1.silent'

CMD rm -rf /installplugin
CMD rm -rf /installtb

docker build -t liuruibnu/bw641:v1 .
docker run -it liuruibnu/bw641:v1 ls /opt/tibco
docker run -it liuruibnu/bw641:v1 ls ~/.TIBCO/

Customer Case

• Bring efficiency with deployment
• Single provider with capability in all regions
• Take hot sql server dumps and put into S3
• Issue with Oracle RAC with AWS
• All the environment must be PCI compliant (Payment Card Industry Data Security Standard )

Reference

https://youtu.be/x-DynRJUugU

API Gateway and Lambda

• Make user of IAM to manage security
• Swagger import and client sdk – we can automate most workflows
• Deloyment of API is done by Swagger

Key feature of API Gateway

1. define host different versions of APIs
2. manage network traffic
3. auth

References

https://youtu.be/ZBxWZ9bgd44