Reference
Terminology
- massively parallel, share Nothing Columnar architecture
Best Practices: Encoding & Compression
https://youtu.be/lj8oaSpCFTc?t=657
- Use AZt4
AWS - HPC
AWS - KMS
- If you want to use AWS managed keys, then you can’t control key rotation, it would be every 3 years.
- If you want to use Customer Managed Keys (CMK), you can turn on automatic rotation for sysmetric keys, it would be every year.
- CMK sysmetric key and asysmetric private key never left KMS unencrypted
- How to choose from Sysmetric and Asysmetric key
https://docs.aws.amazon.com/kms/latest/developerguide/symm-asymm-choose.html
LoraWAN
References
Terminology
Lora vs LoraWAN
- Lora is the protocol, __Lo__ng __Ra__nge ; LoRa is Layer2
- LoraWAN is the IoT solution based on Lora technology
Lora Pro/Cons
- ISM Open frequency(415,868,915MHz, free ; no license required
- Interference ; low data rate
Limitations / Parameters
Target: transmission message about 10 km and the battery last for 2 years.
- Frequency: Pay attention to band requirement per country
- Tx power (transmission power): 2-14 dbm / 5-20 dBm; the higher the power , the longer distance signals can cover
- Bandwidth (125/250/500 KHz): the higher the more data can be include in one transmission; the higher the bandwidth, the shorter battery life, the shorter range and more interference.(??); checked the local laws
- spreading factor: (7-12), the larger spreading factor, the longer distance and shorter battery life.
- coding rate: 4/5, 4/6, 4/7, 4/8,
4/5 means 5 error bits used to correct 4 bit of data. The more coding rate, means your data can transfer longer distance but lower battery life.
AWS - Security
Reference
AWS re:Inforce 2019: Security Best Practices the Well-Architected Way (SDD318)
Incident response
https://d1.awsstatic.com/whitepapers/aws_security_incident_response.pdf
Playbook vs Runbook: run book have more details
https://docs.aws.amazon.com/guardduty/latest/ug/guardduty_finding-types-active.html
- predefined query against cloudwatch event
IAM
AWS - Reference Case, API First
AWS - Kinesis
Reference
Why Streaming
- Data loses value quickly over time
- “Time critical decisions” need streaming data
- inject as it’s generated, process on the fly and do real-time analytics/ML/Alert/Action
- Common streaming use case
- Smart home / automation / log / Data Lake / IoT
- Real time analytics demo (User Dashboard)
Streams Producers and Consumers
Producer limits
- bandwidth limitation: 1MB/sec/shard
- if not, aggregate your message, and use throughput limitation: 1k record/sec/shard
Normal consumer
-
The slowest consumer will also impact number of shards, you might need increase the shards to allow the slowest consumer can process the message concurrently to pick up all the messages
-
The fastest speed you can get the data is one trasaction per 200ms
-
Multiple consumers share the 5 transaction/sec/shard and 1M data / sec /shard limitations.
- Multiple consumers will decrease the troughput as well as increase your latency
-
Workaround , use master stream and copied slave stream