Posted on |

Nifi Kafka Cluster Test Cases

Topology

Due to environment limitation, the test environment are two Linux servers creating an Ambari cluster with,

  • A zookeeper cluster shared by nifi and Kafka cluster,
  • A nifi cluster with two nodes,
  • A Kafka cluster

Message Sequencing Tests

Message Sequencing Test – topic without partition

Test0, Basic scenario, Subscribe & Process Kafka messages from Nifi Cluster (Single Threading)

image

  • Pre-Step, put sequentially 10000 messages in TopicA and 10000 messages in TopicB
    • both topic with 2 replicas, no partition
    • messages are putting into the queue with sequence, (like Msg1, Msg2…)
  • Test Step, Start Nifi Flows
    • one flow subscribe TopicA and put subscribed message into TopicA.D
    • one flow subscribe TopicB and put subscribed message into TopicB.D
    • all nifi processors running on Nifi cluster and configured as “Concurrent Tasks =1” and Execution = "Primary node"
      image
    • subscribe processor is set to consume from “earliest” and with unique client Id.
  • After the step, read messages from TopicA.D and TopicB.D, verify that messages are in sequence.

Conclusion: messages still remained the sequence, although the Nifi have 2 instances.

Read more »

Posted on |

common IOT scenario

device: sensor -->micro controller

common IOT Protocols

wifi , zigbee, mqtt, sigbox

Demo 1

  1. Connect sensor with micro controller to connect to computer
  2. upload the code into sensor , test when connected
  • sensor the height between the sensor to the ground (or blocking level)

Demo 2

Based on Demo 1, send the data to cloud platform

Read more »

Posted on |

Big data and hadoop for beginners

This is based on,
https://www.udemy.com/big-data-and-hadoop-for-beginners/learn/v4/overview

Big data Overview

  • structured data (database, excel) , semi structured data (Xml , Json), unstructured data (log)
  • 5 Vs of big data: Volume (Terabytes, ZettaBytes) ; Vilocity (the speed of data generating or moving around); Variety ( structured~unstructured); Veracity (data quality); Value (value etracted from data)

big data use case

  • mobile advertisement company
  • telco, finiance, retailer

big data jobs

big data analyst; hadoop administrator ; big data engineer; big data scentist;
big data manager; big data solution archi ; chief data officer

ETL vs ELT

Read more »

Posted on |
  • Symmetric vs Asymmetric encryption

https://www.ssl2buy.com/wiki/symmetric-vs-asymmetric-encryption-what-are-differences

  • 对称密钥的意思就是加密解密用一样的密钥
  • 非对称加密就是公钥私钥交叉加密的方式

x.509 certification

An X.509 certificate contains a public key and an identity (a hostname, or an organization, or an individual), and is either signed by a certificate authority or self-signed.

The structure of an X.509 v3 digital certificate is as follows:

Certificate
Version Number
Serial Number
Signature Algorithm ID
Issuer Name
Validity period
Not Before
Not After
Subject name
Subject Public Key Info
Public Key Algorithm
Subject Public Key
Issuer Unique Identifier (optional)
Subject Unique Identifier (optional)
Extensions (optional)
…
Certificate Signature Algorithm
Certificate Signature

Certificate filename extensions

There are several commonly used filename extensions for X.509 certificates. Unfortunately, some of these extensions are also used for other data such as private keys.

  • .pem – (Privacy-enhanced Electronic Mail) Base64 encoded DER certificate, enclosed between “-----BEGIN CERTIFICATE-----” and “-----END CERTIFICATE-----”
  • .cer, .crt, .der – usually in binary DER form, but Base64-encoded certificates are common too (see .pem above)
  • .p7b, .p7c – PKCS#7 SignedData structure without data, just certificate(s) or CRL(s)
  • .p12 – PKCS#12, may contain certificate(s) (public) and private keys (password protected)
  • .pfx – PFX, predecessor of PKCS#12 (usually contains data in PKCS#12 format, e.g., with PFX files generated in IIS)
Read more »

Posted on |

updates

  • 12 May 2018
    • update Hexo , Next Theme ; add DISQUS

How to update the Hexo & Theme

  1. update Hexo

update the package.json

  1. clone the Hexo Branch back to local
1
2
3
4
5
6
7
8
9
10
11
git clone -b hexo git@github.com:racheliurui/racheliurui.github.io.git

# for example we want next v5.1.2
mkdir themes/next512
curl -L https://api.github.com/repos/iissnan/hexo-theme-next/tarball/v5.1.2 | tar -zxv -C themes/next512 --strip-components=1

# update racheliurui.github.io/_config.yml pointing to new theme (folder name)
# backup & update new theme's _config.yml file (referring to old theme config file)
# Refer to Theme's github website to do any extra config
# git add -A; git commit "udpates"
git push  origin hexo

other themes

https://github.com/ptsteadman/hexo-theme-corporate-example

https://www.duyidong.com/2017/03/07/Deploy-Hexo-to-S3/

Read more »

Posted on |

for Redhat Linux

  • list all the repositories

  • find current folder size
    findmnt /tmp -o SOURCE,FSTYPE,SIZE,USED,AVAIL,USE%,TARGET

  • centos minimal might not contains netstat

yum install net-tools

https://cyruslab.net/2014/07/11/installing-netstat-on-centos-7-minimal-installation/

list block device

1
lsblk

https://www.centos.org/docs/5/html/Deployment_Guide-en-US/ch-lvm.html

Type:
disk
part
lvm
rom

SSH Tunnels

https://scriptingosx.com/2017/07/ssh-tunnels/

Read more »

Posted on |

for Redhat Linux

  • list all the repositories

Common Linux

  • find current folder size
1
2
findmnt /tmp -o SOURCE,FSTYPE,SIZE,USED,AVAIL,USE%,TARGET
findmnt /var -o AVAIL
  • check current user group
1
2
$ groups username
$ usermod -a -G groupname username
  • check RAM and CPU information
1
cat /proc/cpuinfo

subscription-manager

Read more »
0%