Docker command cheet sheet

Posted on 2018-07-08 |

build a image

docker build -t account/repo:tag .
#  specify the build file
docker build -t account/repo:tag --file dockerfilename .

#debug generated image


# create a volumn
$ docker run -it -v "$PWD":/usr/src/mymaven -v "$HOME/.m2":/root/.m2 -v "$PWD/target:/usr/src/mymaven/target" -w /usr/src/mymaven maven mvn clean package  

docker volume create --name workspace

docker run -it -v workspace:/workspace maven mvn archetype:generate # will download artifacts
docker run -it -v workspace:/workspace maven mvn archetype:generate # will reuse downloaded artifacts
docker run etpartner/tibco:bw642ftl ls /opt/tibco/bw/bw/6.4/bin/bwdesign


docker cloud
creat repo ; then
docker login
docker push

AWS - Cloudformation for EC2

Posted on 2018-07-08 |

ssh -i /path/my-key-pair.pem ec2-user@ec2-198-51-100-1.compute-1.amazonaws.com

Cloudformation for EC2

https://github.com/awslabs/aws-cloudformation-templates/blob/master/aws/services/EC2/EC2InstanceWithSecurityGroupSample.yaml

AWSTemplateFormatVersion: '2010-09-09'
Metadata:
  License: Apache-2.0
Description: 'AWS CloudFormation Sample Template EC2InstanceWithSecurityGroupSample:
  Create an Amazon EC2 instance running the Amazon Linux AMI. The AMI is chosen based
  on the region in which the stack is run. This example creates an EC2 security group
  for the instance to give you SSH access. **WARNING** This template creates an Amazon
  EC2 instance. You will be billed for the AWS resources used if you create a stack
  from this template.'
Parameters:
  KeyName:
    Description: Name of an existing EC2 KeyPair to enable SSH access to the instance
    Type: AWS::EC2::KeyPair::KeyName
    ConstraintDescription: must be the name of an existing EC2 KeyPair.
  InstanceType:
    Description: WebServer EC2 instance type
    Type: String
    Default: t2.small
    AllowedValues: [t1.micro]
    ConstraintDescription: must be a valid EC2 instance type.
  SSHLocation:
    Description: The IP address range that can be used to SSH to the EC2 instances
    Type: String
    MinLength: 9
    MaxLength: 18
    Default: 0.0.0.0/0
    AllowedPattern: (\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})/(\d{1,2})
    ConstraintDescription: must be a valid IP CIDR range of the form x.x.x.x/x.
Mappings:
  AWSInstanceType2Arch:
    t1.micro:
      Arch: PV64
  AWSInstanceType2NATArch:
    t1.micro:
      Arch: NATPV64
  AWSRegionArch2AMI:
    us-east-1:
      PV64: ami-2a69aa47
      HVM64: ami-6869aa05
      HVMG2: ami-50b4f047
    us-east-2:
      PV64: NOT_SUPPORTED
      HVM64: ami-f6035893
      HVMG2: NOT_SUPPORTED
Resources:
  EC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: !Ref 'InstanceType'
      SecurityGroups: [!Ref 'InstanceSecurityGroup']
      KeyName: !Ref 'KeyName'
      ImageId: !FindInMap [AWSRegionArch2AMI, !Ref 'AWS::Region', !FindInMap [AWSInstanceType2Arch,
          !Ref 'InstanceType', Arch]]
  InstanceSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Enable SSH access via port 22
      SecurityGroupIngress:
      - IpProtocol: tcp
        FromPort: 22
        ToPort: 22
        CidrIp: !Ref 'SSHLocation'
Outputs:
  InstanceId:
    Description: InstanceId of the newly created EC2 instance
    Value: !Ref 'EC2Instance'
  AZ:
    Description: Availability Zone of the newly created EC2 instance
    Value: !GetAtt [EC2Instance, AvailabilityZone]
  PublicDNS:
    Description: Public DNSName of the newly created EC2 instance
    Value: !GetAtt [EC2Instance, PublicDnsName]
  PublicIP:
    Description: Public IP address of the newly created EC2 instance
    Value: !GetAtt [EC2Instance, PublicIp]

AWS - Cloudformation for S3

Posted on 2018-07-08 |

Template for S3 bitbucket

Delete the stack will delete the S3 bucket

AWSTemplateFormatVersion: '2010-09-09'
Description: "Sample CloudFormation Template: this template will define a statck with S3 bucket"
Resources:
   MyS3Bucket:
       Type: AWS::S3::Bucket
       Properties:
           AccessControl: PublicRead
           Tags:
             -
               Key: "S3BucketName"
               Value: "Dev"             
Outputs:
  BucketName:
    Value: !Ref 'MyS3Bucket'
    Description: Name of the sample Amazon S3 bucket with a notification configuration.

AWS - CloudSearch

Posted on 2018-07-02 |

AWS CloudSearch

Overview

Key difference between db query and search engine:
DB query will give you exact result, search engine only gives you best result.

DeepDive

Core based on Apache Solr
Key info with deepdive
- setting up
- Scaling
- Queyring
- Architecting

Setting up

Create and config a domain ( cli )
Create batches ( go to data store, change data format to Solr supported type )
- Use maximum batches (5M bytes) – use max sized batches
- conver data: remove bad charactors
integrate with IAM (who can connect to which domain)
integrate with cloudtrail

Scaling

tip: increase instance type for load-in
- Test against different type of data (tweets vs web data)
- Options will have effect on index size
- muti-threads to upload data ( test the limites to avoid 500 error)
set multi partition
Pre-warm for traffic spike

Query

AWS - Workspace

Posted on 2018-07-02 |

AWS WorkSpaces and WorkSpaces Application Manager

One user one VM; data based on EBS
AWS workspaces is for desktops; AWS EC2 is for servers
Integrate with existing tools: AD;Intranet; MFA; SCCM (System Center Configuration Manager)
Work well with a lot of patterns
- BYOD (bring your own devices)

Updates

For standard bundle, no upgrade cost
support BYOL (bring your won lisence)
Volumn Encryption with AWS KMS
FMA (Radius )
- Remote Authentication Dial-In User Service (RADIUS) is a networking protocol, operating on port 1812 that provides centralized Authentication, Authorization
Certification - SOC 1,2 ISO9001 and 27001

Demo

after setting up, connect to existin AD
Using wizard to launch workspace
- create user in AD
- select bundle ( laptop with certain images )
- wait 20 min till workspace fully launched
User use workspace client to connect to the server

WAM (workspaces Application manager)

Deploy track and update apps on user’s workspaces
bring your won apss or subscribe apps from aws marketpalce
Gain availabity and control over app usage
support app versioning

Demo

WAM has a catelog containing all apps the current account owns
select muti apps from catelog and assign to AD user or groups
during assign wizard, configure options allows
- assign certain version of the apps
- installation type (optional or required);
- auto install or optional install
uninstall from WAM
- by removing the user from app subscription

Tibco -- FTL

Posted on 2018-07-02 |

Tibco FTL Community Edition

Default installation location on MAC.
The package was installed into /opt/tibco/ftl/5.4

start a realm server

1	sudo /opt/tibco/ftl/current-version/bin/tibrealmserver

start developing java client

# install client jar into maven
mvn install:install-file -Dfile=/opt/tibco/ftl/current-version/lib/tibftl.jar -DgroupId=tibco.ftl.client \
    -DartifactId=ftlclient -Dversion=5.4 -Dpackaging=jar

# For code using  com.tibco.ftl.group.*;
mvn install:install-file -Dfile=/opt/tibco/ftl/current-version/lib/tibftlgroup.jar -DgroupId=tibco.ftl.client \
    -DartifactId=ftlgroupclient -Dversion=5.4 -Dpackaging=jar

When running java , using -Djava.library.path=/opt/tibco/ftl/current-version/lib

References

Doc home
https://docs.tibco.com/pub/ftl/5.4.0/doc/html/GUID-C0EBAF36-A682-445D-B3EE-8E683330BE07-homepage.html

AWS - Config

Posted on 2018-06-29 |

AWS Config

Like the Ambari console config change

Key Features

Record config -->S3
- Record metadata;
- attributes;
- relationships; ( for example , an EBS is attached to an EC2)
- current config;
- related CloudTrail events
Normalize
Store
Deliver: report , snapshot
- can aggregate to a single S3 from multi accounts across multi Region
- Can make use of SNS ->SQS to aggregate

AWS Config Rules

make use of the AWS Config Deliver to set up rules to check config changes ; visualize compliance and identify offending changes

Trigger by change

Tag: for example anything with tag “Production” changed then the rule is triggerred
Resource type
Resource ID

Trigger by frequency

Use Cases

AWS - IAM

Posted on 2018-06-29 |

10 IAM best practises

Identity and Credential Management

Create Individual Users
Configure a credential policy
Rotate the credentials Regularly
Enable MFA for privileged users (software or hardware)
Manage permissions by group
Grant least privilege
Use IAM roles to share access; (benefit, no password share and no need to long-term store credentials)

For example, PROD env account trust Dev env account; and then config a role to have access to database; Dev account then assign certain user to assume the role to manage the database.;

Use IAM roles for EC2 instance.(launch the EC2 with role)
Use CloudTrail to get logs of API calls
Reduce or remove the use of root credential

fine grained access control by resource tags

Account management

Reference

https://youtu.be/SGntDzEn30s

Maven - Refresh

Posted on 2018-06-25 |

Key Terminology

https://maven.apache.org/guides/

LifeCycles
Project Inheritence vs Aggregation vs Mixed

https://maven.apache.org/guides/introduction/introduction-to-the-pom.html

Build profile
- per Project
- per user
- Global
- Profile Descriptor (dynamically loaded in project)

https://maven.apache.org/guides/introduction/introduction-to-profiles.html

# Maven Command

# mvn groupId:artifactId:goal -Denvironment=test -P profile1,!profile2

mvn com.mycom.app1:module1:deploy -Denvironment=test -P profile1, !profile2
mvn help:active-profiles -Denv=dev
mvn help:effective-pom -P appserverConfig-dev

Maven with Docker

Hands on to build a maven java project as a docker image
https://examples.javacodegeeks.com/devops/docker/introduction-docker-java-developers/

Tibco -- BusinessWorks

Posted on 2018-06-22 |

#Start the IDE
~/tibco/bwce/studio/4.0/eclipse/TIBCOBusinessStudio.app/Contents/MacOS/TIBCOBusinessStudio

rm -rf ~/Downloads/tempworkspace
mkdir ~/Downloads/tempworkspace

cd ~/tibco/bwce/bwce/2.3/bin
./bwdesign -data ~/Downloads/tempworkspace system:import -f /Users/ruiliu/Downloads/temp/workspacetest/tib_bw_ci_poc
./bwdesign -data ~/Downloads/tempworkspace system:export hello.world.application ~/Downloads

ls ~/Downloads

Quick Start

https://aws-quickstart.s3.amazonaws.com/quickstart-tibco-bwce/doc/tibco-bwce-on-the-aws-cloud.pdf

This pricing model enables you to pay only for the number of containers running per hour and gives you flexibility to
scale on demand and manage software costs as you scale.

Software Pricing Details
TIBCO BusinessWorks™ Container Edition and Plug-ins for AWS
$0.4 /Host/hr
Infrastructure Pricing Details
Estimated Infrastructure Cost
$5/month
The table shows current software and infrastructure pricing for services hosted in Asia Pacific (Singapore). Additional taxes or fees may apply.

TIBCO BusinessWorks™ Container Edition and Plug-ins for AWS
Unit Type /Host/hr
Unit = 1 TIBCO BusinessWorks Consumption Unit $0.2
1 TIBCO BWCE App Container = 5 TIBCO BusinessWorks Consumption Units $1
1 TIBCO BW Plug-in = 2 TIBCO BusinessWorks Consumption Units $0.4

------ ECS
onfiguration and 24x7 usage. Different CloudFormation configurations may result in different infrastructure costs

EC2 2 x m4.large machine or equivalent
ELB 20 GB Per Month
NAT Gateway 2 Connections using 10 GB Per Month
EBS 60 GB General Purpose SSD

-----ami
Estimated Infrastructure Cost
$0.125 EC2/hr
running on m4.large
-----Docker

Estimated Infrastructure Cost
$5/month
Estimated infrastructure costs are based on the following default deployment configuration and 24x7 usage. Different CloudFormation configurations may result in different infrastructure costs