Posted on | 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
[root@hdf-server01 ambari-server]# ambari-server setup-ldap
Using python  /usr/bin/python
Setting up LDAP properties...
Primary URL* {host:port}: ldapServer:389
Secondary URL {host:port} :
Use SSL* [true/false] (false):
User object class* (person):
User name attribute* (sAMAccountName):
Group object class* (group):
Group name attribute* (cn):
Group member attribute* (member):
Distinguished name attribute* ():CN=UserToPullUserData,OU=IT Department,DC=hortonworks
Base DN* :OU=IT Department,OU=IT hortonworks,DC=hortonworks
Referral method [follow/ignore] (ignore):
Bind anonymously* [true/false] (false):
Handling behavior for username collisions [convert/skip] for LDAP sync* (skip):
Manager DN* :CN=UserToPullUserData,OU=IT Department,DC=hortonworks
Base DN* :OU=IT Department,OU=IT hortonworks,DC=hortonworks
Enter Manager Password* :
Re-enter password:
====================
Review Settings
====================
authentication.ldap.managerDn: CN=UserToPullUserData,OU=IT Department,DC=hortonworks
Base DN* :OU=IT Department,OU=IT hortonworks,DC=hortonworks
authentication.ldap.managerPassword: *****
Save settings [y/n] (y)?
Saving...done
Ambari Server 'setup-ldap' completed successfully.

After setup , run below command to trigger the sync.

1
2
ambari restart
ambari-server sync-ldap --all
Read more »

Posted on | 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
[root@hdf-server01 ambari-server]# ambari-server setup-ldap
Using python  /usr/bin/python
Setting up LDAP properties...
Primary URL* {host:port}: ldapServer:389
Secondary URL {host:port} :
Use SSL* [true/false] (false):
User object class* (person):
User name attribute* (sAMAccountName):
Group object class* (group):
Group name attribute* (cn):
Group member attribute* (member):
Distinguished name attribute* ():CN=UserToPullUserData,OU=IT Department,DC=hortonworks
Base DN* :OU=IT Department,OU=IT hortonworks,DC=hortonworks
Referral method [follow/ignore] (ignore):
Bind anonymously* [true/false] (false):
Handling behavior for username collisions [convert/skip] for LDAP sync* (skip):
Manager DN* :CN=UserToPullUserData,OU=IT Department,DC=hortonworks
Base DN* :OU=IT Department,OU=IT hortonworks,DC=hortonworks
Enter Manager Password* :
Re-enter password:
====================
Review Settings
====================
authentication.ldap.managerDn: CN=UserToPullUserData,OU=IT Department,DC=hortonworks
Base DN* :OU=IT Department,OU=IT hortonworks,DC=hortonworks
authentication.ldap.managerPassword: *****
Save settings [y/n] (y)?
Saving...done
Ambari Server 'setup-ldap' completed successfully.

After setup , run below command to trigger the sync.

1
2
ambari restart
ambari-server sync-ldap --all
Read more »

Posted on |

forgot the username and password

https://community.hortonworks.com/content/supportkb/49508/how-to-change-grafana-admin-password-when-the-pass.html

Ranger UI password missing after switching the UI authentication

https://community.hortonworks.com/questions/4408/is-there-any-way-to-reset-ranger-admin-ui-password.html

1
vi /var/lib/pgsql/data/pg_hba.conf

add below line to give access,

1
local all angerdba trust
1
psql rangerdb -U rangerdba
1
update x_portal_user set password = 'ceb4f32325eda6142bd65215f4c0f371' where login_id = 'admin';
Read more »

Posted on |

https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka

https://medium.com/@andrew_schofield/does-apache-kafka-do-acid-transactions-647b207f3d0e

So, does Apache Kafka do ACID transactions? Absolutely not. No way. Can you get a similar effect? If you design your applications in the right way, yes. Does it matter? In many cases, not really, but when it does, you absolutely don’t want to get it wrong. Just take the time to understand the guarantees that you need to make your system reliable and choose accordingly.

DB->Topic xa transaction, can do

Topic->DB xa transaction, hard to implmement

Read more »

Posted on |

Kafka monitoring

The JMX for kafka is by default turned on by Ambari

  • default installation will set JMX port at 16667
  • default installation with no security turned on

Check the default JMX monitoring settings

From local machine, switch to the JDK folder and run jconsole

1
<JDK_Home>/bin/jconsole

With the UI, use kafkaBrokerhostName:16667 as the connection string to connect to Kafka.
jConsole_Kafka

Read more »

Posted on | 
1
2
3
4
5
mkdir -p /opt/kafka-tools
cp -R /path/kafka-manager-1.3.3.7 /opt/kafka-tools
nohup /opt/kafka-tools/kafka-manager-1.3.3.7/bin/kafka-manager -Dconfig.file=/opt/kafka-tools/kafka-manager-1.3.3.7/conf/hdf-server.conf -Dhttp.port=8888 >/dev/null 2>&1 &

nohup java -jar /opt/kafka-tools/kafkadrop/kafdrop-2.0.0.jar --zookeeper.connect=hdf-server03:2181,hdf-server04:2181,hdf-server05:2181 --server.port=8889 >/dev/null 2>&1 &
Read more »

Posted on |

Kafka Common configuration

Auto start

For Ambari managed Kafka, configure autostart from Console

http://[ambari-host]:8080/#/main/admin/serviceAutoStart

By default, the Kafka and zookeeper “autostart” function provided by Ambari is stopped. For any env other than DEV this is be modified to enabled.

Change Kafka configuration

To change kakfa configuration,

  • For Ambari managed Kafka Cluster,
    • Modify from Ambari Console
  • For manually configured Kafka Cluster
  • modify the kafka.properties file.

Configurations

  • Allow Topic auto created
Read more »

Posted on |

Kerberos

Kafka SASL relying on Kerberized cluster.

configurations for enable Kerberos via Ambari wizard

Configuration Name Value
KDC Type Existing Active Directory
KDC hosts kdcserver1,kdcserver2
Realm name DOMAINNAME.CAPITAL.NET
LDAP url ldaps://ldapserver1.domainname.capital.net:636
Container DN OU=AmbariCluster, DC=net
Domains DOMAINNAME
Kadmin host kdcserver1
Admin principal SUPERUSER
Admin password password

mandatory configuration for Nifi when Kerberos is enabled

Specify the kerberos provider

make sure the kerberos-provider details is defined at “Template for login-identity-providers.xml”.

1
2
3
4
5
6
<provider>
        <identifier>kerberos-provider</identifier>
        <class>org.apache.nifi.kerberos.KerberosProvider</class>
        <property name="Default Realm">DOMAINNAME.CAPITAL.NET</property>
        <property name="Authentication Expiration">12 hours</property>
</provider>

Check the user mapping

After kerbereros is enabled, the ldap user name logged in may contains domain like username@domain.com

Read more »

Posted on | 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
[root@hdf-server01 ambari-server]# ambari-server setup-ldap
Using python  /usr/bin/python
Setting up LDAP properties...
Primary URL* {host:port}: ldapServer:389
Secondary URL {host:port} :
Use SSL* [true/false] (false):
User object class* (person):
User name attribute* (sAMAccountName):
Group object class* (group):
Group name attribute* (cn):
Group member attribute* (member):
Distinguished name attribute* ():CN=UserToPullUserData,OU=IT Department,DC=hortonworks
Base DN* :OU=IT Department,OU=IT hortonworks,DC=hortonworks
Referral method [follow/ignore] (ignore):
Bind anonymously* [true/false] (false):
Handling behavior for username collisions [convert/skip] for LDAP sync* (skip):
Manager DN* :CN=UserToPullUserData,OU=IT Department,DC=hortonworks
Base DN* :OU=IT Department,OU=IT hortonworks,DC=hortonworks
Enter Manager Password* :
Re-enter password:
====================
Review Settings
====================
authentication.ldap.managerDn: CN=UserToPullUserData,OU=IT Department,DC=hortonworks
Base DN* :OU=IT Department,OU=IT hortonworks,DC=hortonworks
authentication.ldap.managerPassword: *****
Save settings [y/n] (y)?
Saving...done
Ambari Server 'setup-ldap' completed successfully.

After setup , run below command to trigger the sync.

1
2
ambari restart
ambari-server sync-ldap --all
Read more »
0%