Skip to content

Commit 31805e7

Browse files
author
Mate Szalay-Beko
committed
Merge remote-tracking branch 'apache/master' into ZOOKEEPER-3188
2 parents 0f95678 + 794adf1 commit 31805e7

59 files changed

Lines changed: 3589 additions & 89 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎conf/log4j.properties‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,3 +63,20 @@ log4j.appender.TRACEFILE.File=${zookeeper.tracelog.dir}/${zookeeper.tracelog.fil
6363
log4j.appender.TRACEFILE.layout=org.apache.log4j.PatternLayout
6464
### Notice we are including log4j's NDC here (%x)
6565
log4j.appender.TRACEFILE.layout.ConversionPattern=%d{ISO8601} [myid:%X{myid}] - %-5p [%t:%C{1}@%L][%x] - %m%n
66+
#
67+
# zk audit logging
68+
#
69+
zookeeper.auditlog.file=zookeeper_audit.log
70+
zookeeper.auditlog.threshold=INFO
71+
audit.logger=INFO, RFAAUDIT
72+
log4j.logger.org.apache.zookeeper.audit.Log4jAuditLogger=${audit.logger}
73+
log4j.additivity.org.apache.zookeeper.audit.Log4jAuditLogger=false
74+
log4j.appender.RFAAUDIT=org.apache.log4j.RollingFileAppender
75+
log4j.appender.RFAAUDIT.File=${zookeeper.log.dir}/${zookeeper.auditlog.file}
76+
log4j.appender.RFAAUDIT.layout=org.apache.log4j.PatternLayout
77+
log4j.appender.RFAAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
78+
log4j.appender.RFAAUDIT.Threshold=${zookeeper.auditlog.threshold}
79+
80+
# Max log file size of 10MB
81+
log4j.appender.RFAAUDIT.MaxFileSize=10MB
82+
log4j.appender.RFAAUDIT.MaxBackupIndex=10

‎zookeeper-docs/src/main/resources/markdown/html/header.html‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,9 @@
104104
</div>
105105
<div class="menuitem">
106106
<a href="zookeeperReconfig.html">Dynamic Reconfiguration</a>
107+
</div>
108+
<div class="menuitem">
109+
<a href="zookeeperAuditLogs.html">Audit Logs</a>
107110
</div>
108111
</div>
109112
<div onclick="SwitchMenu('menu_4', 'skin/')" id="menu_4Title" class="menutitle">Contributor</div>

‎zookeeper-docs/src/main/resources/markdown/images/zkAuditLogs.jpg‎

27.8 KB
Loading

‎zookeeper-docs/src/main/resources/markdown/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@ archives.
5353
+ [ZooKeeper CLI](zookeeperCLI.html) - a guide on how to use the ZooKeeper command line interface
5454
+ [ZooKeeper Tools](zookeeperTools.html) - a guide on how to use a series of tools for ZooKeeper
5555
+ [ZooKeeper Monitor](zookeeperMonitor.html) - a guide on how to monitor the ZooKeeper
56+
+ [Audit Logging](zookeeperAuditLogs.html) - a guide on how to configure audit logs in ZooKeeper Server and what contents are logged.
5657
+ **Contributors**
5758
Documents for Developers Contributing to the ZooKeeper Open Source Project
5859
+ [ZooKeeper Internals](zookeeperInternals.html) - assorted topics on the inner workings of ZooKeeper

‎zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -956,6 +956,18 @@ property, when available, is noted below.
956956
and restart ZooKeeper process so ZooKeeper can continue normal data
957957
consistency check during recovery process.
958958
Default value is false.
959+
* *audit.enable* :
960+
(Java system property: **zookeeper.audit.enable**)
961+
**New in 3.6.0:**
962+
By default audit logs are disabled. Set to "true" to enable it. Default value is "false".
963+
See the [ZooKeeper audit logs](zookeeperAuditLogs.html) for more information.
964+
965+
* *audit.impl.class* :
966+
(Java system property: **zookeeper.audit.impl.class**)
967+
**New in 3.6.0:**
968+
Class to implement the audit logger. By default log4j based audit logger org.apache.zookeeper.audit
969+
.Log4jAuditLogger is used.
970+
See the [ZooKeeper audit logs](zookeeperAuditLogs.html) for more information.
959971

960972
* *largeRequestMaxBytes* :
961973
(Java system property: **zookeeper.largeRequestMaxBytes**)

‎zookeeper-docs/src/main/resources/markdown/zookeeperAuditLogs.md‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
<!--
2+
Copyright 2002-2004 The Apache Software Foundation
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
//-->
16+
17+
# ZooKeeper Audit Logging
18+
19+
* [ZooKeeper Audit Logs](#ch_auditLogs)
20+
* [ZooKeeper Audit Log Configuration](#ch_reconfig_format)
21+
* [Who is taken as user in audit logs?](#ch_zkAuditUser)
22+
<a name="ch_auditLogs"></a>
23+
24+
## ZooKeeper Audit Logs
25+
26+
Apache ZooKeeper supports audit logs from version 3.6.0. By default audit logs are disabled. To enable audit logs
27+
configure audit.enable=true in conf/zoo.cfg. Audit logs are not logged on all the ZooKeeper servers, but logged only on the servers where client is connected as depicted in below figure.
28+
29+
![Audit Logs](images/zkAuditLogs.jpg)
30+
31+
32+
The audit log captures detailed information for the operations that are selected to be audited. The audit information is written as a set of key=value pairs for the following keys
33+
34+
| Key | Value |
35+
| ----- | ----- |
36+
|session | client session id |
37+
|user | comma separated list of users who are associate with a client session. For more on this, see [Who is taken as user in audit logs](#ch_zkAuditUser).
38+
|ip | client IP address
39+
|operation | any one of the selected operations for audit. Possible values are(serverStart, serverStop, create, delete, setData, setAcl, multiOperation, reconfig, ephemeralZNodeDeleteOnSessionClose)
40+
|znode | path of the znode
41+
|znode type | type of znode in case of creation operation
42+
|acl | String representation of znode ACL like cdrwa(create, delete,read, write, admin). This is logged only for setAcl operation
43+
|result | result of the operation. Possible values are (success/failure/invoked). Result "invoked" is used for serverStop operation because stop is logged before ensuring that server actually stopped.
44+
45+
Below are sample audit logs for all operations, where client is connected from 192.168.1.2, client principal is [email protected], server principal is zookeeper/1[email protected]
46+
47+
user=zookeeper/192.168.1.3 operation=serverStart result=success
48+
session=0x19344730000 user=192.168.1.2,[email protected] ip=192.168.1.2 operation=create znode=/a znode_type=persistent result=success
49+
session=0x19344730000 user=192.168.1.2,[email protected] ip=192.168.1.2 operation=create znode=/a znode_type=persistent result=failure
50+
session=0x19344730000 user=192.168.1.2,[email protected] ip=192.168.1.2 operation=setData znode=/a result=failure
51+
session=0x19344730000 user=192.168.1.2,[email protected] ip=192.168.1.2 operation=setData znode=/a result=success
52+
session=0x19344730000 user=192.168.1.2,[email protected] ip=192.168.1.2 operation=setAcl znode=/a acl=world:anyone:cdrwa result=failure
53+
session=0x19344730000 user=192.168.1.2,[email protected] ip=192.168.1.2 operation=setAcl znode=/a acl=world:anyone:cdrwa result=success
54+
session=0x19344730000 user=192.168.1.2,[email protected] ip=192.168.1.2 operation=create znode=/b znode_type=persistent result=success
55+
session=0x19344730000 user=192.168.1.2,[email protected] ip=192.168.1.2 operation=setData znode=/b result=success
56+
session=0x19344730000 user=192.168.1.2,[email protected] ip=192.168.1.2 operation=delete znode=/b result=success
57+
session=0x19344730000 user=192.168.1.2,[email protected] ip=192.168.1.2 operation=multiOperation result=failure
58+
session=0x19344730000 user=192.168.1.2,[email protected] ip=192.168.1.2 operation=delete znode=/a result=failure
59+
session=0x19344730000 user=192.168.1.2,[email protected] ip=192.168.1.2 operation=delete znode=/a result=success
60+
session=0x19344730001 user=192.168.1.2,[email protected] ip=192.168.1.2 operation=create znode=/ephemral znode_type=ephemral result=success
61+
session=0x19344730001 user=zookeeper/192.168.1.3 operation=ephemeralZNodeDeletionOnSessionCloseOrExpire znode=/ephemral result=success
62+
session=0x19344730000 user=192.168.1.2,[email protected] ip=192.168.1.2 operation=reconfig znode=/zookeeper/config result=success
63+
user=zookeeper/192.168.1.3 operation=serverStop result=invoked
64+
65+
<a name="ch_auditConfig"></a>
66+
67+
## ZooKeeper Audit Log Configuration
68+
69+
By default audit logs are disabled. To enable audit logs configure audit.enable=true in conf/zoo.cfg. Audit logging is done using log4j. Following is the default log4j configuration for audit logs in conf/log4j.properties
70+
71+
#
72+
# zk audit logging
73+
#
74+
zookeeper.auditlog.file=zookeeper_audit.log
75+
zookeeper.auditlog.threshold=INFO
76+
audit.logger=INFO, RFAAUDIT
77+
log4j.logger.org.apache.zookeeper.audit.Log4jAuditLogger=${audit.logger}
78+
log4j.additivity.org.apache.zookeeper.audit.Log4jAuditLogger=false
79+
log4j.appender.RFAAUDIT=org.apache.log4j.RollingFileAppender
80+
log4j.appender.RFAAUDIT.File=${zookeeper.log.dir}/${zookeeper.auditlog.file}
81+
log4j.appender.RFAAUDIT.layout=org.apache.log4j.PatternLayout
82+
log4j.appender.RFAAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
83+
log4j.appender.RFAAUDIT.Threshold=${zookeeper.auditlog.threshold}
84+
85+
# Max log file size of 10MB
86+
log4j.appender.RFAAUDIT.MaxFileSize=10MB
87+
log4j.appender.RFAAUDIT.MaxBackupIndex=10
88+
89+
Change above configuration to customize the auditlog file, number of backups, max file size, custom audit logger etc.
90+
91+
<a name="ch_zkAuditUser"></a>
92+
93+
## Who is taken as user in audit logs?
94+
95+
By default there are only four authentication provider:
96+
97+
* IPAuthenticationProvider
98+
* SASLAuthenticationProvider
99+
* X509AuthenticationProvider
100+
* DigestAuthenticationProvider
101+
102+
User is decided based on the configured authentication provider:
103+
104+
* When IPAuthenticationProvider is configured then authenticated IP is taken as user
105+
* When SASLAuthenticationProvider is configured then client principal is taken as user
106+
* When X509AuthenticationProvider is configured then client certificate is taken as user
107+
* When DigestAuthenticationProvider is configured then authenticated user is user
108+
109+
Custom authentication provider can override org.apache.zookeeper.server.auth.AuthenticationProvider.getUserName(String id)
110+
to provide user name. If authentication provider is not overriding this method then whatever is stored in
111+
org.apache.zookeeper.data.Id.id is taken as user.
112+
Generally only user name is stored in this field but it is up to the custom authentication provider what they store in it.
113+
For audit logging value of org.apache.zookeeper.data.Id.id would be taken as user.
114+
115+
In ZooKeeper Server not all the operations are done by clients but some operations are done by the server itself. For example when client closes the session, ephemeral znodes are deleted by the Server. These deletion are not done by clients directly but it is done the server itself these are called system operations. For these system operations the user associated with the ZooKeeper server are taken as user while audit logging these operations. For example if in ZooKeeper server principal is zookeeper/hadoop.[email protected] then this becomes the system user and all the system operations will be logged with this user name.
116+
117+
user=zookeeper/[email protected] operation=serverStart result=success
118+
119+
120+
If there is no user associate with ZooKeeper server then the user who started the ZooKeeper server is taken as the user. For example if server started by root then root is taken as the system user
121+
122+
user=root operation=serverStart result=success
123+
124+
125+
Single client can attach multiple authentication schemes to a session, in this case all authenticated schemes will taken taken as user and will be presented as comma separated list. For example if a client is authenticate with principal [email protected] and ip 127.0.0.1 then create znode audit log will be as:
126+
127+
session=0x10c0bcb0000 [email protected],127.0.0.1 ip=127.0.0.1 operation=create znode=/a result=success
128+
129+

‎zookeeper-docs/src/main/resources/markdown/zookeeperOver.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -146,6 +146,11 @@ receives a packet saying that the znode has changed. If the
146146
connection between the client and one of the ZooKeeper servers is
147147
broken, the client will receive a local notification.
148148

149+
**New in 3.6.0:** Clients can also set
150+
permanent, recursive watches on a znode that are not removed when triggered
151+
and that trigger for changes on the registered znode as well as any children
152+
znodes recursively.
153+
149154
<a name="Guarantees"></a>
150155

151156
### Guarantees

‎zookeeper-docs/src/main/resources/markdown/zookeeperProgrammers.md‎

Lines changed: 25 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ limitations under the License.
3232
* [ZooKeeper Sessions](#ch_zkSessions)
3333
* [ZooKeeper Watches](#ch_zkWatches)
3434
* [Semantics of Watches](#sc_WatchSemantics)
35+
* [Persistent, Recursive Watches](#sc_WatchPersistentRecursive)
3536
* [Remove Watches](#sc_WatchRemoval)
3637
* [What ZooKeeper Guarantees about Watches](#sc_WatchGuarantees)
3738
* [Things to Remember about Watches](#sc_WatchRememberThese)
@@ -640,6 +641,11 @@ general this all occurs transparently. There is one case where a watch
640641
may be missed: a watch for the existence of a znode not yet created will
641642
be missed if the znode is created and deleted while disconnected.
642643
644+
**New in 3.6.0:** Clients can also set
645+
permanent, recursive watches on a znode that are not removed when triggered
646+
and that trigger for changes on the registered znode as well as any children
647+
znodes recursively.
648+
643649
<a name="sc_WatchSemantics"></a>
644650
645651
### Semantics of Watches
@@ -657,6 +663,21 @@ the events that a watch can trigger and the calls that enable them:
657663
* **Child event:**
658664
Enabled with a call to getChildren.
659665
666+
<a name="sc_WatchPersistentRecursive"></a>
667+
668+
### Persistent, Recursive Watches
669+
670+
**New in 3.6.0:** There is now a variation on the standard
671+
watch described above whereby you can set a watch that does not get removed when triggered.
672+
Additionally, these watches trigger the event types *NodeCreated*, *NodeDeleted*, and *NodeDataChanged*
673+
and, optionally, recursively for all znodes starting at the znode that the watch is registered for. Note
674+
that *NodeChildrenChanged* events are not triggered for persistent recursive watches as it would be redundant.
675+
676+
Persistent watches are set using the method *addWatch()*. The triggering semantics and guarantees
677+
(other than one-time triggering) are the same as standard watches. The only exception regarding events is that
678+
recursive persistent watchers never trigger child changed events as they are redundant.
679+
Persistent watches are removed using *removeWatches()* with watcher type *WatcherType.Any*.
680+
660681
<a name="sc_WatchRemoval"></a>
661682
662683
### Remove Watches
@@ -671,6 +692,8 @@ successful watch removal.
671692
Watcher which was added with a call to getChildren.
672693
* **Data Remove event:**
673694
Watcher which was added with a call to exists or getData.
695+
* **Persistent Remove event:**
696+
Watcher which was added with a call to add a persistent watch.
674697
675698
<a name="sc_WatchGuarantees"></a>
676699
@@ -693,11 +716,11 @@ guarantees:
693716
694717
### Things to Remember about Watches
695718
696-
* Watches are one time triggers; if you get a watch event and
719+
* Standard watches are one time triggers; if you get a watch event and
697720
you want to get notified of future changes, you must set another
698721
watch.
699722
700-
* Because watches are one time triggers and there is latency
723+
* Because standard watches are one time triggers and there is latency
701724
between getting the event and sending a new request to get a watch
702725
you cannot reliably see every change that happens to a node in
703726
ZooKeeper. Be prepared to handle the case where the znode changes

‎zookeeper-jute/src/main/resources/zookeeper.jute‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,14 @@ module org.apache.zookeeper.proto {
7373
vector<ustring>existWatches;
7474
vector<ustring>childWatches;
7575
}
76+
class SetWatches2 {
77+
long relativeZxid;
78+
vector<ustring>dataWatches;
79+
vector<ustring>existWatches;
80+
vector<ustring>childWatches;
81+
vector<ustring>persistentWatches;
82+
vector<ustring>persistentRecursiveWatches;
83+
}
7684
class RequestHeader {
7785
int xid;
7886
int type;
@@ -180,6 +188,10 @@ module org.apache.zookeeper.proto {
180188
class SetACLResponse {
181189
org.apache.zookeeper.data.Stat stat;
182190
}
191+
class AddWatchRequest {
192+
ustring path;
193+
int mode;
194+
}
183195
class WatcherEvent {
184196
int type; // event type
185197
int state; // state of the Keeper client runtime

‎zookeeper-server/src/main/java/org/apache/zookeeper/AddWatchMode.java‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing, software
13+
* distributed under the License is distributed on an "AS IS" BASIS,
14+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
* See the License for the specific language governing permissions and
16+
* limitations under the License.
17+
*/
18+
19+
package org.apache.zookeeper;
20+
21+
/**
22+
* Modes available to {@link ZooKeeper#addWatch(String, Watcher, AddWatchMode)}
23+
*/
24+
public enum AddWatchMode {
25+
/**
26+
* <p>
27+
* Set a watcher on the given path that does not get removed when triggered (i.e. it stays active
28+
* until it is removed). This watcher
29+
* is triggered for both data and child events. To remove the watcher, use
30+
* <tt>removeWatches()</tt> with <tt>WatcherType.Any</tt>. The watcher behaves as if you placed an exists() watch and
31+
* a getData() watch on the ZNode at the given path.
32+
* </p>
33+
*/
34+
PERSISTENT(ZooDefs.AddWatchModes.persistent),
35+
36+
/**
37+
* <p>
38+
* Set a watcher on the given path that: a) does not get removed when triggered (i.e. it stays active
39+
* until it is removed); b) applies not only to the registered path but all child paths recursively. This watcher
40+
* is triggered for both data and child events. To remove the watcher, use
41+
* <tt>removeWatches()</tt> with <tt>WatcherType.Any</tt>
42+
* </p>
43+
*
44+
* <p>
45+
* The watcher behaves as if you placed an exists() watch and
46+
* a getData() watch on the ZNode at the given path <strong>and</strong> any ZNodes that are children
47+
* of the given path including children added later.
48+
* </p>
49+
*
50+
* <p>
51+
* NOTE: when there are active recursive watches there is a small performance decrease as all segments
52+
* of ZNode paths must be checked for watch triggering.
53+
* </p>
54+
*/
55+
PERSISTENT_RECURSIVE(ZooDefs.AddWatchModes.persistentRecursive)
56+
;
57+
58+
public int getMode() {
59+
return mode;
60+
}
61+
62+
private final int mode;
63+
64+
AddWatchMode(int mode) {
65+
this.mode = mode;
66+
}
67+
}

0 commit comments

Comments
 (0)