Skip to content

Governance/session log#22415

Merged
vitlibar merged 5 commits intoClickHouse:masterfrom
Enmk:governance/session_log
Sep 7, 2021
Merged

Governance/session log#22415
vitlibar merged 5 commits intoClickHouse:masterfrom
Enmk:governance/session_log

Conversation

@Enmk
Copy link
Copy Markdown
Contributor

@Enmk Enmk commented Mar 31, 2021

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

  • New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Logging all successful and failed login, and logout events to a new system.session_log table.
...

Detailed description / Documentation draft:
New system.session_log table, which has all the info about LogIn, LogOut and LogIn Failure events.
Info that is logged:

  • User name
  • event type (LogIn, LogOut, LoginFailure)
  • Event date\time\time with microseconds
  • authentication type (same as for IDENTIFIED BY of CREATE USER statement)
  • array of active settings profiles upon login
  • array of active roles upon login
  • client address and port
  • interface (TCP\HTTP\MySQL\PostgreSQL, etc.)
  • client info (name, version info)
  • optional LoginFailure reason text message.

implementation note: User 'sessions' are implicit and rely on the proper co-operation from Context users.

Added some tests to verify that events are properly saved with all necessary info via the following interfaces:

  • TCP
  • HTTP
  • MySQL

...

@robot-clickhouse robot-clickhouse added doc-alert pr-feature Pull request with new product feature labels Mar 31, 2021
@vitlibar vitlibar self-assigned this Apr 6, 2021
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code style: please move const UUID & user_id to the next line.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, thanks

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

inconsistent indent

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure we need the column authenticated by because it's already included in the system.users table. Or it's better to name it auth_type, i.e. exactly how it's named in system.users.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

profiles and roles can be changed by a logged user via commands SET PROFILE and SET ROLE. Maybe it's worth to log those changes too, what do you think?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well that could be a good extension to add after merging this PR

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

settings_profile seems to be not used

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Custom code in destructors rarely looks good. I think we can add a new member ext::scope_guard add_logout_event_to_session_log to the Context class, and set it in setUser() right after you add a login event to system.session_log. It allows to keep the destructor default, keep all session_log-related code in one place, and ensure that we add a logout event only if we add a login event.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unfortunately, ext::scope_guard would make Context uncopyable and I would have to craft a custom full-blown constructor.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to move more code away from Context.cpp. We could add a function like

SessionLog::addLoginEvent(const Context & context)

which will collect all the required information from the session context, make a session log element and add it just in one function.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't throw any exceptions not derived from std::exception, so this part seems to be unnecessary.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a bit more complicated. A parent profile can have its own parent profile, and so on. substituteProfiles() is a better place to collect all such parent profiles. And are you sure we need those parent profiles in system.session_log? Just a list of enabled roles isn't enough?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC profiles can be assigned not only to roles, but to users too, and maybe to something else. So roles alone are definitely not enough.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are two kinds of active roles. Current roles are the roles which are chosen to be used by user. Enabled roles includes

  1. all the current roles, and
  2. all the roles which are granted to the current roles, and
  3. all the roles which are granted to 2, and
  4. all the roles which are granted to 3 and so on.

I mean maybe we want to see the current roles in the session_log table, not the enabled ones, what do you think?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, makes sense

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's maybe better to use Authentication::TypeInfo to fill those enum values. Just to repeat the names of these constants one less time.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it would be better to turn your test to an integration test. Just to affect other tests less. But I don't insist.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function's name doesn't look good. The function doesn't terminate any session, because to terminate a session you kind of should destroy the corresponding TCPHandler. And because of you're definitely not going to do something like this here I think it's a bad name for the function. Let's rename the function to addLogOutToSessionLog() and move the condition where you check if it's a session context to the destructor.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's more obvious to pass *this (i.e. the context) to addLogOut(), not access. And the same thing for addLogIn(). Also access (the private variable) can be null, please use the function getAccess() (which never returns null) instead.

And too many checks here, it seems the following would be enough:

auto session_log = getSessionLog();
if (session_log)
  session_log->addLogOut(*this);

Here no locks are required.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've thought about this more. It seems after all the destructor is simply not the best place for this logic. I believe we should call this function from handlers, i.e. from TCPHandler, HTTPHandler and so on. The reasons:

  1. Handlers provide more flexibility for the moment when we adds a logout event (for example we can do what we want with named sessions in HTTPHandler if we are not bound to the destructor);
  2. Destructors look better without any logic except freeing the resources;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved out all session-related code to a separate handle-like class: Session. So the log-out event is now saved in Session's destructor.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like your idea with the new Session class, it's nice to move some session-related logic outside of the quite overcomplicated Context class.

Copy link
Copy Markdown
Member

@vitlibar vitlibar May 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's name those profiles current, not active.
Likewise how roles are named: we have current roles and enabled roles. So we would have current profiles, and enabled profiles (enabled profiles would include current profiles, plus their parents, plus parents of those parents, and so on). Well you don't need enabled profiles for your PR but I think we still should name those profiles as current for consistency.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SettingsProfileCache keeps a map named all_profiles, which can be used to get all those names. Searching in the map is faster than the call manager->readName().

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can easily implement this by adding a new member variable of type const SettingsProfileCache & to EnabledSettings and also the function EnabledSettings::getCurrentProfileNames().

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done, in a bit different way

Copy link
Copy Markdown
Member

@vitlibar vitlibar May 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's insert one more And to make the name like setSettingsAndConstraintsAndProfiles(). Because settings and settings constraints are different things. Or I think we can call it simply set() because there is no other set function in this class.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

= default looks a bit better

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't forget to remove the dead code like this.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AppliedProfiles -> CurrentProfiles ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

applied_profiles -> current_profiles ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function should not return const std::vector<UUID> &. It should return either std::shared_ptr<const std::vector<UUID>> or std::vector<UUID> instead. Because if it returns const std::vector<UUID> & it can be read outside the internal lock and be modified in setSettingsAndConstraintsAndProfiles() at the same time.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems there is no point for this function to handle multiple profiles at once. This logic can be implemented in the caller and so it would allow make this function more simple. I mean this function can be declared instead as std::optional<String> getProfileName(const UUID & profile_id) const and the caller will iterate through IDs if it wants.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, fixed

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Who owns this session?

Copy link
Copy Markdown
Contributor Author

@Enmk Enmk Jun 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Somebody else, this is non-owning pointer. I had to poke this "hole" to remove MySQL-specific struct from context. So the MySQL-stream can access the sequence_id from MySQLSession (which was previously in Context::mysql)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a comment to clarify it

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is context_to_copy? Is this the global context?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That depends, usually it is a server context or something similar.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does the function getUserAuthentication() exist? I think we already have the function setUser() for checking password.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are some complex authentication scenarios, e.g. in PostgreSql or MySQL handlers:

auto user = connection_context->getAccessControlManager().read<User>(user_name);

Copy link
Copy Markdown
Member

@vitlibar vitlibar Jun 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So now we have two separate classes: Session and NamedSession which look completely different. It doesn't look good to me. Can we somehow combine them? For example we could add all the fields from NamedSession into Session and make the class NamedSessions to keep Sessions instead of NamedSessions.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed NamedSession from Context.cpp, and now we have a NamedSessionData in the Session.cpp.

Enmk added 3 commits August 30, 2021 18:28
Which logs all the info about LogIn, LogOut and LogIn Failure events.
Additional info that is logged:
- User name
- event type (LogIn, LogOut, LoginFailure)
- Event date\time\time with microseconds
- authentication type (same as for IDENTIFIED BY of CREATE USER statement)
- array of active settings profiles upon login
- array of active roles upon login
- array of changed settings with corresponding values
- client address and port
- interface (TCP\HTTP\MySQL\PostgreSQL, etc.)
- client info (name, version info)
- optional LoginFailure reason text message.

Added some tests to verify that events are properly saved with all necessary info via following interfaces:
- TCP
- HTTP
- MySQL

Known limitations
- Not tested against named HTTP sessions, PostgreSQL and gRPC, hence those are not guaranteed to work 100% properly.
@vitlibar vitlibar merged commit 59148fa into ClickHouse:master Sep 7, 2021
@vitlibar
Copy link
Copy Markdown
Member

vitlibar commented Sep 7, 2021

01747_system_session_log_long takes about 40 seconds which is too long to pass Functional stateless tests flaky check (address), but it will not be running in master this way so it's kind of OK. Other failed tests are not related.

@gyuton
Copy link
Copy Markdown
Contributor

gyuton commented Sep 15, 2021

Internal documentation ticket: DOCSUP-14784.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-feature Pull request with new product feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants