ATLP-446. Added kerberos in image and validation check for krb user by aklochkova · Pull Request #137 · OHDSI/ArachneExecutionEngine

aklochkova · 2018-07-27T10:41:23Z

No description provided.

added kerberos in image added validation check for krb user

OdysseusBot · 2018-07-27T10:42:06Z

Can one of the admins verify this patch?

wivern · 2018-07-27T12:03:35Z

                CommandBuilder builder = CommandBuilder.newCommand();
                switch (dataSource.getKrbAuthMethod()) {
                    case PASSWORD:
+                        if (StringUtils.isBlank(dataSource.getKrbUser())) {


krbUser is required in both cases, why not to check it once before switch

fixed user checking

wivern · 2018-08-01T10:06:21Z

+
+	[realms]
+	  ODYSSEUSINC.COM = {
+	    admin_server = ovh08.odysseusinc.com


do we really want to tell everyone on github our server address?

wivern · 2018-08-01T10:11:09Z

                            IOUtils.write(dataSource.getKrbKeytab(), out);
                        }
-                        builder.statement("kinit")
+                        builder.statement("C:\\Program Files\\MIT\\Kerberos\\bin\\kinit")


path hardcode!!!

Moreover, this should not work inside docker container

add ability to append realms to krb5.conf

wivern · 2018-08-02T13:57:39Z

+        context.setVariables(parameters);
+        String htmlString = templateEngine.process(templateName, context);
+
+        Source source = new Source(htmlString);


I like thymeleaf but don't you think there's some overkill to construct html and then convert it to text for a simple config file with two lines of content?

wivern · 2018-08-02T14:04:58Z

+
+        File config = new File(configPath);
+        long fileLength = config.length();
+        try (RandomAccessFile raf = new RandomAccessFile(config, "rw"); FileChannel channel = raf.getChannel()) {


Potentially [realms] might be not the latest section of krb5.conf, so simply adding realm to the end of file is dangerous.
Why not to add line just under the [realms] ?
And what about concurrency with RandomAccessFile?

pavgra · 2018-08-03T20:12:45Z


    private static final Logger log = LoggerFactory.getLogger(KerberosService.class);
    public static final String LOG_FILE = "kinit_out.txt";
+    private Map<String, String> krbEnvProps = new HashMap<>();


How is it going to work in multi-threaded env? At least, it should be ThreadLocal, but I'd prefer functional approach

pavgra · 2018-08-03T20:14:02Z

-                    process.destroy();
-                    if (log.isDebugEnabled()) {
-                        klist(workDir);
+                if (environmentMode == RuntimeServiceMode.SINGLE) {


Too big for a single function. Also is not atomic for a single function

pavgra · 2018-08-03T20:16:09Z

+                    String commandStr = StringUtils.join(command, " ");
+                    commandStr = commandStr.replace(tempKeytabName, "krb.keytab");
+                    commandStr = commandStr.replace(kinitPath, "");
+                    krbEnvProps.put("KRB_CONF", "/etc/krb5.conf");


I'd proceed with a new (temp) dedicated file for each chroot instance

…b-to-ee' into ATLP-446-propagate-impalas-keytab-to-ee

wivern · 2018-08-10T11:51:41Z

 import com.odysseusinc.arachne.executionengine.util.ResultCallback;
 import java.io.File;
+import java.util.Map;
+import javafx.util.Pair;


Are you really want to use JavaFX class?

wivern · 2018-08-10T11:52:59Z

+import java.util.Map;
 import java.util.Objects;
 import java.util.concurrent.TimeUnit;
+import javafx.util.Pair;


The same as above

wivern · 2018-08-10T11:56:15Z

            <artifactId>spring-boot-starter-aop</artifactId>
        </dependency>

+        <dependency>


And where the thymeleaf is used?

wivern · 2018-08-10T12:00:26Z

                    } finally {
                        if (!isExternalJail()) {
                            FileUtils.deleteQuietly(runFile);
+                            for (Path path : tmpPaths) {


I see the tmpPaths is used only to cleanup temporary files, but why not do that outside of this method?
Moreover, it looks like some kind of anti-patterns.

agackovka · 2018-08-10T12:23:22Z

            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-aop</artifactId>
        </dependency>
+        <dependency>


i think com.github.jknack.handlebars can be used instead of it?

pavgra · 2018-08-10T14:34:33Z

 COHORT_TARGET_TABLE=$COHORT_TARGET_TABLE PATH=$PATH \
 HOME=$HOME IMPALA_DRIVER_PATH=$IMPALA_DRIVER_PATH \
- Rscript /$ANALYSIS_FILE
+ /etc/krb-with-R.sh "$RUN_KINIT" "$ANALYSIS_FILE"


Why the krb-with-R.sh is used always? What if we run in non-kerberos mode? it is going to fail then

pavgra · 2018-08-10T14:40:06Z

+
+        File config = new File(configPath);
+        if (isInIsolatedMode) {
+            tempConfigPath.set(Files.createTempFile("", ".conf"));


Pls, use functional approach

Impala keytab refactoring

* ARACHNE-2359. Back-merge of multi-statement SQL files execution fix (#138) * ARACHNE-2359. Merge fix (#140) * ATLP-446. Added keytab propagation for Impala (#137) * ATLP-446. Added [domain-realm] section for correct login to impala (#144) * ATLP-446. Fixed cp command in jail.sh (#146) * ATLP-446. Fixed kinit command for user-password mode (#148) * ATLP-446. Fixed execution of kinit on host (#149) * ATLP-446. Extended krb5.conf both for host and for jail areas (#152) * ARACHNE-2556 EE: set default timezone in docker container to UTC (#155) * ARACHNE-2555. Added connection pooling for DN health check (#154) (cherry picked from commit 2bceaad) * Fixed * Fixed * Fixed * Switched container timezone from UTC to GMT (#158) (cherry picked from commit e4187dd) * SqlRender version * Reverted dockerImagePrefix (#165) * ARACHNE-2575. Changed key used in DataSource-s cache to get a new connection when ds username/password changed [back-merge] (#167) * ojdbc version fix * ARACHNE-2627. Added logging and moved metadata extraction into async mode (#171) * fix DatabaseConnector & FeatureExtraction versions (#174) * ARACHNE-2648 Datanode health check does not work as expected * update version to 1.13.1-QA (#188) * 1.14.0-QA (#197) * Merge fix * Merge fix #2 * Removed redundant metadata extraction (merge issue) (#199) * Added DATA_SOURCE_NAME to chroot params (#201) (cherry picked from commit d0c4814) * Start R scripts in isolated PID namespace when chroot mode is enabled (#224) * Start R scripts in isolated PID namespace when chroot mode is enabled * Do not show env variables in ps * Attempt to fix script * fixes issue with non-root user * removes redundant param * Qa backmerge from master (#235) * ARACHNE-1717 1.10.0 * ARACHNE-1854 Updated version * ARACHNE-2005 Release 1.11.2 * ARACHNE-2106 Release 1.11.3 * ARACHNE-2357 Release 1.12.1 * ARACHNE-1668. Updated drivers to support Postgres 10 (cherry picked from commit 41fb866) * Changed version 1.12.2-qa -> 1.12.2 * ARACHNE-2521. Fixed return of execution status * ARACHNE-2539. Fixed execution of SQL containing multiple queries per file in Oracle * Release 1.12.3 * ARACHNE-2555. Added connection pooling for DN health check (#154) * ARACHNE-2556 EE: set default timezone in docker container to UTC (#157) * Switched container timezone from UTC to GMT (#158) * Release 1.14.0 (#203) * Updated version of r-env. Removed unused docker files (#206) (#208) * Updated version of r-env. Removed unused docker files (#206) * rearranged libraries (#206) * change version to qa * change version to qa (#236) * change impala jdbc driver version (#238) * change impala jdbc driver version * change impala jdbc driver version * fix readme * fixes download link * change r-env version (#241) * QA 1.15.0 (#244) * Fixes removal of Keyfile from result zip (#247) * Fixes R dist build. Fixes jail mode for BQ (#249) * Removes extra installs. Updates to reflect changes from R-env (#251) * Adds CDM v5.3.1 detection (#253) * ARACHNE-2722 set right version for sql-render (#261) * Updates python to v3.6 making PLP models run succesfully (#257) * Fixes issue with ojdbc and timezone region (#262) * ARACHNE-2722 - Cohort (simple counts) fails on BigQuery (#264) - use SingleStatementSqlExecutor for BigQuery request

* ATLP-446 added kerberos in image added validation check for krb user * New version: 1.14.0 * ATLP-446 fixed user checking * ATLP-446 add ability to append realms to krb5.conf * ATLP-446. Fixed notes * ATLP-446. Moved kinit to jail * enable kerberos for chroot'ed env * enable kerberos for chroot'ed env * ATLP-446. Fixed notes * ARACHNE-2359. Back-merge of multi-statement SQL files execution fix * ATLP-446. Made kinit in chroot mode * ATLP-446. Changed krb auth enum class * ATLP-446. added spaces to template * ATLP-446. removed extra lines * ATLP-446. turned off cleanup environment for debug * ATLP-446. reverted debug notes back * ATLP-446. set krb admin server properly * ATLP-446. Fixed notes * ATLP-446. Removed unnecessary dependency * ATLP-446. Added condition for kerberos init, fixed notes * Refactoring * ATLP-446. made some refactoring * ATLP-446. made some refactoring * ATLP-446. Added [domain-realm] section for correct login to impala (#145) * ATLP-446. Fixed cp command in jail.sh (#147) * ATLP-446. Fixed execution of kinit on host (#150) * ATLP-446. Extended krb5.conf both for host and for jail areas (#151) * ATLP-524. Replaced kerberos login logic with module from ArachneCommons (#159) * Added login to kerberos by password (#160) * ARACHNE-2555. Added connection pooling for DN health check (#154) (cherry picked from commit 2bceaad) * ARACHNE-2556 EE: set default timezone in docker container to UTC (#155) (cherry picked from commit 4de1a00) * Fixed dockerImagePrefix * Switched container timezone from UTC to GMT (#158) (cherry picked from commit e4187dd) * SqlRender version (cherry picked from commit 10c9884) * Reverted dockerImagePrefix * ARACHNE-2575. Changed key used in DataSource-s cache to get a new connection when ds username/password changed (#169) * Changed ojdbc6 and impala driver versions (#170) * ARACHNE-2627. Added logging and moved metadata extraction into async mode [back_merge] (#172) * Fix R libraries versions (#178) * Fix R libraries versions for Atlas 2.7 (#179) * Adds Support of BQ keyfile json for authentication of EE jobs (#180) * Updates DatabaseConnector version (#184) * Upgrades R packages versions (#186) * Adds Netezza support (#185) * Adds cron scheduled job to refresh Kerberos ticket (#187) * Added parameter to define whether execution results should be deleted after callback (#190) * Prevents R packages updates (#191) * Exposes data source name as Environment variable (#194) * Excluded extra folders from analyses results (#196) * added default exclusion for analysis * Added R packages for new CC analysis. Adds ANALYSIS_ID env param (#195) * added DATA_SOURCE_NAME to params (#201) * Added synchronous mode of analysis execution (#202) * Added synchronous mode of analysis execution * Cleanup * Cleanup * Forgotten diff for sync execution mode (#204) * Added synchronous mode of analysis execution * Cleanup * Cleanup * Forgotten diff for sync mode * Updated version of r-env. Removed unused docker files (#206) (#208) (#209) * Updated version of r-env. Removed unused docker files (#206) * rearranged libraries (#206) (cherry picked from commit 1783b94) * Back merge from 1.14 & version increment (#205) * Build JAR that can be used as a dependency in other projects additionally to executable one (#211) * Use R libs file from DockerEnv repo to avoid duplication (#210) * Fixes build process conforming PLP deps (#216) * Fixes docker build after classifier has been set (#217) * Attempt to fix docker build * Fixes docker build * run apt-get update in non-interactive mode * Remove redundant dependencies (#218) * Adds geospatial libs (#219) * Fixes user home dir (#222) * improve swagger documentation (#220) * Re-uses DataSourceAuthResolver-s from arachne-commons (#230) * Fixes NPE during non-Kerberos auth (#231) * Updates R-env image version (#232) * Develop backmerge from qa (#237) * change r-env version (#240) * Upgrade to zip4j 2.1.1 (#242) * Upgrade to zip4j 2.1.1 * Removed unused ZipException declarations * Fixes removal of Keyfile from result zip (#246) * Fixes R dist build. Fixes jail mode for BQ (#248) * Back merge release 1.15.0 (#266) * ARACHNE-2359. Back-merge of multi-statement SQL files execution fix (#138) * ARACHNE-2359. Merge fix (#140) * ATLP-446. Added keytab propagation for Impala (#137) * ATLP-446. Added [domain-realm] section for correct login to impala (#144) * ATLP-446. Fixed cp command in jail.sh (#146) * ATLP-446. Fixed kinit command for user-password mode (#148) * ATLP-446. Fixed execution of kinit on host (#149) * ATLP-446. Extended krb5.conf both for host and for jail areas (#152) * ARACHNE-2556 EE: set default timezone in docker container to UTC (#155) * ARACHNE-2555. Added connection pooling for DN health check (#154) (cherry picked from commit 2bceaad) * Fixed * Fixed * Fixed * Switched container timezone from UTC to GMT (#158) (cherry picked from commit e4187dd) * SqlRender version * Reverted dockerImagePrefix (#165) * ARACHNE-2575. Changed key used in DataSource-s cache to get a new connection when ds username/password changed [back-merge] (#167) * ojdbc version fix * ARACHNE-2627. Added logging and moved metadata extraction into async mode (#171) * fix DatabaseConnector & FeatureExtraction versions (#174) * ARACHNE-2648 Datanode health check does not work as expected * update version to 1.13.1-QA (#188) * 1.14.0-QA (#197) * Merge fix * Merge fix #2 * Removed redundant metadata extraction (merge issue) (#199) * Added DATA_SOURCE_NAME to chroot params (#201) (cherry picked from commit d0c4814) * Start R scripts in isolated PID namespace when chroot mode is enabled (#224) * Start R scripts in isolated PID namespace when chroot mode is enabled * Do not show env variables in ps * Attempt to fix script * fixes issue with non-root user * removes redundant param * Qa backmerge from master (#235) * ARACHNE-1717 1.10.0 * ARACHNE-1854 Updated version * ARACHNE-2005 Release 1.11.2 * ARACHNE-2106 Release 1.11.3 * ARACHNE-2357 Release 1.12.1 * ARACHNE-1668. Updated drivers to support Postgres 10 (cherry picked from commit 41fb866) * Changed version 1.12.2-qa -> 1.12.2 * ARACHNE-2521. Fixed return of execution status * ARACHNE-2539. Fixed execution of SQL containing multiple queries per file in Oracle * Release 1.12.3 * ARACHNE-2555. Added connection pooling for DN health check (#154) * ARACHNE-2556 EE: set default timezone in docker container to UTC (#157) * Switched container timezone from UTC to GMT (#158) * Release 1.14.0 (#203) * Updated version of r-env. Removed unused docker files (#206) (#208) * Updated version of r-env. Removed unused docker files (#206) * rearranged libraries (#206) * change version to qa * change version to qa (#236) * change impala jdbc driver version (#238) * change impala jdbc driver version * change impala jdbc driver version * fix readme * fixes download link * change r-env version (#241) * QA 1.15.0 (#244) * Fixes removal of Keyfile from result zip (#247) * Fixes R dist build. Fixes jail mode for BQ (#249) * Removes extra installs. Updates to reflect changes from R-env (#251) * Adds CDM v5.3.1 detection (#253) * ARACHNE-2722 set right version for sql-render (#261) * Updates python to v3.6 making PLP models run succesfully (#257) * Fixes issue with ojdbc and timezone region (#262) * ARACHNE-2722 - Cohort (simple counts) fails on BigQuery (#264) - use SingleStatementSqlExecutor for BigQuery request * Optimize cdm detect (#267) * optimization for CDM version detection * Hello Spring boot 2! * Fix logger configuration - failing after spring boot 2 migration (#268) * Migrate deprecated properties to spring-boot-2 schema (#269) * Migrate deprecated properties to spring-boot-2 schema Fix tomcat 9.0.26 packed into spring boot 2.1.9 is have a bug, rolling back to 2.1.8 to stable tomcat 9.0.25 - failing after spring boot 2 migration * downgrade embedded tomcat to 9.0.24 * upgrade embedded tomcat to 9.0.27 * Adds support for Apache Hive LLAP (#272) Fixes: OHDSI/WebAPI#1168 Adds support for Apache Hive LLAP Co-authored-by: anastasiia klochkova <[email protected]> Co-authored-by: Anton Gackovka <[email protected]> Co-authored-by: Vitaly Koulakov <[email protected]> Co-authored-by: Pavel Grafkin <[email protected]> Co-authored-by: mpozhidaeva <[email protected]> Co-authored-by: OdysseusBot <[email protected]> Co-authored-by: ssuvorov-fls <[email protected]> Co-authored-by: AntonStepanof <[email protected]> Co-authored-by: Alex Cumarav <[email protected]>

* Qa backmerge from master (#235) * ARACHNE-1717 1.10.0 * ARACHNE-1854 Updated version * ARACHNE-2005 Release 1.11.2 * ARACHNE-2106 Release 1.11.3 * ARACHNE-2357 Release 1.12.1 * ARACHNE-1668. Updated drivers to support Postgres 10 (cherry picked from commit 41fb866) * Changed version 1.12.2-qa -> 1.12.2 * ARACHNE-2521. Fixed return of execution status * ARACHNE-2539. Fixed execution of SQL containing multiple queries per file in Oracle * Release 1.12.3 * ARACHNE-2555. Added connection pooling for DN health check (#154) * ARACHNE-2556 EE: set default timezone in docker container to UTC (#157) * Switched container timezone from UTC to GMT (#158) * Release 1.14.0 (#203) * Updated version of r-env. Removed unused docker files (#206) (#208) * Updated version of r-env. Removed unused docker files (#206) * rearranged libraries (#206) * change version to qa * change version to qa (#236) * change impala jdbc driver version (#238) * change impala jdbc driver version * change impala jdbc driver version * fix readme * fixes download link * change r-env version (#241) * QA 1.15.0 (#244) * Fixes removal of Keyfile from result zip (#247) * Fixes R dist build. Fixes jail mode for BQ (#249) * Removes extra installs. Updates to reflect changes from R-env (#251) * Adds CDM v5.3.1 detection (#253) * ARACHNE-2722 set right version for sql-render (#261) * Updates python to v3.6 making PLP models run succesfully (#257) * Fixes issue with ojdbc and timezone region (#262) * ARACHNE-2722 - Cohort (simple counts) fails on BigQuery (#264) - use SingleStatementSqlExecutor for BigQuery request * New version: 1.15.1-QA (#270) * new parameter for cran distrib name was added * update formatting * added missing parameter in getopts * QA 1.16.0 (#273) * ATLP-446 added kerberos in image added validation check for krb user * New version: 1.14.0 * ATLP-446 fixed user checking * ATLP-446 add ability to append realms to krb5.conf * ATLP-446. Fixed notes * ATLP-446. Moved kinit to jail * enable kerberos for chroot'ed env * enable kerberos for chroot'ed env * ATLP-446. Fixed notes * ARACHNE-2359. Back-merge of multi-statement SQL files execution fix * ATLP-446. Made kinit in chroot mode * ATLP-446. Changed krb auth enum class * ATLP-446. added spaces to template * ATLP-446. removed extra lines * ATLP-446. turned off cleanup environment for debug * ATLP-446. reverted debug notes back * ATLP-446. set krb admin server properly * ATLP-446. Fixed notes * ATLP-446. Removed unnecessary dependency * ATLP-446. Added condition for kerberos init, fixed notes * Refactoring * ATLP-446. made some refactoring * ATLP-446. made some refactoring * ATLP-446. Added [domain-realm] section for correct login to impala (#145) * ATLP-446. Fixed cp command in jail.sh (#147) * ATLP-446. Fixed execution of kinit on host (#150) * ATLP-446. Extended krb5.conf both for host and for jail areas (#151) * ATLP-524. Replaced kerberos login logic with module from ArachneCommons (#159) * Added login to kerberos by password (#160) * ARACHNE-2555. Added connection pooling for DN health check (#154) (cherry picked from commit 2bceaad) * ARACHNE-2556 EE: set default timezone in docker container to UTC (#155) (cherry picked from commit 4de1a00) * Fixed dockerImagePrefix * Switched container timezone from UTC to GMT (#158) (cherry picked from commit e4187dd) * SqlRender version (cherry picked from commit 10c9884) * Reverted dockerImagePrefix * ARACHNE-2575. Changed key used in DataSource-s cache to get a new connection when ds username/password changed (#169) * Changed ojdbc6 and impala driver versions (#170) * ARACHNE-2627. Added logging and moved metadata extraction into async mode [back_merge] (#172) * Fix R libraries versions (#178) * Fix R libraries versions for Atlas 2.7 (#179) * Adds Support of BQ keyfile json for authentication of EE jobs (#180) * Updates DatabaseConnector version (#184) * Upgrades R packages versions (#186) * Adds Netezza support (#185) * Adds cron scheduled job to refresh Kerberos ticket (#187) * Added parameter to define whether execution results should be deleted after callback (#190) * Prevents R packages updates (#191) * Exposes data source name as Environment variable (#194) * Excluded extra folders from analyses results (#196) * added default exclusion for analysis * Added R packages for new CC analysis. Adds ANALYSIS_ID env param (#195) * added DATA_SOURCE_NAME to params (#201) * Added synchronous mode of analysis execution (#202) * Added synchronous mode of analysis execution * Cleanup * Cleanup * Forgotten diff for sync execution mode (#204) * Added synchronous mode of analysis execution * Cleanup * Cleanup * Forgotten diff for sync mode * Updated version of r-env. Removed unused docker files (#206) (#208) (#209) * Updated version of r-env. Removed unused docker files (#206) * rearranged libraries (#206) (cherry picked from commit 1783b94) * Back merge from 1.14 & version increment (#205) * Build JAR that can be used as a dependency in other projects additionally to executable one (#211) * Use R libs file from DockerEnv repo to avoid duplication (#210) * Fixes build process conforming PLP deps (#216) * Fixes docker build after classifier has been set (#217) * Attempt to fix docker build * Fixes docker build * run apt-get update in non-interactive mode * Remove redundant dependencies (#218) * Adds geospatial libs (#219) * Fixes user home dir (#222) * improve swagger documentation (#220) * Re-uses DataSourceAuthResolver-s from arachne-commons (#230) * Fixes NPE during non-Kerberos auth (#231) * Updates R-env image version (#232) * Develop backmerge from qa (#237) * change r-env version (#240) * Upgrade to zip4j 2.1.1 (#242) * Upgrade to zip4j 2.1.1 * Removed unused ZipException declarations * Fixes removal of Keyfile from result zip (#246) * Fixes R dist build. Fixes jail mode for BQ (#248) * Back merge release 1.15.0 (#266) * ARACHNE-2359. Back-merge of multi-statement SQL files execution fix (#138) * ARACHNE-2359. Merge fix (#140) * ATLP-446. Added keytab propagation for Impala (#137) * ATLP-446. Added [domain-realm] section for correct login to impala (#144) * ATLP-446. Fixed cp command in jail.sh (#146) * ATLP-446. Fixed kinit command for user-password mode (#148) * ATLP-446. Fixed execution of kinit on host (#149) * ATLP-446. Extended krb5.conf both for host and for jail areas (#152) * ARACHNE-2556 EE: set default timezone in docker container to UTC (#155) * ARACHNE-2555. Added connection pooling for DN health check (#154) (cherry picked from commit 2bceaad) * Fixed * Fixed * Fixed * Switched container timezone from UTC to GMT (#158) (cherry picked from commit e4187dd) * SqlRender version * Reverted dockerImagePrefix (#165) * ARACHNE-2575. Changed key used in DataSource-s cache to get a new connection when ds username/password changed [back-merge] (#167) * ojdbc version fix * ARACHNE-2627. Added logging and moved metadata extraction into async mode (#171) * fix DatabaseConnector & FeatureExtraction versions (#174) * ARACHNE-2648 Datanode health check does not work as expected * update version to 1.13.1-QA (#188) * 1.14.0-QA (#197) * Merge fix * Merge fix #2 * Removed redundant metadata extraction (merge issue) (#199) * Added DATA_SOURCE_NAME to chroot params (#201) (cherry picked from commit d0c4814) * Start R scripts in isolated PID namespace when chroot mode is enabled (#224) * Start R scripts in isolated PID namespace when chroot mode is enabled * Do not show env variables in ps * Attempt to fix script * fixes issue with non-root user * removes redundant param * Qa backmerge from master (#235) * ARACHNE-1717 1.10.0 * ARACHNE-1854 Updated version * ARACHNE-2005 Release 1.11.2 * ARACHNE-2106 Release 1.11.3 * ARACHNE-2357 Release 1.12.1 * ARACHNE-1668. Updated drivers to support Postgres 10 (cherry picked from commit 41fb866) * Changed version 1.12.2-qa -> 1.12.2 * ARACHNE-2521. Fixed return of execution status * ARACHNE-2539. Fixed execution of SQL containing multiple queries per file in Oracle * Release 1.12.3 * ARACHNE-2555. Added connection pooling for DN health check (#154) * ARACHNE-2556 EE: set default timezone in docker container to UTC (#157) * Switched container timezone from UTC to GMT (#158) * Release 1.14.0 (#203) * Updated version of r-env. Removed unused docker files (#206) (#208) * Updated version of r-env. Removed unused docker files (#206) * rearranged libraries (#206) * change version to qa * change version to qa (#236) * change impala jdbc driver version (#238) * change impala jdbc driver version * change impala jdbc driver version * fix readme * fixes download link * change r-env version (#241) * QA 1.15.0 (#244) * Fixes removal of Keyfile from result zip (#247) * Fixes R dist build. Fixes jail mode for BQ (#249) * Removes extra installs. Updates to reflect changes from R-env (#251) * Adds CDM v5.3.1 detection (#253) * ARACHNE-2722 set right version for sql-render (#261) * Updates python to v3.6 making PLP models run succesfully (#257) * Fixes issue with ojdbc and timezone region (#262) * ARACHNE-2722 - Cohort (simple counts) fails on BigQuery (#264) - use SingleStatementSqlExecutor for BigQuery request * Optimize cdm detect (#267) * optimization for CDM version detection * Hello Spring boot 2! * Fix logger configuration - failing after spring boot 2 migration (#268) * Migrate deprecated properties to spring-boot-2 schema (#269) * Migrate deprecated properties to spring-boot-2 schema Fix tomcat 9.0.26 packed into spring boot 2.1.9 is have a bug, rolling back to 2.1.8 to stable tomcat 9.0.25 - failing after spring boot 2 migration * downgrade embedded tomcat to 9.0.24 * upgrade embedded tomcat to 9.0.27 * Adds support for Apache Hive LLAP (#272) Fixes: OHDSI/WebAPI#1168 Adds support for Apache Hive LLAP Co-authored-by: anastasiia klochkova <[email protected]> Co-authored-by: Anton Gackovka <[email protected]> Co-authored-by: Vitaly Koulakov <[email protected]> Co-authored-by: Pavel Grafkin <[email protected]> Co-authored-by: mpozhidaeva <[email protected]> Co-authored-by: OdysseusBot <[email protected]> Co-authored-by: ssuvorov-fls <[email protected]> Co-authored-by: AntonStepanof <[email protected]> Co-authored-by: Alex Cumarav <[email protected]> * fixing merge issue * fixes tests Co-authored-by: ssuvorov-fls <[email protected]> Co-authored-by: Alex Cumarav <[email protected]> Co-authored-by: Pavel Grafkin <[email protected]> Co-authored-by: molodkov yaroslav <[email protected]> Co-authored-by: Sergey Suvorov <[email protected]> Co-authored-by: anastasiia klochkova <[email protected]> Co-authored-by: Anton Gackovka <[email protected]> Co-authored-by: mpozhidaeva <[email protected]> Co-authored-by: OdysseusBot <[email protected]> Co-authored-by: AntonStepanof <[email protected]>

* ARACHNE-2359. Back-merge of multi-statement SQL files execution fix (#138) * ARACHNE-2359. Merge fix (#140) * ATLP-446. Added keytab propagation for Impala (#137) * ATLP-446. Added [domain-realm] section for correct login to impala (#144) * ATLP-446. Fixed cp command in jail.sh (#146) * ATLP-446. Fixed kinit command for user-password mode (#148) * ATLP-446. Fixed execution of kinit on host (#149) * ATLP-446. Extended krb5.conf both for host and for jail areas (#152) * ARACHNE-2556 EE: set default timezone in docker container to UTC (#155) * ARACHNE-2555. Added connection pooling for DN health check (#154) (cherry picked from commit 2bceaad) * Fixed * Fixed * Fixed * Switched container timezone from UTC to GMT (#158) (cherry picked from commit e4187dd) * SqlRender version * Reverted dockerImagePrefix (#165) * ARACHNE-2575. Changed key used in DataSource-s cache to get a new connection when ds username/password changed [back-merge] (#167) * ojdbc version fix * ARACHNE-2627. Added logging and moved metadata extraction into async mode (#171) * fix DatabaseConnector & FeatureExtraction versions (#174) * ARACHNE-2648 Datanode health check does not work as expected * update version to 1.13.1-QA (#188) * 1.14.0-QA (#197) * Merge fix * Merge fix #2 * Removed redundant metadata extraction (merge issue) (#199) * Added DATA_SOURCE_NAME to chroot params (#201) (cherry picked from commit d0c4814) * Start R scripts in isolated PID namespace when chroot mode is enabled (#224) * Start R scripts in isolated PID namespace when chroot mode is enabled * Do not show env variables in ps * Attempt to fix script * fixes issue with non-root user * removes redundant param * Qa backmerge from master (#235) * ARACHNE-1717 1.10.0 * ARACHNE-1854 Updated version * ARACHNE-2005 Release 1.11.2 * ARACHNE-2106 Release 1.11.3 * ARACHNE-2357 Release 1.12.1 * ARACHNE-1668. Updated drivers to support Postgres 10 (cherry picked from commit 41fb866) * Changed version 1.12.2-qa -> 1.12.2 * ARACHNE-2521. Fixed return of execution status * ARACHNE-2539. Fixed execution of SQL containing multiple queries per file in Oracle * Release 1.12.3 * ARACHNE-2555. Added connection pooling for DN health check (#154) * ARACHNE-2556 EE: set default timezone in docker container to UTC (#157) * Switched container timezone from UTC to GMT (#158) * Release 1.14.0 (#203) * Updated version of r-env. Removed unused docker files (#206) (#208) * Updated version of r-env. Removed unused docker files (#206) * rearranged libraries (#206) * change version to qa * change version to qa (#236) * change impala jdbc driver version (#238) * change impala jdbc driver version * change impala jdbc driver version * fix readme * fixes download link * change r-env version (#241) * QA 1.15.0 (#244) * Fixes removal of Keyfile from result zip (#247) * Fixes R dist build. Fixes jail mode for BQ (#249) * Removes extra installs. Updates to reflect changes from R-env (#251) * Adds CDM v5.3.1 detection (#253) * ARACHNE-2722 set right version for sql-render (#261) * Updates python to v3.6 making PLP models run succesfully (#257) * Fixes issue with ojdbc and timezone region (#262) * ARACHNE-2722 - Cohort (simple counts) fails on BigQuery (#264) - use SingleStatementSqlExecutor for BigQuery request * New version: 1.15.1-QA (#270) * new parameter for cran distrib name was added * update formatting * added missing parameter in getopts * QA 1.16.0 (#273) * ATLP-446 added kerberos in image added validation check for krb user * New version: 1.14.0 * ATLP-446 fixed user checking * ATLP-446 add ability to append realms to krb5.conf * ATLP-446. Fixed notes * ATLP-446. Moved kinit to jail * enable kerberos for chroot'ed env * enable kerberos for chroot'ed env * ATLP-446. Fixed notes * ARACHNE-2359. Back-merge of multi-statement SQL files execution fix * ATLP-446. Made kinit in chroot mode * ATLP-446. Changed krb auth enum class * ATLP-446. added spaces to template * ATLP-446. removed extra lines * ATLP-446. turned off cleanup environment for debug * ATLP-446. reverted debug notes back * ATLP-446. set krb admin server properly * ATLP-446. Fixed notes * ATLP-446. Removed unnecessary dependency * ATLP-446. Added condition for kerberos init, fixed notes * Refactoring * ATLP-446. made some refactoring * ATLP-446. made some refactoring * ATLP-446. Added [domain-realm] section for correct login to impala (#145) * ATLP-446. Fixed cp command in jail.sh (#147) * ATLP-446. Fixed execution of kinit on host (#150) * ATLP-446. Extended krb5.conf both for host and for jail areas (#151) * ATLP-524. Replaced kerberos login logic with module from ArachneCommons (#159) * Added login to kerberos by password (#160) * ARACHNE-2555. Added connection pooling for DN health check (#154) (cherry picked from commit 2bceaad) * ARACHNE-2556 EE: set default timezone in docker container to UTC (#155) (cherry picked from commit 4de1a00) * Fixed dockerImagePrefix * Switched container timezone from UTC to GMT (#158) (cherry picked from commit e4187dd) * SqlRender version (cherry picked from commit 10c9884) * Reverted dockerImagePrefix * ARACHNE-2575. Changed key used in DataSource-s cache to get a new connection when ds username/password changed (#169) * Changed ojdbc6 and impala driver versions (#170) * ARACHNE-2627. Added logging and moved metadata extraction into async mode [back_merge] (#172) * Fix R libraries versions (#178) * Fix R libraries versions for Atlas 2.7 (#179) * Adds Support of BQ keyfile json for authentication of EE jobs (#180) * Updates DatabaseConnector version (#184) * Upgrades R packages versions (#186) * Adds Netezza support (#185) * Adds cron scheduled job to refresh Kerberos ticket (#187) * Added parameter to define whether execution results should be deleted after callback (#190) * Prevents R packages updates (#191) * Exposes data source name as Environment variable (#194) * Excluded extra folders from analyses results (#196) * added default exclusion for analysis * Added R packages for new CC analysis. Adds ANALYSIS_ID env param (#195) * added DATA_SOURCE_NAME to params (#201) * Added synchronous mode of analysis execution (#202) * Added synchronous mode of analysis execution * Cleanup * Cleanup * Forgotten diff for sync execution mode (#204) * Added synchronous mode of analysis execution * Cleanup * Cleanup * Forgotten diff for sync mode * Updated version of r-env. Removed unused docker files (#206) (#208) (#209) * Updated version of r-env. Removed unused docker files (#206) * rearranged libraries (#206) (cherry picked from commit 1783b94) * Back merge from 1.14 & version increment (#205) * Build JAR that can be used as a dependency in other projects additionally to executable one (#211) * Use R libs file from DockerEnv repo to avoid duplication (#210) * Fixes build process conforming PLP deps (#216) * Fixes docker build after classifier has been set (#217) * Attempt to fix docker build * Fixes docker build * run apt-get update in non-interactive mode * Remove redundant dependencies (#218) * Adds geospatial libs (#219) * Fixes user home dir (#222) * improve swagger documentation (#220) * Re-uses DataSourceAuthResolver-s from arachne-commons (#230) * Fixes NPE during non-Kerberos auth (#231) * Updates R-env image version (#232) * Develop backmerge from qa (#237) * change r-env version (#240) * Upgrade to zip4j 2.1.1 (#242) * Upgrade to zip4j 2.1.1 * Removed unused ZipException declarations * Fixes removal of Keyfile from result zip (#246) * Fixes R dist build. Fixes jail mode for BQ (#248) * Back merge release 1.15.0 (#266) * ARACHNE-2359. Back-merge of multi-statement SQL files execution fix (#138) * ARACHNE-2359. Merge fix (#140) * ATLP-446. Added keytab propagation for Impala (#137) * ATLP-446. Added [domain-realm] section for correct login to impala (#144) * ATLP-446. Fixed cp command in jail.sh (#146) * ATLP-446. Fixed kinit command for user-password mode (#148) * ATLP-446. Fixed execution of kinit on host (#149) * ATLP-446. Extended krb5.conf both for host and for jail areas (#152) * ARACHNE-2556 EE: set default timezone in docker container to UTC (#155) * ARACHNE-2555. Added connection pooling for DN health check (#154) (cherry picked from commit 2bceaad) * Fixed * Fixed * Fixed * Switched container timezone from UTC to GMT (#158) (cherry picked from commit e4187dd) * SqlRender version * Reverted dockerImagePrefix (#165) * ARACHNE-2575. Changed key used in DataSource-s cache to get a new connection when ds username/password changed [back-merge] (#167) * ojdbc version fix * ARACHNE-2627. Added logging and moved metadata extraction into async mode (#171) * fix DatabaseConnector & FeatureExtraction versions (#174) * ARACHNE-2648 Datanode health check does not work as expected * update version to 1.13.1-QA (#188) * 1.14.0-QA (#197) * Merge fix * Merge fix #2 * Removed redundant metadata extraction (merge issue) (#199) * Added DATA_SOURCE_NAME to chroot params (#201) (cherry picked from commit d0c4814) * Start R scripts in isolated PID namespace when chroot mode is enabled (#224) * Start R scripts in isolated PID namespace when chroot mode is enabled * Do not show env variables in ps * Attempt to fix script * fixes issue with non-root user * removes redundant param * Qa backmerge from master (#235) * ARACHNE-1717 1.10.0 * ARACHNE-1854 Updated version * ARACHNE-2005 Release 1.11.2 * ARACHNE-2106 Release 1.11.3 * ARACHNE-2357 Release 1.12.1 * ARACHNE-1668. Updated drivers to support Postgres 10 (cherry picked from commit 41fb866) * Changed version 1.12.2-qa -> 1.12.2 * ARACHNE-2521. Fixed return of execution status * ARACHNE-2539. Fixed execution of SQL containing multiple queries per file in Oracle * Release 1.12.3 * ARACHNE-2555. Added connection pooling for DN health check (#154) * ARACHNE-2556 EE: set default timezone in docker container to UTC (#157) * Switched container timezone from UTC to GMT (#158) * Release 1.14.0 (#203) * Updated version of r-env. Removed unused docker files (#206) (#208) * Updated version of r-env. Removed unused docker files (#206) * rearranged libraries (#206) * change version to qa * change version to qa (#236) * change impala jdbc driver version (#238) * change impala jdbc driver version * change impala jdbc driver version * fix readme * fixes download link * change r-env version (#241) * QA 1.15.0 (#244) * Fixes removal of Keyfile from result zip (#247) * Fixes R dist build. Fixes jail mode for BQ (#249) * Removes extra installs. Updates to reflect changes from R-env (#251) * Adds CDM v5.3.1 detection (#253) * ARACHNE-2722 set right version for sql-render (#261) * Updates python to v3.6 making PLP models run succesfully (#257) * Fixes issue with ojdbc and timezone region (#262) * ARACHNE-2722 - Cohort (simple counts) fails on BigQuery (#264) - use SingleStatementSqlExecutor for BigQuery request * Optimize cdm detect (#267) * optimization for CDM version detection * Hello Spring boot 2! * Fix logger configuration - failing after spring boot 2 migration (#268) * Migrate deprecated properties to spring-boot-2 schema (#269) * Migrate deprecated properties to spring-boot-2 schema Fix tomcat 9.0.26 packed into spring boot 2.1.9 is have a bug, rolling back to 2.1.8 to stable tomcat 9.0.25 - failing after spring boot 2 migration * downgrade embedded tomcat to 9.0.24 * upgrade embedded tomcat to 9.0.27 * Adds support for Apache Hive LLAP (#272) Fixes: OHDSI/WebAPI#1168 Adds support for Apache Hive LLAP Co-authored-by: anastasiia klochkova <[email protected]> Co-authored-by: Anton Gackovka <[email protected]> Co-authored-by: Vitaly Koulakov <[email protected]> Co-authored-by: Pavel Grafkin <[email protected]> Co-authored-by: mpozhidaeva <[email protected]> Co-authored-by: OdysseusBot <[email protected]> Co-authored-by: ssuvorov-fls <[email protected]> Co-authored-by: AntonStepanof <[email protected]> Co-authored-by: Alex Cumarav <[email protected]> * fixes tests (#275) * new parameter for cran distrib name was added * update formatting * added missing parameter in getopts * changes cran mirror to actual * changes mirror to Canada provider * changes mirror to Canada provider * fixes path * One more try fo url * Rc 1.16.1 (#284) * execution engine 1.16.1 RC * degrading guava * make env vars formatting bash compatible Co-authored-by: alexandr cumarav <https://localhost:8080/study-manager/studies/6> * source short name was sanitized to prevent zipping problem in case when short name contains slash(OHDSI/WebAPI#1542) * revert(OHDSI/WebAPI#1542) * source short name was sanitized to prevent zipping problem in case when short name contains slash(OHDSI/WebAPI#1542) (#287) Co-authored-by: Sergey Suvorov <[email protected]> * Cumulated fixes for BigQuery in R packages (#288) * Potential fix for SQLRender/BigQuery * Attempt to fix python libs * Adding parameters for CRAN mirror and R branch version * Fix parameters list * Fix typo error * Ubuntu libs were added * Roll back adding Ubuntu packages Co-authored-by: Konstantin Yaroshovets <[email protected]> * CDM detection improvements (#285) * Support optional fields for CDM detection, introduce one more results file for the cdm_detection_notes.txt Co-authored-by: alexandr cumarav <https://localhost:8080/study-manager/studies/6> * use the lates r-env image * Failback to scikit-learn 0.22.2 * upload submission: exceeds the configured maximum [ARACHNE-2818] (#294) * updated r-env version * updated python version for dist * updated version of R * use miniconda * add link for miniconda Co-authored-by: Pavel Grafkin <[email protected]> Co-authored-by: Vitaly Koulakov <[email protected]> Co-authored-by: Maria Pozhidaeva <[email protected]> Co-authored-by: mpozhidaeva <[email protected]> Co-authored-by: Konstantin Iaroshovets <[email protected]> Co-authored-by: Alex Cumarav <[email protected]> Co-authored-by: AntonStepanof <[email protected]> Co-authored-by: ssuvorov-fls <[email protected]> Co-authored-by: Alex Cumarav <[email protected]> Co-authored-by: molodkov yaroslav <[email protected]> Co-authored-by: Sergey Suvorov <[email protected]> Co-authored-by: Anton Gackovka <[email protected]> Co-authored-by: OdysseusBot <[email protected]> Co-authored-by: Konstantin Yaroshovets <[email protected]>

ATLP-446

e1ea01f

added kerberos in image added validation check for krb user

aklochkova requested review from pavgra and wivern July 27, 2018 10:41

wivern requested changes Jul 27, 2018

View reviewed changes

ATLP-446

cc33435

fixed user checking

wivern requested changes Aug 1, 2018

View reviewed changes

ATLP-446

60729f5

add ability to append realms to krb5.conf

wivern requested changes Aug 2, 2018

View reviewed changes

ATLP-446. Fixed notes

91b163b

wivern previously approved these changes Aug 3, 2018

View reviewed changes

ATLP-446. Moved kinit to jail

9ca2451

aklochkova dismissed wivern’s stale review via 9ca2451 August 3, 2018 17:53

pavgra suggested changes Aug 3, 2018

View reviewed changes

wivern and others added 11 commits August 6, 2018 14:58

enable kerberos for chroot'ed env

b190c5d

Merge remote-tracking branch 'origin/ATLP-446-propagate-impalas-keyta…

e026829

…b-to-ee' into ATLP-446-propagate-impalas-keytab-to-ee

enable kerberos for chroot'ed env

c391daf

ATLP-446. Fixed notes

8e9c358

ATLP-446. Made kinit in chroot mode

c2f93ed

ATLP-446. Changed krb auth enum class

4377eba

ATLP-446. added spaces to template

581438b

ATLP-446. removed extra lines

6a3d4a8

ATLP-446. turned off cleanup environment for debug

d8dbe1c

ATLP-446. reverted debug notes back

d036d87

ATLP-446. set krb admin server properly

8d1215e

wivern requested changes Aug 10, 2018

View reviewed changes

ATLP-446. Fixed notes

c679e80

wivern previously approved these changes Aug 10, 2018

View reviewed changes

agackovka reviewed Aug 10, 2018

View reviewed changes

ATLP-446. Removed unnecessary dependency

c5e40d4

aklochkova dismissed wivern’s stale review via c5e40d4 August 10, 2018 12:32

pavgra suggested changes Aug 10, 2018

View reviewed changes

aklochkova and others added 5 commits August 10, 2018 21:55

ATLP-446. Added condition for kerberos init, fixed notes

ebee68b

Refactoring

7e4ed92

Merge pull request #142 from OHDSI/impala-keytab-refactoring

ae4d808

Impala keytab refactoring

ATLP-446. made some refactoring

7e511c8

ATLP-446. made some refactoring

4f7001e

pavgra approved these changes Aug 13, 2018

View reviewed changes

pavgra merged commit d064dee into qa Aug 13, 2018

Uh oh!

Conversation

aklochkova commented Jul 27, 2018

Uh oh!

OdysseusBot commented Jul 27, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants