would it be sufficient to be able to specify the jnidispatch name prior to JNA load, for instance by setting a jna.library.name property?

On Jul 20, 2011, at 6:27 PM, tussis wrote:

Please consider the following scenarios that warrant the inclusion of side-by-side deployment of jnidispatch libraries on Windows 64-bit OS with WoW support for 32-bit programs.

We have an API module that is written in Java and distributed in one of two ways on Windows platforms:

• a Java jar file
• a managed-code .NET DLL - a.NET version of the API which is generated from the Java jar file with IKVM resulting in the DLL
  We support multiple windows platforms and the code may be used in a JVM (JRE) or a .NET application (CLR).

In the case where the .NET API programmer is creating a module to be deployed to the Windows GAC (Global Assembly Cache), the programmer

• will not be targeting any specific architecture and will most likely not have the ability to dictate execution path requirements.
• is typically is not familiar with Java and various JRE requirements and only wants to use our module like it was a native .NET module.

Our API documentation states for the API user to copy DLLs to the assemly's bin folder:

• several architecture-neutral (managed code) IKVM DLLs
• two IKVM native (unmanaged code) DLLs (one for 32-bit and one for 64-bit)
• our architecture-neutral DLL

This allows the module to be loaded without problem by either 64-bit or 32-bit Windows processes from the single location in the Windows System GAC.

If the shut-down-on-exit bug were not an issue or the fix worked reliably and did not leave stray DLLs, we would have no further problems. Also, it obviously preferable to deploy the jnidispatch DLLs since it is more efficient than creating and deleting temporary DLLs every time a process runs.

So, it seems that having a simple technique for finding the platform-specific jnidispatch library by naming convention would be unobtrusive and easy to deploy for our customers. We would simply add the two DLLs (jnidispatch-win32-amd64.dll and jnidispath-win32-x86.dll) to the assembly's bin directory.

Reply to this email directly or view it on GitHub:
#13

Thank you for reading my explanation above - I know it was rather lengthy. The crux of the problem in the situations we find our code running is that we have little control of the execution environment. The code can be loaded by processes written by third parties and in environments that are very different from traditional JVMs. In particular, being able to be used in a stateless web application environment is the worst. What I'm trying to avoid most is burdening our API users with tricky initialization problems or shift the "execution setup" problem to them.

Are you suggesting that some of the initialization code we write as part of our distributed classes set this (system) property based on the environment we find ourselves in? Then the the JNA code would use that name if the property is set? Obviously, that type of thing would work. However, it seems onerous to require that of any JNA user in this situation re-write such code for various modules. However, I may just be misunderstanding your approach.

The only real difference I made in Native.java was (after searching the boot path) in the Windows case attempt loadlibrary of platform-specific-named version of the library then try the default name. Please ignore the interim commit since it did this for all platforms and even the boot path which was too much.

Thanks again for your patience on this topic. It would greatly help our deployment issues if this change (or similar) could be incorporated into the main.

On Jul 20, 2011, at 9:02 PM, tussis wrote:

Are you suggesting that some of the initialization code we write as part of our distributed classes set this (system) property based on the environment we find ourselves in? Then the the JNA code would use that name if the property is set? Obviously, that type of thing would work. However, it seems onerous to require that of any JNA user in this situation re-write such code for various modules. However, I may just be misunderstanding your approach.

I don't think it too onerous to require of those that use JNA in peculiar situations to tweak a few settings to allow JNA to work in their environment, rather than customizing JNA itself to work in that specific environment.

Your library/module would do this before the first access of JNA (basically before JNA attempts to load its dispatch library):
System.setProperty("jna.library.file", is64Bit ? "jnidispatch64.dll" : "jnidispatch.dll");

If not set, JNA uses a default. If set, you can point it at pretty much any path in your system, relative or absolute. This is both simpler than, and less specific, than trying out some name variations looking for a match.

Whether the property is set or not, JNA's code path remains unchanged, and only the data used is different.

Hi Chris,

On Jul 23, 2011, at 2:57 PM, Chris Milam wrote:

My main concern is that there is a documented and consistent manner to allow safe deployment of (out) modules using JNA in the environments mandated by our customers. Please realize that under some circumstances, the customer's IT department dictates how DLLs may be installed and sometimes do not allow "temporary" DLL injection - especially unmanaged code. Another aspect is that if other code in the same process/class loader, there is no coordination of how to consistently handle loading the jnidispatch library in these situations.

In most cases where JNA is being used in an environment is regularly loaded and unloaded, the most common solution has been to incorporate JNA into the container environment rather than having it attached to the transient module. Whatever the actual solution, though, the automatic unpacking and disposal of the JNA native library is not appropriate for these situations, if for no other reason than because it's a waste of resources to repeatedly unpack the jnidispatch library.

The issue from JNA's perspective is how best to allow these custom configurations the latitude they need for a successful deployment. In almost all cases, this is done by providing the jnidispatch library in the system PATH. That PATH is going to be unique for 32-bit and 64-bit processes.

Now it may be that in the case of some ikvm deployments (and it would help here if you gave a full description of said deployment), only part of PATH is unique, and it's only the non-unique part of PATH you get to write to.

I believe that adding a property for the library name would make things worse in certain ways. If this is the technique you would provide then my instructions to our customers would be something along the lines of:

To allow your component using APIClass to run successfully in either 32-bit or 64-bit windows process on your Windows 64-bit OS:

1. Place the following DLLs in a directory in your (System) PATH: jnidispatch-win32-x86.dll and jnidispatch-win32-amd64.dll

2. Do not call any JNA code prior to invoking APIClass.

3. If you need to call JNA code and cannot guarantee that APIClass has first been loaded, please add the following in every instance where this may occur:

   boolean is64Bit = !"i386".equals(System.getProperty( os.arch));
   System.setProperty("jna.library.file", is64Bit ? "
   jnidispatch-win32-amd64.dll" : "jnidispatch-win32-x86.dll");

4. If you are using other modules from different vendors who may also be using JNA, please ensure that these modules properly select the correct windows architecture jnidispatch DLL since other vendors may handle this in some other manner.

I think if you carefully consider the scenarios in which a user would need steps 2-4, they really have nothing to do with JNA and everything to do with the "solution" that you're providing. If someone is using JNA outside of your module, with or without some other 3rd-party module, they will have already encountered the issue you are trying to address, and will already have implemented a solution for it. How likely is that to be your solution? If they have a solution, then your code will already be set up to use JNA properly, without any extra configuration. If they have a solution and follow your instructions, they may very well break their existing setup.

Your main issue boils down to this:

• Can't depend on finding the right "jnidispatch.dll" in PATH

Possible options:

1. look up some other name in PATH
2. look up "jnidispatch.dll" in some other path
3. look up some other name in some other path

The solution JNA already provides (#2):

• jna.boot.library.path - provide an alternative to PATH for where to find "jnidispatch.dll"

Your suggestion (variant #1):

• hard-code some different names to try in PATH

My suggestion (combination of #1 and #3)

• provide name alternatives, relative (Initial documentation in markdown format. #1) or absolute (Correct the Callback documentation #3) based on pre-load configuration

I hope you can appreciate the deployment issues I'm trying to solve and understand why I would prefer not to push the burden of this to our customers. I would also not like to make initialization decisions in our code that might be incompatible with other uses of JNA in the same class loader/process.

I certainly appreciate that you have deployment issues, that you're under pressure to solve them, and I'm more than willing to try to find something that makes it possible for you to deploy in as clean a manner as possible. And if you want to pay me to come up with that solution, that's fine, too.

But I can't just take patches from everyone who insists that "this is just a small patch" or "it won't interfere with anything else". There are very few people I trust to submit patches like that, and they actually don't make patches like that without providing accompanying tests.

I think having the combination as you suggest for #1 and #3 is very good in general. However for many of the specific cases we have to deal with we cannot set Java properties for the host process at start-up (usually a Windows service like IIS or Sharepoint Server). Again, this may be outside the intended use of JNA since we are extending its use into the Microsoft .NET CLR via IKVM. So it could be that I should just close this pull request and maintain a fork that we distribute to accommodate these requirements of a very foreign platform.

I completely understand that you would not take updates to JNA from some random dude in github. I wasn't sure how to provide you with the tests I've been doing since it doesn't lend itself to incorporation into jUnit. I have a batch file that sets up the environment in very specific ways to test the following:

• Marker file is created when the temporary DLL is created and not deleted before exit - the current implementation fails to do this on very simple test cases and leaves DLLs in the %TEMP% directory as noted earlier
• Can use side-by-side versions of the jnidispatch library for 32-bit and 64-bit architectures

It uses a simple test program written in Java to invoke a windows API call (GetUserNameEx in Secur32.dll). The PATH and TEMP environment variables are tightly controlled by the test script. It is meant to run on a 64-bit Windows OS and runs test for both 32-bit and 64-bit JRE. Specifically the following tests are run (for each architecture):

1. The temporary DLL is created and marked for deletion and/or deleted
2. The temporary DLL cannot be created because %TEMP% dir not writable
3. Install jnidispatch.dll in the directory and no temporary file created
4. Install jnidispatch.dll with wrong architecture in the directory and temporary file created after load fails; temporary dll marked and/or deleted
5. Install jnidispatch.dll with wrong architecture in the directory and temporary file cannot be created to compensate
6. Install side-by-side libraries (named by convention described in Native.loadNativeLibrary) and a jnidispatch.dll with wrong architecture in the directory; no temporary dll created and correct architecture DLL is selected by JNA
7. Install side-by-side libraries and a jnidispatch.dll with wrong architecture in the directory and temporary file cannot be created correct arch DLL is selected by JNA

When I run the tests with JNA built from the current master, there are failures - specifically it fails to create marker files for tests 1, 4 and 6. Obviously, it fails test 7 as well because it is doesn't have the side-by-side library feature that I added in my fork. All these tests pass with the sxslib branch.

I'm not sure how to incorporate these types of tests into the JNA project. I would be happy to do so if you could provide me with a direction you see is appropriate. I will include the batch file and test program here.

------------------ testJnidispatch.java -------------------
import com.sun.jna.Native;
import com.sun.jna.win32.StdCallLibrary;
import com.sun.jna.win32.W32APIOptions;
import com.sun.jna.ptr.IntByReference;

/**
 * Simple class with main() that tests loading of jnidispath library
 * and call of a specific Win32 API call.  Used in conjunction with
 * sxslibtest.bat to run tests in various invocation environments.
 */
public class testJnidispatch {
    public static void main( String[] args ) {
    try {
        int itype = 8; // NameUserPrincipal
        char[] chars = new char[ 100 ];
        IntByReference iref = new IntByReference();
        iref.setValue( chars.length - 1 );
        if( !TestSecur32.INSTANCE.GetUserNameEx( itype, chars, iref ) ) {
        throw new Exception( "Cannot get user name" );
        }

        System.out.println( "User: " + Native.toString( chars ) );
        System.out.flush();
        System.exit( 0 );
    }
    catch( Throwable e ) {
        System.out.println( "Exception: " + e );
        System.out.flush();
        System.exit( 1 );
    }
    }

    static public interface TestSecur32 extends StdCallLibrary {
    public abstract boolean GetUserNameEx(int i, char ac[], IntByReference intbyreference);
    public static final TestSecur32 INSTANCE = (TestSecur32)Native.loadLibrary("Secur32", TestSecur32.class, W32APIOptions.UNICODE_OPTIONS );
    }
}
------------------ end testJnidispatch.java -------------------


------------------- sxslibtest.bat ----------------------------
@echo off
rem This batch file tests various environment scenarios for JNA use in Windows.
rem %ERRORLEVEL% is set to the number of failures encountered at exit
rem
rem The test enviroment should be a Windows machine with 64-bit architecture
rem but 32-bit Windows is supported as well.
rem
rem Configuration of the test environment:
rem
rem Create a folder for testing.
rem Put this script in the directory.
rem Create a folder called jre64 and copy a 64-bit JRE there
rem Create a folder called jre32 and copy a 32-bit JRE there
rem Create a folder called test and copy jna.jar, testJnidispatch.class and testJnidispatch$TestSecur32.class there
rem Create a folder called lib and place jnidispatch-win32-x86.dll and jnidispatch-win32-amd64.dll there
rem
rem Run the batch file.  All test should PASS.  The final output indicates if
rem any tests had unexpected results.
rem
rem For both 32-bit and 64-bit architectures, the following is tested:
rem
rem - The temporary DLL is created and marked for deletion and/or deleted
rem   testJnidispatch returns 0 (no error)
rem
rem - The temporary DLL cannot be created because %TEMP% dir not writable
rem   testJnidispatch returns 1 (error)
rem
rem - Install jnidispatch.dll in the directory and no temporary file created
rem   testJnidispatch returns 0 (no error)
rem
rem - Install jnidispatch.dll with wrong architecture in the directory
rem   and temporary file created after load fails; temporary dll marked
rem   and/or deleted
rem   testJnidispatch returns 0 (no error)
rem
rem - Install jnidispatch.dll with wrong architecture in the directory
rem   and temporary file cannot be created to compensate
rem   testJnidispatch returns 1 (error)
rem
rem - Install side-by-side libraries and a jnidispatch.dll with wrong
rem   architecture in the directory; no temporary dll created and correct arch
rem   DLL is selected by JNA
rem   testJnidispatch returns 0 (no error)
rem
rem - Install side-by-side libraries and a jnidispatch.dll with wrong
rem   architecture in the directory and temporary file cannot be created
rem   correct arch DLL is selected by JNA
rem   testJnidispatch returns 0 (no error)
rem
rem
set TESTD=%CD%\test
set OLDTMP=%TMP%
set OLDTEMP=%TEMP%
set TMP=%CD%\tmp
set TEMP=%CD%\tmp
set JNILIBS=%CD%\lib
set TESTCP=%TESTD%\jna.jar;%TESTD%
set /a FAILURES=0

call :runTest 0 32 temp
call :runTest 0 64 temp

call :runTest 1 32 notemp
call :runTest 1 64 notemp

call :runTest 0 32 std
call :runTest 0 64 std

call :runTest 0 32 stdbad
call :runTest 0 64 stdbad

call :runTest 1 32 stdbadnotemp
call :runTest 1 64 stdbadnotemp

call :runTest 0 32 arch
call :runTest 0 64 arch

call :runTest 0 32 archnotemp
call :runTest 0 64 archnotemp

if not "%FAILURES%"==0 echo Failures: %FAILURES%

set TMP=%OLDTMP%
set TEMP=%OLDTEMP%
rem pause
call:clear_temp
goto:eos

:runTest
rem Usage: call :runTest exitval arch libopt
rem        exitval: expected exit value from test program 0 (no error) or 1 (error) 
rem        arch:    32 or 64 - Use 32-bit or 64-bit JRE
rem        libopt:  temp = allow temporary DLL to be created (should be marked after exit or deleted)
rem                 notemp = don't allow temporary DLL to be created
rem                 std = copy standard-named library to %CD% with correct arch
rem                 stdbad = copy standard-named library to %CD% with incorrect arch
rem                 stdbadnotemp = copy standard-named library to %CD% with incorrect arch and do not allow temporary file to be created
rem                 arch = copy both arch-specifc-named libraries to %CD%
rem 
rem
rem Example #1: runTest 0 32 temp
rem Means the exit code should be 0, run with 32-bit JRE, allow temporary DLL to be created

rem Example #1: runTest 1 64 stdbad
rem Means the exit code should be 1, run with 64-bit JRE, copy 32-bit DLL to %CD%\jnidispatch.dll

set RESNAME=Exception
set EXITVAL=%1
if "%EXITVAL%"=="0" set RESNAME=User name printed
set JREARCH=%2
set JNIARCH=x86
set JNIARCHBAD=amd64
if "%JREARCH%"=="64" set JNIARCH=amd64
if "%JREARCH%"=="64" set JNIARCHBAD=x86
set LOPT=%3

rem if exist %TEMP% dir %TEMP% /b /s
call:clear_temp

rem Deal with the TMP directory depending on options
if "%LOPT%"=="notemp" call:no_temp_dir
if "%LOPT%"=="stdbadnotemp" call:no_temp_dir
if "%LOPT%"=="archnotemp" call:no_temp_dir
mkdir %TEMP% >nul 2>&1

echo Test %JREARCH%-bit with %LOPT% library option expect %RESNAME%

if "%LOPT%"=="std" goto:opt_std
if "%LOPT%"=="stdbad" goto:opt_stdbad
if "%LOPT%"=="stdbadnotemp" goto:opt_stdbad
if "%LOPT%"=="arch" goto:opt_arch
if "%LOPT%"=="archnotemp" goto:opt_arch
goto:doTest

:opt_std
copy %JNILIBS%\jnidispatch-win32-%JNIARCH%.DLL %CD%\jnidispatch.dll >nul 2>&1
goto:doTest

:opt_stdbad
copy %JNILIBS%\jnidispatch-win32-%JNIARCHBAD%.DLL %CD%\jnidispatch.dll >nul 2>&1
goto:doTest

:opt_arch
rem copies both arch-specific names and a default one of the WRONG arch
copy %JNILIBS%\jnidispatch-win32-*.DLL %CD%\ >nul 2>&1
copy %JNILIBS%\jnidispatch-win32-%JNIARCHBAD%.DLL %CD%\jnidispatch.dll >nul 2>&1
goto:doTest


:doTest
set JREBIN=%CD%\jre%JREARCH%\bin
set OLDPATH=%PATH%
set PATH=%JREBIN%;%WINDIR%\system32
%JREBIN%\java -cp "%TESTCP%" testJnidispatch
set EXITVALACTUAL=%ERRORLEVEL%
rem ERRORLEVEL of 216 means this is 32-bit OS and 64-bit JRE isn't valid
if %EXITVALACTUAL%==216 goto:cleanup

if "%LOPT%"=="temp" goto:check_temp
if "%LOPT%"=="stdbad" goto:check_temp
goto:test_exit

:check_temp
rem make sure that either no temp DLL exists or a DLL and a marker file exist
for %%i in (%TEMP%\jna\jna*.dll) do if not exist %%i.x call:record_fail "No marker file made for %%i" & goto :cleanup

:test_exit
if %EXITVALACTUAL%==%EXITVAL% goto:cleanup
call:record_fail "Expected exit status of %EXITVAL%, got %EXITVALACTUAL%"

:cleanup
echo.
set PATH=%OLDPATH%
goto :eof


:no_temp_dir
rem create a file in place of temp directory to prevent writing temporary files
echo x > %TEMP%
attrib +R %TEMP%
goto:eof

:clear_temp
if exist %TEMP% rmdir /s /q %TEMP% >nul 2>&1
if exist %TEMP% del /f /q %TEMP% >nul 2>&1
del /f /q jnidispatch*.dll >nul 2>&1
goto:eof


:record_fail
echo FAIL: %1
set /A FAILURES=FAILURES+1
goto:eof


:eos
exit /b %FAILURES%
----------- end sxslibtest.bat ------------

I've amended the "mark for deletion" code to actually get called. Let me know if that part works in your tests.

In your situation, is it indeterminate whether your code will run before the JNA static initialization code? When compiling, does iKVM do a dependency analysis to determine the order to load things? It's my understanding that it doesn't really perform class loading in the typical Java fashion, but I'm not familiar with the details.

If you can require that your code is loaded first, then System.setProperty("jna.library.name") would do the trick (you don't have to do that at VM launch, only before the Native class is loaded).

I got a copy of the latest changes and ran my tests. The "mark for deletion" now works properly. Thanks for fixing that. And also for adding the feature to disable unpacking.

IKVM class loading is indeed very tricky because of the way .NET Framework assemblies work (a class loader for each + a system class loader). This imposes constraints on packaging that seem unnatural in the Java world. Details can be found here: IKVM Class Loading Architecture.

I believe that IKVM does correctly initialize class statics based on dependency. And I appreciate that using this knowledge and a new property to initialize JNA could work. However, I believe this would make the modules we supply more fragile over time. The situation where this is most risky is in a web service. The customer is actually writing the Web Service in regular .NET (C#) and uses our IKVM'd versions of custom APIs. There are several independent "entry" classes so to speak packaged in assemblies/jars that correspond to different logical areas of access to the various server functional groups. So, right at the outset, that's some duplicated "init" code for JNA in each of the different packages. Okay, that's not too bad. However, as time goes by we will definitely be using more JNA code to access features in Windows. Each time we will have to review the code path to ensure that the JNA init code still will be called before any possible first access to JNA. As the surface area of our APIs increase over time so does the susceptibility to jndispatch init failures because of platform issues. The technique I've implemented in the sxslib branch has no such issues since it is handled in a standard manner in the core of JNA. Basically, I'm trying to save future maintenance problems/risks for our modules.

On Aug 8, 2011, at 5:44 PM, tussis wrote:

I got a copy of the latest changes and ran my tests. The "mark for deletion" now works properly. Thanks for fixing that. And also for adding the feature to disable unpacking.

Good to hear that it works this time :)

IKVM class loading is indeed very tricky because of the way .NET Framework assemblies work (a class loader for each + a system class loader). This imposes constraints on packaging that seem unnatural in the Java world. Details can be found here: IKVM Class Loading Architecture.

I believe that IKVM does correctly initialize class statics based on dependency. And I appreciate that using this knowledge and a new property to initialize JNA could work. However, I believe this would make the modules we supply more fragile over time. The situation where this is most risky is in a web service. The customer is actually writing the Web Service in regular .NET (C#) and uses our IKVM'd versions of custom APIs. There are several independent "entry" classes so to speak packaged in assemblies/jars that correspond to different logical areas of access to the various server functional groups. So, right at the outset, that's some duplicated "init" code for JNA in each of the different packages. Okay, that's not too bad. However, as time goes by we will definitely be using more JNA code to access features in Windows. Each time we will have to review the code path to ensure that the JNA init code still will be called before any possible first access to JNA. As
the surface area of our APIs increase over time so does the susceptibility to jndispatch init failures because of platform issues. The technique I've implemented in the sxslib branch has no such issues since it is handled in a standard manner in the core of JNA. Basically, I'm trying to save future maintenance problems/risks for our modules.

I don't quite follow how your modules would become more fragile over time. I think you are suggesting that an increased number of modules means increased administrative overhead of ensuring the modules adhere to certain startup requirements.

Presumably all your modules have (or could have) some sort of initialization already, in a static initializer or otherwise; if this includes nothing more than access to a central JNA initialization class, then I don't see an issue, other than requiring that your modules be written properly. Since they have to do something to access a JNA library mapping anyway that's not exactly a burdensome requirement. The following is only one method of doing so, I'm sure you can think of others. Even if you "compile" these separately, they're still going to be performing the same initialization (whether the common code is duplicated or not may be important to you, but is really outside the scope of JNA).

interface JNAUsage {
Object INIT = new Object() { { /* perform JNA properties init here */ }};
}

class MyModule1 implements JNAUsage {
}

class MyModule2 implements JNAUsage {
}

So the only review you have to do here is to ensure each new module has the proper "implements" clause. That's the sort of thing you could easily automate in a unit test or integration tests.

I understand your position on this feature and will close this pull request. Since this is mainly a problem with usage in a non-Java environment, we'll just deal with this particular problem and other troubles that .NET brings. Again thank you for fixing the temporary DLL mark/delete problem.