Wow !

What do you mean by the scary:

completely breaks backwards compatibility

I mean is this related to Bash somehow ?

No, that's just the whole LOOT going from a C API to a C++ API thing.

Wow - it is faster. One needs to download the x86 Visual C++ Redistributable for Visual Studio 2015 otherwise (same with x64 one):

Traceback (most recent call last):
  File "Wrye Bash Launcher.pyw", line 88, in <module>
    bash.main()
  File "bash\bash.py", line 361, in main
    import bosh
  File "Wrye Bash Launcher.pyw", line 76, in load_module
    exec compile(code, initfile, 'exec') in mod.__dict__
  File "bash\bosh\__init__.py", line 51, in <module>
    from .mods_metadata import ConfigHelpers
  File "Wrye Bash Launcher.pyw", line 63, in load_module
    mod = imp.load_source(fullname,filename+ext,fp)
  File "bash\bosh\mods_metadata.py", line 38, in <module>
    import loot_api
ImportError: caused by ImportError(u"caused by ImportError('DLL load failed: The specified module could not be found.',)",)

We should add import guards there as Bash should not fail to start in case we fail to import - and a nice warning on start like loot unavailable, download blah blah from https://www.microsoft.com/en-us/download/confirmation.aspx?id=48145

Timings

mods_metadata.py  220 __init__: Using LOOT API version: 0.10.0
mods_metadata.py  271 refreshBashTags: An error occurred while parsing file C:\Users\MrD\AppData\Local\LOOT\Oblivion\masterlist.yaml:
Traceback (most recent call last):
  File "bash\bosh\mods_metadata.py", line 267, in refreshBashTags
    lootDb.load_lists(path.s)
RuntimeError: yaml-cpp: error at line 168, column 9: bad conversion: 'utility' key missing from 'cleaning data' object

         1526093 function calls (1525878 primitive calls) in 5.317 seconds

bashbugdump-loot10.txt

pInit2 at ecefb5b

mods_metadata.py  220 __init__: Using LOOT API version: 0.6.0
         1529259 function calls (1529044 primitive calls) in 9.130 seconds

bashbugdump-loot6.txt

That's almost double, OMDG !

For that timing comparison, the v0.10 test is encountering a syntax error because you're trying to parse a v0.8 masterlist, which will be affecting the results. Try running the v0.10 test again, using the v0.10 branch masterlist.

Also, on the subject of import guards, I dislike committing binary files, and if we're making it so that Bash can run without the LOOT API present (which is sensible), how about we treat it like any of the other external dependencies (wxPython, comtypes, etc) and have the installer download and install it, and provide instructions for manually installing it?

I dislike committing binary files

Tell me about it... :D

if we're making it so that Bash can run without the LOOT API present

Well, currently, as seen in the code, if loot fails to initialize we go on normally - it would not fail on import loot however - so adding another except ImportError would not change current behavior (I can do the nitty gritty warnings and stuff no worries)

how about we treat it like any of the other external dependencies (wxPython, comtypes, etc) and have the installer download and install it, and provide instructions for manually installing it?

Hmmmmm - just as I got used to the idea to have loot binaries committed with bash...

I can't readily tell really - on the one hand that's more work for the respected python version User - and more chances for our hacked up installer to fail. Tags functionality was always in there and is essential - but yes, as of old the user had to install BOSS manually. Sounds a bit like a step back though.

On the other hand yes, these binary files are a very bad idea - and if we have to keep up with LOOT fast pace this means a binary commit every so often (incidentally this can be somehow avoided if the masterlist format stays the same)

Started a discussion in the thread : http://forums.bethsoft.com/topic/1606578-wrye-bash-thread-111/#entry25234884 - let's see what people say

Already a point brought up there - any chance we make this MSVC 2012 ? Breaks Wine for one

I can't readily tell really - on the one hand that's more work for the respected python version User - and more chances for our hacked up installer to fail. Tags functionality was always in there and is essential - but yes, as of old the user had to install BOSS manually. Sounds a bit like a step back though.

As an alternative to providing manual instructions then, perhaps a script to do the same that the user can run? It could be run by the installer as a post-install step (or on Bash's first run), to avoid complicating the installer scripting further.

(As an aside, I switched to Inno Setup for LOOT's installer a while ago, and have found it much simpler - not saying Bash should convert too, but it would be my choice for any new installer I had to write.)

Already a point brought up there - any chance we make this MSVC 2012 ? Breaks Wine for one

No, LOOT uses C++11 and C++14 features that any earlier version of MSVC just doesn't support. Microsoft showed their usual pacing with regards to standards-compliance in making MSVC 2015 the first release to decently support the 2011 standard...

As for it not working under Wine, surely that's not a significant issue? Of all Bash's users, those running it under Wine must be a tiny, if vocal, minority, and I'm personally of the opinion that if you want to run an application on an OS that it doesn't target, you should be willing to work out any issues that arise from that yourself.

In this instance, given that Bash is written in Python (a language that enjoys strong support on Linux and is relatively accessible), uses wxPython, and that all its bundled binary dependencies are either available or buildable on Linux (7-zip, CBash, libbsa, LOOT API), I'd imagine that effort to get it to run on Linux would be best put into getting it to do so natively, not through Wine. (Admittedly I don't know what the comtypes and pywin32 dependencies are used for, and how difficult it would be to make that stuff cross-platform or optional.)

(or on Bash's first run)

Yep, that would be needed anyway for python users. A nice dialog, an ini option to suppress, various options - needs thinking. We could go that way for the beta and see how it's received - but let's see what the regular users say...

As an aside, I switched to Inno Setup for LOOT's installer a while ago, and have found it much simpler

I dont like the current installer, at all. However a much welcome installer redesign should go hand to hand with rethinking Bash install policies - the "dump Bash into game folder" thingy is a relic really from oblivion time. Bash should leverage the ini mechanism to create profiles - so one Bash install (not one per game), so current installer detecting games and all that - binned.

But I really have no time to dig into installer scripting. Bash development is like fighting the Hydra - squash a bug two others pop up. To be fair, the code is a couple orders of magnitude better but a lot remains to be done.

As for it not working under Wine, surely that's not a significant issue? Of all Bash's users, those running it under Wine must be a tiny, if vocal, minority

No, it is important to me. Not only, as I said before, because from this tiny and vocal minority help may arise, but as an important (very important) code quality metric. From what you've seen in Bash I hope you trust me by now as programmer. There are 10 types of programmers - hackers and engineers - and much as I admire hackers (who get the job done), I'm the definition of an engineer. I don't care about features - I care about my data structures. I believe I proved my point with Bash - if you care to run 305 and then 307 - you'd be surprised. But I achieved that not by attacking the bugs (there was a lot of that too ofc) but by refactoring.

Wine gives me precious insights into some of the more obscure stuff (and horrible hacks like the env module, which was not a module but windows ravioli code all over, I'm rather proud of being able to encapsulate that mess there - 646df2c).

I'd imagine that effort to get it to run on Linux would be best put into getting it to do so natively, not through Wine

Well that's one of my goals - #243 (and links there will give an idea that's an effort I started long ago). But to get there (and it's a long way) I need to be able to run on wine as a warm up at least.

That being said, let's see how the discussion goes. I would sacrifice loot support in Wine (crossing my fingers that msvc 2015 gets Wine support soon) if it turns out to be necessary.

No, it is important to me.

Fair enough then.

Now just waiting on feedback about bundling vs. downloading the API before I update the installer and readme.

Well - I was just about to say alea jacta est and let's go with downloading. Anyway there is a 307 beta on the making so we will get all feedback on the world and we can reverse this. The code for the installer will be needed one way or the other and this is an excellent opportunity to write the code for an "update loot" menu which will save us from binary commits even if we go back to bundling - the user can update and we can bundle new loot every new release or something at our convenience.

We need a specification for the new behavior however - I leave that to you and I can code the python dialogs and such once the specification is fleshed so you don't need to dig deeply in Bash code

I don't think it's worth having an update UI, I doubt there would be many non-breaking LOOT releases between Bash releases (there have been only two since v306's release, and one breaking release).

Incidentally, what's the upgrade procedure for Bash, are users supposed to replace their Mopy folder (and does the installer do this?), so that the API files would get replaced too, or is there some sort of differential upgrade I'd have to work with?

I don't think it's worth having an update UI, I doubt there would be many non-breaking LOOT releases between Bash releases (there have been only two since v306's release, and one breaking release).

Well as you think but if we have all the pieces throwing in a menu item would be trivial - and Bash as I am the only dev working on it is super slow on its releases. You could even provide a url for latest snapshots so daring users could update to that. Just thoughts

Incidentally, what's the upgrade procedure for Bash, are users supposed to replace their Mopy folder (and does the installer do this?), so that the API files would get replaced too, or is there some sort of differential upgrade I'd have to work with?

I don't remember but no clearly no differential updates - a long time since I looked at the installer code - this may be of help:

$ gitk --all scripts/ &

There is however code for removing versions from older Bash installs that I made required recently. Since there are new packages etc it is essential that those older files are removed.

I would like to bundle this in my wip branch for users to test - I would need also fallout4 files in

fallout4 files in

What fallout 4 files do you mean, the taglists?

Yep and probably edit the mktaglist script too to include fallout 4 lists generation

I've just rebased on top of the latest wrye-bash dev branch and have added the install script and removed the LOOT API binary. The script currently doesn't handle the MSVC redist, I'll add that later.

First and foremost I am not using or testing the EXE installer. Many users do not either and download the zip file for the Standalone EXE and extract the files manually before copying them to the Mopy folder. I did notice that there is a commit named "Add LOOT API install script". Does that only run when you use the EXE installer?

I was testing regular Skyrim with https://github.com/WrinklyNinja/wrye-bash/commit/c392e0282e40e61b0fa895f67c3b9d9a9622c3ea but I don't understand something. Once the archives are built I have an EXE installer, a Python version in a 7zip file, and and Standalone EXE in a 7zip file.

If I run the Standalone EXE this is my BashBugDump.log.

Wrye Bash starting
Using Wrye Bash Version 307
Python version: 2.7.8
wxPython version: 2.8.12.1 (msw-unicode)
input encoding: None; output encoding: None; locale: ('en_US', 'cp1252')
filesystem encoding: mbcs 
Searching for game to manage:
bush.pyo   71 _supportedGames: Detected the following supported games via Windows Registry:
bush.pyo   73 _supportedGames:  oblivion: G:\Games\steamapps\common\Oblivion
bush.pyo   73 _supportedGames:  skyrim: F:\Games\steamapps\common\Skyrim
bush.pyo   73 _supportedGames:  fallout4: C:\Games\steamapps\common\Fallout 4
bush.pyo  131 _detectGames: Detecting games via the -o argument, bash.ini and relative path:
bush.pyo  137 _detectGames: Set game mode to skyrim found in parent directory of Mopy:  F:\Games\SteamApps\common\Skyrim
bush.pyo  172 setGame: No preferred game specified.
bush.pyo  152 __setGame:  Using skyrim game: F:\Games\SteamApps\common\Skyrim
mods_metadata.pyo   41 <module>: Failed to import the loot_api module.
testing UAC
__init__.pyo 7582 initDirs: Using libbsa API version: 2.0.0

The log seems to indicate that it can't access the loot api.

The file below is what package_for_release.py generated for the Standalone. I made it a zip file since Git doesn't like 7zip files.

Wrye Bash 307 - Standalone Executable.zip

The loot api binaries are not in the compiled folder. If I copy loot_api.dll and loot_api.pyd from an old commit not WrinklyNinja's branch, then I get the error.

When someone doesn't use the EXE installer and instead extracts the files from the zip, how do they get the loot binaries? Without the loot api binaries how will Wrye access the loot api? Why would manually copying them cause that error?

I did notice that there is a commit named "Add LOOT API install script". Does that only run when you use the EXE installer?

It isn't automatically run at all right now, but I will update the installer to run it.

When someone doesn't use the EXE installer and instead extracts the files from the zip, how do they get the loot binaries?

By running the install script.

Without the loot api binaries how will Wrye access the loot api?

It can't.

Why would manually copying them cause that error?

I'm guessing there's some conflict between the runtimes used by the standalone executable and the LOOT API, but I haven't looked at how the executable is generated so don't really know anything about it.

The install script should now handle downloading and installing the MSVC redist, though I'm unable to properly test what happens if the redistributable isn't installed.

@Sharlikran How do I build the standalone executable, aside from having py2exe installed? What do the ResHacker.exe and upx.exe executables do (and why are they in source control 😢)?

I do not know what ResHacker.exe is or what it does other then what's on the wiki page. I don't really know what upx.exe is either but the website for it says it's some kind of packer for EXE files. There seems to be GitHub source if you want to view it.

What I did a long time ago, is I looked at the top portion of the wiki page titled Making A Release to make sure I had all the required modules like PyGithub and GitPython. I don't remember how I got those installed. To be honest, it was a pain.

Then when I go to build everything git isn't in my system path or the scripts don't read things correctly so I add the location in a .bat file.

@echo off
title Wrye Flash Release Builder
echo.
echo.

cd wrye-bash
cd scripts
py package_for_release.py -g C:\Users\DanoPDX\AppData\Local\Programs\Git\bin

echo.
echo.

echo. All Done!
pause

My .bat file is outside the repo so just adjust the change directory lines as needed.

I just thought of something: if a user installs the standalone using the installer, they won't necessarily have Python installed, so won't be able to run the install script I've written.

I could also implement the script in the NSIS installer itself so that the Python script is only used by manual install users, but that's duplicating functionality. Does anyone have any better ideas?

It seems we need:

• a nsis script extension to download msvc/loot
• a python script for users to download msvc/loot
• instructions for standalone users to download msvc loot

Except if we move the script inside the python code - as in mods metadata - so we add a menu for downloading loot, that both standalone and python users can invoke. Still we need a startup dialog with a set of options - and maybe some ini options.

We still need instructions of course as the scripts may fail - plus a nice section explaining all this in the readme

What I did a long time ago, is I looked at the top portion of the wiki page titled Making A Release to make sure I had all the required modules like PyGithub and GitPython. I don't remember how I got those installed. To be honest, it was a pain.

Wow. Contrast LOOT's release process, which is still at least a third longer than it could be.

Except if we move the script inside the python code - as in mods metadata - so we add a menu for downloading loot, that both standalone and python users can invoke. Still we need a startup dialog with a set of options - and maybe some ini options.

Would you rather do that or have me extend the NSIS script? I'm not interested in mucking around with Bash's code (beyond the almost find/replace I did to update the API usage), and you'd do a better job of it anyway. I'll update the readme whatever you decide.

Would you rather do that or have me extend the NSIS script?

We need both - the majority of the users use the installer - so we need the installer updated to download loot api and msvc. If this fails and for standalone/python users we need to have an option to install the api and msvc from within bash. I still need a specification however - from frazing the messages to the options given.

I still need a specification however - from frazing the messages to the options given.

How am I in any position to decide what those things should be?

Trying using the scripts - two things for now:

• the install_loot_api should be plugged into package for release - before trying to pack the api it should check it's there and if not download it. The hardcoded version in this script will be the version current dev code is using
• the mktaglist script should be updated to ignore the registry and just compile the tag lists - however a test that the masterlists correspond to the packed API version would be ideal
• edit - instal_api script:

Extracting LOOT API Python wrapper to Mopy
Traceback (most recent call last):
 File "C:\Dropbox\eclipse_workspaces\python\wrye-bash\scripts\install_loot_api.py", line 86, in <module>
   installLootApi(lootApiWrapperVersion, lootApiWrapperRevision, 'Mopy')
 File "C:\Dropbox\eclipse_workspaces\python\wrye-bash\scripts\install_loot_api.py", line 71, in installLootApi
   subprocess.call([sevenZipPath, 'e', archivePath, '-y', '-o' + destinationPath, '*/loot_api.dll', '*/loot_api.pyd'])
 File "C:\_\Python27\lib\subprocess.py", line 523, in call
   return Popen(*popenargs, **kwargs).wait()
 File "C:\_\Python27\lib\subprocess.py", line 711, in __init__
   errread, errwrite)
 File "C:\_\Python27\lib\subprocess.py", line 959, in _execute_child
   startupinfo)
WindowsError: [Error 2] The system cannot find the file specified

fails to resolve the 7z path - since the script is in scripts/ it should probably try to run from there (not from Mopy)

Fair enough - you will just need to inform me which version I should finally package - and let me know of important updates.

Whatever version is hardcoded in the install script (0.10.2 release ATM) is the one you should use. 😉

Or you mean that masterlist updates are backported to older formats ?

Nobody's volunteered to do that so far, but they can be.

the mktaglist script should be updated to ignore the registry and just compile the tag lists - however a test that the masterlists correspond to the packed API version would be ideal

The API needs a game path. It could be mocked using a Data folder containing files with the same names as the games' master files, but there needs to be something - I can update the script to create the mocks in a temporary directory. As for testing the masterlists versions, they'll fail to load if they're not the right version, that's your test.

fails to resolve the 7z path - since the script is in scripts/ it should probably try to run from there (not from Mopy)

Are the scripts not supposed to run from the repository root? That's how I've been running mktaglists and prepare_for_release.

Right, I think I'm done.

Thanks - will test ASAP

Are the scripts not supposed to run from the repository root? That's how I've been running mktaglists and prepare_for_release.

I run it (via pycharm) thus:

C:\_\Python27\python.exe C:/Dropbox/eclipse_workspaces/python/wrye-bash/scripts/package_for_release.py -g c:\\_\\Git\\bin --non-repo COPY

How do you run them from repo root (actually that statement had puzzled me that's why I left be9e384 separate)

How do you run them from repo root

With Python and Git in my PATH:

cd ~/Documents/GitHub/wrye-bash
python scripts/install_loot_api.py

@WrinklyNinja : ready to merge ? Remains I think:

• mktaglist doc - I can do that if you want but easier you do it as you wrote the code
• latest taglists - except if that's sorted

Meanwhile, will pack a nightly and offer it to the thread regulars to test

I will then, when you give me green light, proceed to rebase merge

Wow - proved a hard nut

The taglists are already up to date, that was my last commit (only the Skyrim SE taglist needed updating). What do you want documented for mktaglist and where?

Thanks perfect (will squash that one with the loot 10 commit)

The mktaglist.py script has a doc at the top that's pretty oudated

The mktaglist.py script has a doc at the top that's pretty oudated

Updated.

As Alt3rn1ty mentioned in the forum thread, I missed a step when updating the mods_metadata.py file, I'll fix that in a bit.

If the loot api exists even if compatible the install_loot_api srcipt fails with:

Traceback (most recent call last):
  File "C:\Dropbox\eclipse_workspaces\python\wrye-bash\scripts\install_loot_api.py", line 78, in <module>
    installLootApi(lootApiWrapperVersion, lootApiWrapperRevision, 'Mopy')
  File "C:\Dropbox\eclipse_workspaces\python\wrye-bash\scripts\install_loot_api.py", line 63, in installLootApi
    os.remove(os.path.join(destinationPath, 'loot_api.dll'))
WindowsError: [Error 5] Access is denied: 'Mopy\\loot_api.dll'

What locks the loot_api.dll is the:

 loot_api.WrapperVersion.string() == version and
        loot_api.WrapperVersion.revision == revision

as far as I can see - note that string() needs the parens

Tried the obvious (del loot_api) but it didn't work and if this post is by Beazley:

http://swig.10945.n7.nabble.com/unload-a-extension-lib-pyd-in-python-tp10073p10074.html

then it's downright impossible

So you need to call the dll from a subprocess to do the version check - you need a separate python interpreter to let go of the dll. See also:

http://kun-hong.blogspot.com/2009/06/how-to-unload-python-extension-dll.html

for a ctypes hack