Specification

The build:windows and subsequently integration:windows jobs involve using chocolatey to download packages to be usable on Windows. In particular nodejs and python.

The current configuration in .gitlab-ci.yml will redownload packages and reinstall each time. This is slow, error prone and results in our CI/CD getting rate limited from chocolatey (#394 (comment)).

On top of that the chocolatey community software packages on https://community.chocolatey.org/packages, which is the default source of packages is not entirely secure, or at least not recommended for organisational usage https://docs.chocolatey.org/en-us/community-repository/community-packages-disclaimer.

Furthermore when downloading and installing chocolatey packages, we are simply relying on the checksum specified by the package maintainer. This is the case with nixpkgs, but with nixpkgs we can pin our package set from nixpkgs-overlay, with chocolatey, this is not the case. We can get halfway there by doing TOFU, and acquiring the checksum for a trusted version of the package, and then enforce that installing the package again must also have the same checksum.

This is particularly important because some packages download things over the internet during installation (this is due to chocolatey lacking distribution rights, thus the "package" is just an instruction set on how to download & install, but not the actual software runtime being installed). Having a checksum specified at our CI/CD level just ensures some level of end-to-end trust of immutability.

Additional context

• https://docs.chocolatey.org/en-us/faqs#what-is-a-trusted-package - verify that our packages are "trusted" first
• https://docs.chocolatey.org/en-us/information/security#security-for-the-community-package-repository - this details the process of chocolatey packaging

  Packages are pushed to the site over HTTPS. The site grabs a SHA512 checksum of the package, then forwards it on to where packages are stored securely. You can see this package checksum in 0.9.10+ if you call choco info .

• https://docs.chocolatey.org/en-us/features/host-packages#local-folder-unc-share-cifs
• Based on https://docs.chocolatey.org/en-us/choco/commands/install, use the --checksum and below parameters and find out which we can use to ensure integrity of our installed packages on chocolatey.

       --checksum, --downloadchecksum, --download-checksum=VALUE
       Download Checksum - a user provided checksum for downloaded resources 
         for the package. Overrides the package checksum (if it has one).  
         Defaults to empty. Available in 0.10.0+.
  
       --checksum64, --checksumx64, --downloadchecksumx64, --download-checksum-x64=VALUE
       Download Checksum 64bit - a user provided checksum for 64bit downloaded 
         resources for the package. Overrides the package 64-bit checksum (if it 
         has one). Defaults to same as Download Checksum. Available in 0.10.0+.
  
       --checksumtype, --checksum-type, --downloadchecksumtype, --download-checksum-type=VALUE
       Download Checksum Type - a user provided checksum type. Overrides the 
         package checksum type (if it has one). Used in conjunction with Download 
         Checksum. Available values are 'md5', 'sha1', 'sha256' or 'sha512'. 
         Defaults to 'md5'. Available in 0.10.0+.
  
       --checksumtype64, --checksumtypex64, --checksum-type-x64, --downloadchecksumtypex64, --download-checksum-type-x64=VALUE
       Download Checksum Type 64bit - a user provided checksum for 64bit 
         downloaded resources for the package. Overrides the package 64-bit 
         checksum (if it has one). Used in conjunction with Download Checksum 
         64bit. Available values are 'md5', 'sha1', 'sha256' or 'sha512'. 
         Defaults to same as Download Checksum Type. Available in 0.10.0+.
  

• There is also VERIFICATION.txt files available in the installation packages. Note that we are often using meta packages that then point to a particular package under dependencies. These dependencies then have Files, one of them being VERIFICATION.txt. This can be used to acquire checksums as well, however you have to check whether this is actually different or related to the package checksum. If it is possible to rely on the package checksum which itself contains checksums on its contents, then prefer to use the package checksum.
  
• https://stackoverflow.com/questions/34268673/how-do-i-configure-a-local-chocolatey-repository
• https://4sysops.com/archives/install-internalized-chocolatey-packages-from-your-offline-repository/

Tasks

1. Iterate on this using our matrix-win-1 computer first before applying to gitlab cicd
2. Setup to host chocolatey packages privately, probably over a directory that is cached by gitlab - https://docs.chocolatey.org/en-us/features/host-packages#local-folder-unc-share-cifs
3. Follow this guide https://docs.chocolatey.org/en-us/guides/create/recompile-packages to internalise community packages that we are using nodejs and python into that directory
   • https://community.chocolatey.org/packages/nodejs
   • https://community.chocolatey.org/packages/python
   • Make sure to get the fixed version we are using in .gitlab-ci.yml
4. Acquire the checksum of the trusted packages using choco info <packageName> and specify these checksums when performing a installation
5. Ensure that choco install is installing from the our local directory source by using --source option, and not from the upstream community source, this should remove any 429 too many request rate limiting