feat: Update the sentry release registry with AWS Lambda layer versions by iker-barriocanal · Pull Request #172 · getsentry/craft

iker-barriocanal · 2021-02-01T17:20:57Z

Summary

The latest versions of the AWS Lambda layers will be needed in our docs (manual setup) as well as in Sentry (automated setup). This PR updates the release registry with the information of the latest updated layers, by generating files that describe them during the release process.

Registry updates

The aws-lambda-layer target will have a change in .craft.yml, where compatibleRuntimes will be a list of objects containing a name and a list of versions (e.g. name: node and versions: ['nodejs10.x', 'nodejs12.x']). The new files will be placed in the directory aws-lambda-layers/{runtime name}/, and will be named {version to release}.json.

Each version file contains both dynamic and static attributes. Dynamic attributes are extracted when layers are created:

canonical. The canonical name of the layer, following the pattern aws-layer:{runtime name}.
sdk_version. The version to be released.
account_number. The account number of the account creating the layers. This can be found in the ARN of the created layer (arn:aws:lambda:{region}:{account number}:layer:{layer name}:{layer version}).
layer_name. The name of the layer. This value is taken from the .craft.yml project config file.
regions. A list of mappings of regions to versions, since different regions may have different versions.

Static attributes are common attributes to all the versions and won't usually change. They are located in the base.json file, placed in each runtime directory. All the attributes in this file will be added to the new version files. name, repo_url and main_docs_url are the only static attributes so far.

It's important that the directories already exist in the registry, since no new directories are created (already existing for node getsentry/sentry-release-registry#35). If the directory structure doesn't exist, a warning message will appear; and ARNs will always be logged. Besides, symlinks targeting to the major, minor and latest version will be created. latest.json will always link to the latest created version.

PR changes

The main changes in this PR are the new src/utils/awsLambdaLayerManager, a file containing the interaction of Craft and AWS. All the functionality of this file was previously on src/targets/awsLambdaLayer, but a couple of util functions. This abstraction has been made to make the code easier to deal with.

The src/targets/awsLambdaLayer, where the core functionality of the aws-lambda-layer target is, has now two extra features. On the one hand, the relevant information is collected (static and dynamic attributes), written to a file, and symlinks updated. On the other hand, these new files are created in a cloned repo, and then committed and pushed to the remote.

Lastly, the README has been updated, and tests and mocks have been added too.

The AwsLambdaLayer target interaction with the AWS SDK is abstracted, AwsLambdaLayerManager deals with it now. Still work in progress: functionality, docs and tests missing.

…ng messages

tonyo

Left a few minor comments.

General thoughts:

The registry update now runs unconditionally, wdyt about making it configurable on a target level? Or, do we e.g. plan to release beta layer versions that don't need to be pushed to the registry?
I'm a bit salty that we now have to blur borders of target responsibilities (basically, running the registry target functions from the aws-lambda target). Not your fault @iker-barriocanal, but more like a reminder that we might want to reconsider the overall design of how targets work together and whether we need, for example, some form of state (local or remote) when publishing.

Also, deleted mocked config file for the GithubGlobalConfig and removed mocked call from the test.

… registry targets

…lease The dry-run mode is on, the AWS target doesn't publish new layers nor push anything to the registry. When dry-run mode is off, the AWS target publishes new layers, but information of these layers is not published to the registry unless otherwise specified in the config file with `linkPrereleases`.

…e README file

…c/utils/registry`

tonyo

🏄

tonyo · 2021-02-02T15:46:03Z

Oh, and please update the changelog

…into awslambdalayer-registry

tonyo · 2021-02-02T16:19:57Z

@@ -1,5 +1,9 @@
 # Changelog

+## 0.16.2


Note: if you don't plan to release a new version immediately after the PR, let's not fill the version right away, but write "Unreleased" in the header. Who knows, maybe the next release will be a major bump

BYK · 2021-02-02T20:08:53Z

+            `Did not publish layers for ${runtime.name}. ` +
+              `Something went wrong with AWS: ${error.message}`
+          );
+          return;


This seems redundant?

BYK · 2021-02-02T20:37:58Z

+        const regionsVersions = publishedLayers.map(layer => {
+          return {
+            region: layer.region,
+            version: layer.version.toString(),
+          };
+        });


const regionsVersions = publishedLayers.map(({region, version}) => ({region, version: version.toString()}));

BYK · 2021-02-02T20:42:58Z

+        if (!fs.existsSync(baseFilepath)) {
+          logger.warn(`The ${runtime.name} base file is missing.`);
+          fs.writeFileSync(newVersionFilepath, JSON.stringify(runtimeData));
+        } else {
+          const baseData = JSON.parse(fs.readFileSync(baseFilepath).toString());
+          fs.writeFileSync(
+            newVersionFilepath,
+            JSON.stringify({ ...baseData, ...runtimeData })
+          );
+        }


let baseData; if (fs.existsSync(baseFilePath)) { baseData = JSON.parse(fs.readFileSync(baseFilepath, {encoding: 'utf-8'})); } else { baseData = {}; logger.warn(`The ${runtime.name} base file is missing.`); } fs.writeFileSync( newVersionFilepath, JSON.stringify({ ...baseData, ...runtimeData }), );

BYK · 2021-02-02T20:43:49Z

+  version: string,
+  versionFilepath: string
+): void {
+  const latestVersionPath = path.posix.join(directory, 'latest.json');


'latest.json' should probably be a constant:

const LATEST_VERSION_FILE_NAME='latest.json';

BYK · 2021-02-02T20:50:16Z

+    const previousVersion = fs
+      .readlinkSync(latestVersionPath)
+      .split('.json')[0];


const [previousVersion] = fs.readlinkSync(latestVersionPath).split('.json', 1);

May consider using path.parse() or similar path methods too. I checked but didn't find anything that gives you the full path sans the extension but not sure if that's your real intention here. Looks like path.basename(latestVersionPath, '.json') would fit here.

BYK · 2021-02-02T20:54:51Z

+   * @returns Information about the published layer: region, arn and version.
+   */
+  public async publishLayerToRegion(region: string): Promise<PublishedLayer> {
+    const lambda = new Lambda({ region: region });


const lambda = new Lambda({ region });

BYK · 2021-02-02T20:55:50Z

+  DescribeRegionsCommandOutput
+> {
+  logger.debug('Fetching AWS regions...');
+  const ec2 = new EC2({ region: 'us-east-2' });


Shouldn't this default region be carried out to a global constant?

BYK · 2021-02-02T20:58:19Z

+  const regionNames: string[] = [];
+  awsRegions.Regions?.map(currentRegion => {
+    if (currentRegion.RegionName !== undefined) {
+      regionNames.push(currentRegion.RegionName);
+    }
+  });
+  return regionNames;


return awsRegions.Regions?.filter(currentRegion => currentRegion.RegionName !== undefined) || [];

BYK · 2021-02-02T20:59:10Z

+ * @param arn The ARN of the account.
+ */
+export function getAccountFromArn(arn: string): string {
+  return arn.split(ARN_SEPARATOR)[ARN_ACCOUNT_INDEX];


This kind of screams for a more readable RegExp to me.

BYK · 2021-02-02T20:59:32Z

+ * Returns a GithubRemote object to the sentry release registry.
+ */
+export function getRegistryGithubRemote(): GithubRemote {
+  return new GithubRemote('getsentry', 'sentry-release-registry');


Shouldn't these be constants?

## Summary Resolves **10 open Dependabot alerts** across 5 CVEs and dismisses 2 low-severity esbuild alerts as tolerable risk. ### Alerts Fixed | Alert(s) | Package | Severity | CVE | Fix | |----------|---------|----------|-----|-----| | #180, #181 | `tar` | Medium | GHSA-vmf3-w455-68vh | Bump direct pin `7.5.11` -> `7.5.16` | | #178, #179 | `form-data` | High | GHSA-hmw2-7cc7-3qxx | `pnpm.overrides` for `4.0.6` (v4) and `2.5.6` (v2) | | #172, #176 | `vite` (root) | High + Medium | GHSA-fx2h-pf6j-xcff, GHSA-v6wh-96g9-6wx3 | Add as direct devDep `^7.3.5` + override | | #173, #177 | `vite` (docs) | High + Medium | same | `pnpm.overrides` in docs | | #174, #175 | `astro` | High + Medium | GHSA-2pvr-wf23-7pc7, GHSA-jrpj-wcv7-9fh9 | `pnpm update` -> `6.4.8` | ### Alerts Dismissed (tolerable_risk) | Alert(s) | Package | Severity | Reason | |----------|---------|----------|--------| | #167, #168 | `esbuild` | Low | GHSA-g7r4-m6w7-qqqr - Windows dev-server only vulnerability. Craft is a CLI tool, docs is a static site. Transitive dep of `[email protected]` which pins `esbuild@^0.27.0` - cannot fix without vite 8 major bump. | ## Changes ### Root project (`package.json`) - `tar`: `7.5.11` -> `7.5.16` (direct pin bump) - `vite`: added as devDependency at `^7.3.5` (was only transitive via vitest) - `pnpm.overrides`: added `form-data@>=4: ^4.0.6`, `form-data@<3: ^2.5.6`, `vite: ^7.3.5` ### Docs project (`docs/package.json`) - `astro`: resolved `5.16.11` -> `6.4.8` (specifier `^6.1.10` allowed it) - `@astrojs/starlight`: resolved `0.37.3` -> `0.38.3` - `pnpm.overrides`: added `vite: ^7.3.5` ## Verification - `pnpm build` - passed - `pnpm test` - 1025 passed, 1 skipped - `pnpm lint` - 0 errors - `pnpm docs:build` - 27 pages built successfully

iker-barriocanal added 15 commits January 27, 2021 13:44

ref: Abstract AWS Lambda interaction from the AWS Lambda layer registry

1a7a573

The AwsLambdaLayer target interaction with the AWS SDK is abstracted, AwsLambdaLayerManager deals with it now. Still work in progress: functionality, docs and tests missing.

ref: Add missing docstrings to AwsLambdaLayerManager

97ec88a

ref: Make verbose flag instance variable in the layer manager

d76e836

feat: Write new created layer data files in the AWS Lambda layer target

2d62670

ref: Rename runtime versions to versions

986787e

ref: Minor changes involving renaming variables, docstrings and loggi…

4ab899f

…ng messages

ref: Parallelize AWS Lambda layer runtime publishing with promises

2a7cf32

feat: Add support for remote repo

fbe0d14

ref: Rename publishing action to publishToAllRegions for consistency

6e6218e

feat: Handle use case when the AWS SDK throws an error

64a3693

ref: Remove temporary directory for dealing with git

12d5d63

ref: Remove AWS SDK Lambda wrapper functions

bd2cc33

test: Add tests for AWS Lambda Layer Manager

df844c2

test: Update awsLambdaTest and mock config

69d1503

docs: Update README with changes applied to the AWS Lambda Layer target

766d67c

iker-barriocanal requested a review from tonyo February 1, 2021 17:20

tonyo suggested changes Feb 2, 2021

View reviewed changes

iker-barriocanal added 5 commits February 2, 2021 12:03

ref: Move constants to the top of the file

412c566

ref: Remove GithubGlobalConfig from the AwsLambdaLayer target

829fb93

Also, deleted mocked config file for the GithubGlobalConfig and removed mocked call from the test.

ref: Hardcode test variable to make it more robust

5e58f61

ref: Make anon functions non-anon

fd42086

ref: Extract getting the github remote of the registry in the AWS and…

684fc54

… registry targets

tonyo reviewed Feb 2, 2021

View reviewed changes

Comment thread src/utils/githubApi.ts Outdated

iker-barriocanal added 4 commits February 2, 2021 15:29

feat: Do not publish new AWS Lambda layers when dry run is enabled

317bb77

docs: Add linkPrereleases config option for AWS Lambda target in th…

80290be

…e README file

ref: Move getRegistryGithubRemote from src/utils/githubApi to `sr…

5f0a19e

…c/utils/registry`

iker-barriocanal requested a review from tonyo February 2, 2021 15:11

tonyo approved these changes Feb 2, 2021

View reviewed changes

Comment thread src/targets/registry.ts Outdated

Comment thread src/targets/awsLambdaLayer.ts Outdated

iker-barriocanal added 5 commits February 2, 2021 16:57

ref: Remove duplicated import

40097a9

ref: Typo

9242bb6

Merge branch 'master' into awslambdalayer-registry

ce5320d

meta: Update changelog

ec99e53

Merge branch 'awslambdalayer-registry' of github.com:getsentry/craft …

621156b

…into awslambdalayer-registry

iker-barriocanal merged commit fed4e9a into master Feb 2, 2021

iker-barriocanal deleted the awslambdalayer-registry branch February 2, 2021 16:17

tonyo reviewed Feb 2, 2021

View reviewed changes

iker-barriocanal restored the awslambdalayer-registry branch February 2, 2021 16:21

BYK reviewed Feb 2, 2021

View reviewed changes

BYK mentioned this pull request Jun 23, 2026

fix: resolve 10 Dependabot security alerts #836

Merged

Uh oh!

Uh oh!

Conversation

iker-barriocanal commented Feb 1, 2021

Summary

Registry updates

PR changes

Uh oh!

tonyo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tonyo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tonyo commented Feb 2, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants