Skip to content

Amtool initial implementation#1

Merged
Kellel merged 24 commits intomasterfrom
amtool
Feb 25, 2017
Merged

Amtool initial implementation#1
Kellel merged 24 commits intomasterfrom
amtool

Conversation

@Kellel
Copy link
Owner

@Kellel Kellel commented Jan 11, 2017

This is the first version of an alertmanager tool.

Copy link

@mattbostock mattbostock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added some initial comments.

The build failed so I haven't yet been able to try it out - see my comment on cli/silence_test.go.

@@ -0,0 +1,9 @@
// Copyright © 2016 Kellen Fox <[email protected]>

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The copyright belongs to Cloudflare, assuming you worked on this on your work machine during work hours. I don't think there's a good reason to put the copyright notice in this file but I am not a lawyer.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll just delete. The cobra autogen tool puts a copyright notice in.

cli/root.go Outdated
var RootCmd = &cobra.Command{
Use: "amtool",
Short: "Alertmanager CLI",
Long: `View and modify the current alertmanager state.`,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alertmanager (capital A)

cli/root.go Outdated
Long: `View and modify the current alertmanager state.`,
// Uncomment the following line if your bare application
// has an action associated with it:
// Run: func(cmd *cobra.Command, args []string) { },

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these commands can be removed?

cli/root.go Outdated
func Execute() {
if err := RootCmd.Execute(); err != nil {
fmt.Println(err)
os.Exit(-1)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exit codes are uint8, unless you want to return a 255 exit code?

Suggest os.Exit(1).

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm. I just copy/paste'ed this from the cobra docs. Changed to 1

cli/root.go Outdated
}
}

func getAlertmanagerUrl() {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function isn't used. I've deleted it

cli/root.go Outdated
func getAlertmanagerUrl() {
am := viper.GetString("alertmanager")
if am == "" {
fmt.Println("Alertmanager url not specified in the config file or on the command line")

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

URL (all caps)

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as previous.

cli/silence.go Outdated
return output
}

// Thanks @vendemiat for help with this one

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd put that in the commit message, that way you still get to say thanks but it doesn't clutter the code.

&types.Matcher{Name: "bar", Value: "baz", IsRegex: false},
}

testGroups := MatcherGroup{

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Build fails with:

cli/silence_test.go:54: undefined: MatcherGroup

@Kellel
Copy link
Owner Author

Kellel commented Jan 23, 2017

Hopefully It builds for you now. Currently all the tests seem broken due to the mismatched references to github.com/Kellel/alertmanager vs github.com/prometheus/alertmanager

Not really sure how to deal with this

cli/root.go Outdated

RootCmd.PersistentFlags().StringVar(&cfgFile, "config", "", "config file (default is $HOME/.amtool.yaml)")
RootCmd.PersistentFlags().String("alertmanager", "", "Alertmanager to talk to")
viper.BindPFlag("alertmanager", RootCmd.PersistentFlags().Lookup("alertmanager"))

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

alertmanager.url for consistency with Prometheus' CLI flags.

cli/root.go Outdated
func init() {
cobra.OnInitialize(initConfig)

RootCmd.PersistentFlags().StringVar(&cfgFile, "config", "", "config file (default is $HOME/.amtool.yaml)")

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prometheus and Alertmanager default to a .yml extension.

@mattbostock
Copy link

It would be good to show a friendlier error if someone forgets to specify the Alertmanager URL:

matt➜github.com/prometheus/alertmanager(Kellel-amtool)» ./amtool config
Error: Get /api/v1/status: unsupported protocol scheme ""
Usage:
  amtool config [flags]

Global Flags:
      --alertmanager string   Alertmanager to talk to
      --config string         config file (default is $HOME/.amtool.yaml)
  -o, --output string         Output formatter (simple, extended, json) (default "simple")

Get /api/v1/status: unsupported protocol scheme ""

@mattbostock
Copy link

Ditto for a friendlier error when the URL is incorrect:

matt➜github.com/prometheus/alertmanager(Kellel-amtool)» ./amtool config --alertmanager https://alertmanager-staging.svc.cfdata.org/foo
Error: json: cannot unmarshal number into Go value of type cli.alertmanagerStatusResponse
Usage:
  amtool config [flags]

Global Flags:
      --alertmanager string   Alertmanager to talk to
      --config string         config file (default is $HOME/.amtool.yaml)
  -o, --output string         Output formatter (simple, extended, json) (default "simple")

json: cannot unmarshal number into Go value of type cli.alertmanagerStatusResponse
matt➜github.com/prometheus/alertmanager(Kellel-amtool)»

@mattbostock
Copy link

Rough notes from a pairing session between @Kellel and I, documenting my first time using amtool:

  • error message when URL still needs work
  • add a verbose flag to print which instance connected to, which config file using
  • alert name in separate column
  • should --silenced show only silenced?
  • should --all be named --expired?
  • usage for search
  • partial matches (possibly with a flag?)
  • panic
matt➜github.com/prometheus/alertmanager(Kellel-amtool)» ./amtool alert 'alertname=~searchfo.+'
Using config file: /Users/matt/.amtool.yml
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x13a896]

goroutine 1 [running]:
panic(0x3e8360, 0xc4200140d0)
	/usr/local/Cellar/go/1.7.4/libexec/src/runtime/panic.go:500 +0x1a1
regexp.(*Regexp).get(0x0, 0x3bf400)


	/usr/local/Cellar/go/1.7.4/libexec/src/regexp/regexp.go:208 +0x26
regexp.(*Regexp).doExecute(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc4201488c0, 0xb, 0x0, 0x0, ...)
	/usr/local/Cellar/go/1.7.4/libexec/src/regexp/exec.go:416 +0x40
regexp.(*Regexp).MatchString(0x0, 0xc4201488c0, 0xb, 0x9)
	/usr/local/Cellar/go/1.7.4/libexec/src/regexp/regexp.go:416 +0x88
github.com/prometheus/alertmanager/types.(*Matcher).Match(0xc42012d800, 0xc42012d260, 0x3c4c40)
	/Users/matt/go/src/github.com/prometheus/alertmanager/types/match.go:61 +0xc2
github.com/prometheus/alertmanager/types.Matchers.Match(0xc42002c258, 0x1, 0x1, 0xc42012d260, 0x0)
	/Users/matt/go/src/github.com/prometheus/alertmanager/types/match.go:135 +0x4e
github.com/prometheus/alertmanager/cli.queryAlerts(0x65da40, 0xc420148020, 0x1, 0x1, 0x0, 0x0)
	/Users/matt/go/src/github.com/prometheus/alertmanager/cli/alert.go:129 +0x32a
github.com/prometheus/alertmanager/vendor/github.com/spf13/cobra.(*Command).execute(0x65da40, 0xc420015fd0, 0x1, 0x1, 0x65da40, 0xc420015fd0)
	/Users/matt/go/src/github.com/prometheus/alertmanager/vendor/github.com/spf13/cobra/command.go:632 +0x23e
github.com/prometheus/alertmanager/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x65de80, 0xc42006e058, 0x0, 0xc420015f29)
	/Users/matt/go/src/github.com/prometheus/alertmanager/vendor/github.com/spf13/cobra/command.go:722 +0x367
github.com/prometheus/alertmanager/vendor/github.com/spf13/cobra.(*Command).Execute(0x65de80, 0x0, 0x0)
	/Users/matt/go/src/github.com/prometheus/alertmanager/vendor/github.com/spf13/cobra/command.go:681 +0x2b
github.com/prometheus/alertmanager/cli.Execute()
	/Users/matt/go/src/github.com/prometheus/alertmanager/cli/root.go:23 +0x31
main.main()
	/Users/matt/go/src/github.com/prometheus/alertmanager/cmd/amtool/main.go:6 +0x14
  • if no labels specified, search all labels
  • remove nanoseconds from timestamps
  • remove timezone for now if everything is UTC anyway, leave 'UTC' so that it's obvious
  • double quotes around all label values
  • humanize timestamps (relies on localisation)
  • document all config file options (e.g. comment_required) in README
  • rename --until to --expire-on
  • add usage for silences matchers (not clear what to type)
  • output how many silences were silenced (or say if none)
  • error when no matchers specified for silence
  • print 'simple' formatted output when adding silence to confirm what I just added
  • ability to edit silence comment
  • alert label name duplicated in output when adding multiple silences
matt➜github.com/prometheus/alertmanager(Kellel-amtool✗)» ./amtool silence add -a bob -c 'test comment' -e 4h alertname=~My_First_Alert
Using config file: /Users/matt/.amtool.yml
8813ee76-2f54-4e57-88ca-a19fc4a049b0
matt➜github.com/prometheus/alertmanager(Kellel-amtool✗)» ./amtool silence
Using config file: /Users/matt/.amtool.yml
ID                                    Matchers                                  Ends At                                  Created By  Comment
31dd1b0d-064f-478e-b9e2-6b6d76a5b09c  alertname=My_First_Alert              2017-01-24 20:45:39.810092795 +0000 UTC  bob         test comment
4f19c6eb-31d2-4f88-8978-8d999dfb57dd  alertname~=alertname=~My_First_Alert  2017-01-24 20:49:16.316893423 +0000 UTC  bob         test comment
6df7c7d7-8509-4db5-87fd-3752e20d118c  alertname~=alertname=~My_First_Alert  2017-01-24 20:49:19.268261913 +0000 UTC  bob         test comment
ac703b2f-1cef-4efd-abbb-8152c7fbf3f3  alertname~=alertname=~My_First_Alert  2017-01-24 20:49:19.977599302 +0000 UTC  bob         test comment
a7d156f4-13a0-4615-bb6c-dd04fab68f24  alertname~=alertname=~My_First_Alert  2017-01-24 20:49:20.474452101 +0000 UTC  bob         test comment
8813ee76-2f54-4e57-88ca-a19fc4a049b0  alertname~=alertname=~My_First_Alert  2017-01-24 20:49:21.419070481 +0000 UTC  bob         test comment
  • if silence already exists and matches, maybe we should prevent a new silence from being added
  • mention tab completion in README
  • mention shell expansion on labels
  • document extended output for 'config' action
  • add version flag to CLI (to detect mismatches with server)

cli/silence.go Outdated
var silenceCmd = &cobra.Command{
Use: "silence",
Short: "Manage silences",
Long: `Add, expire or view silences For more information and additional flags see query help`,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can probably remove 'For more information and additional flags see query help', not sure it adds anything.

@Kellel
Copy link
Owner Author

Kellel commented Jan 27, 2017

I've reformatted the list from my session with @mattbostock I've prioritized the items to help myself focus (x=complete, ● = todo)

BUGS

(Need immediate attention)
x alert label name duplicated in output when adding multiple silences

 matt➜github.com/prometheus/alertmanager(Kellel-amtool✗)» ./amtool silence add -a bob -c 'test comment' -e 4h alertname=~My_First_Alert
 Using config file: /Users/matt/.amtool.yml
 8813ee76-2f54-4e57-88ca-a19fc4a049b0
 matt➜github.com/prometheus/alertmanager(Kellel-amtool✗)» ./amtool silence
 Using config file: /Users/matt/.amtool.yml
 ID                                    Matchers                                  Ends At                                  Created By  Comment
 31dd1b0d-064f-478e-b9e2-6b6d76a5b09c  alertname=My_First_Alert              2017-01-24 20:45:39.810092795 +0000 UTC  bob         test comment
 4f19c6eb-31d2-4f88-8978-8d999dfb57dd  alertname~=alertname=~My_First_Alert  2017-01-24 20:49:16.316893423 +0000 UTC  bob         test comment
 6df7c7d7-8509-4db5-87fd-3752e20d118c  alertname~=alertname=~My_First_Alert  2017-01-24 20:49:19.268261913 +0000 UTC  bob         test comment
 ac703b2f-1cef-4efd-abbb-8152c7fbf3f3  alertname~=alertname=~My_First_Alert  2017-01-24 20:49:19.977599302 +0000 UTC  bob         test comment
 a7d156f4-13a0-4615-bb6c-dd04fab68f24  alertname~=alertname=~My_First_Alert  2017-01-24 20:49:20.474452101 +0000 UTC  bob         test comment
 8813ee76-2f54-4e57-88ca-a19fc4a049b0  alertname~=alertname=~My_First_Alert  2017-01-24 20:49:21.419070481 +0000 UTC  bob         test comment

x panic

 matt➜github.com/prometheus/alertmanager(Kellel-amtool)» ./amtool alert 'alertname=~searchfo.+'
 Using config file: /Users/matt/.amtool.yml
 panic: runtime error: invalid memory address or nil pointer dereference
 [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x13a896]

 goroutine 1 [running]:
 panic(0x3e8360, 0xc4200140d0)
    /usr/local/Cellar/go/1.7.4/libexec/src/runtime/panic.go:500 +0x1a1
 regexp.(*Regexp).get(0x0, 0x3bf400)


    /usr/local/Cellar/go/1.7.4/libexec/src/regexp/regexp.go:208 +0x26
 regexp.(*Regexp).doExecute(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc4201488c0, 0xb, 0x0, 0x0, ...)
    /usr/local/Cellar/go/1.7.4/libexec/src/regexp/exec.go:416 +0x40
 regexp.(*Regexp).MatchString(0x0, 0xc4201488c0, 0xb, 0x9)
    /usr/local/Cellar/go/1.7.4/libexec/src/regexp/regexp.go:416 +0x88
 github.com/prometheus/alertmanager/types.(*Matcher).Match(0xc42012d800, 0xc42012d260, 0x3c4c40)
    /Users/matt/go/src/github.com/prometheus/alertmanager/types/match.go:61 +0xc2
 github.com/prometheus/alertmanager/types.Matchers.Match(0xc42002c258, 0x1, 0x1, 0xc42012d260, 0x0)
    /Users/matt/go/src/github.com/prometheus/alertmanager/types/match.go:135 +0x4e
 github.com/prometheus/alertmanager/cli.queryAlerts(0x65da40, 0xc420148020, 0x1, 0x1, 0x0, 0x0)
    /Users/matt/go/src/github.com/prometheus/alertmanager/cli/alert.go:129 +0x32a
 github.com/prometheus/alertmanager/vendor/github.com/spf13/cobra.(*Command).execute(0x65da40, 0xc420015fd0, 0x1, 0x1, 0x65da40, 0xc420015fd0)
    /Users/matt/go/src/github.com/prometheus/alertmanager/vendor/github.com/spf13/cobra/command.go:632 +0x23e
 github.com/prometheus/alertmanager/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x65de80, 0xc42006e058, 0x0, 0xc420015f29)
    /Users/matt/go/src/github.com/prometheus/alertmanager/vendor/github.com/spf13/cobra/command.go:722 +0x367
 github.com/prometheus/alertmanager/vendor/github.com/spf13/cobra.(*Command).Execute(0x65de80, 0x0, 0x0)
    /Users/matt/go/src/github.com/prometheus/alertmanager/vendor/github.com/spf13/cobra/command.go:681 +0x2b
 github.com/prometheus/alertmanager/cli.Execute()
    /Users/matt/go/src/github.com/prometheus/alertmanager/cli/root.go:23 +0x31
 main.main()
    /Users/matt/go/src/github.com/prometheus/alertmanager/cmd/amtool/main.go:6 +0x14

REQUIRED Features

(Required for merge)
x error message when URL still needs work
x document extended output for 'config' action
x rename --until to --expire-on
x should --all be named --expired?
x usage for search
x remove nanoseconds from timestamps
x remove timezone for now if everything is UTC anyway, leave 'UTC' so that it's obvious
┕> Added custom date formats via configfile date.format option and made a cleaner looking default format
x double quotes around all label values
x error when no matchers specified for silence

  • Update README
    • mention tab completion in README
    • document all config file options (e.g. comment_required) in README
  • add usage for silences matchers (not clear what to type)
    ┕> mention shell expansion on labels
    • silence add
      x silence query
      x alert

PROBABLY

(Things that are lower priority, but most likely need to be added before merging to core

  • partial matches (possibly with a flag?)
  • if no labels specified, search all labels
  • if silence already exists and matches, maybe we should prevent a new silence from being added

NICE to haves

(things that I would like to implement, but I find lower priority)

  • add a verbose flag to print which instance connected to, which config file using
  • add version flag to CLI (to detect mismatches with server)
  • humanize timestamps (relies on localisation)

DEBATABLE

(things I'm not sure if I agree with. I will hold on to these and see if the core devs want to see them implemented as well)

  • ability to edit silence comment and other things (this would be cool, but the logic will require delete/add as the api currently doesn't support edit)
  • alert name in separate column
  • should --silenced show only silenced?
  • output how many alerts were silenced (or say if none)
  • print 'simple' formatted output when adding silence to confirm what I just added

Kellen Fox added 3 commits February 24, 2017 15:00
 * also allow for man page generation
 * update vendors to include cobra/doc
 * split silence commands into multiple files
 * move some ommon gits into utils.go
@Kellel Kellel merged commit 597a362 into master Feb 25, 2017
Kellel pushed a commit that referenced this pull request Jun 2, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants