Nice, I like this, and it's also a really unexpectedly small code change 😄

The only issue I see that I'd like to discuss a bit, is what if users want to apply multiple filters? For example, for aws_cloudtrail_events, LookupAttributes are unfortunately limited to a single key and value ("Currently the list can contain only one item."). If CQ users wanted to sync more... well, they'd need to do that in an entirely different sync.

One could argue that this is a limitation of the AWS API, take it up with them. But on the other hand, we do have ways to deal with this. For example, what if instead of using:

type ctAPIs struct {
	LookupEventsOpts cloudtrail.LookupEventsInput `json:"LookupEvents,omitempty"`
}

we did:

type ctAPIs struct {
	LookupEventsOpts []cloudtrail.LookupEventsInput `json:"LookupEvents,omitempty"`
}

(so LookupEvents becomes a slice). Then the config becomes this:

  spec:
    table_options:
      aws_cloudtrail_events:
        LookupEvents:
          - StartTime: 2023-05-01T20:20:52Z
            EndTime:   2023-05-03T20:20:52Z
            LookupAttributes:
              - AttributeKey: EventName
                AttributeValue: RunInstances

Which is almost exactly the same as before, except there's an extra dash. However it can then be used in this way:

  spec:
    table_options:
      aws_cloudtrail_events:
        LookupEvents:
          - StartTime: 2023-05-01T20:20:52Z
            EndTime:   2023-05-03T20:20:52Z
            LookupAttributes:
              - AttributeKey: EventName
                AttributeValue: RunInstances
          - StartTime: 2023-05-01T20:20:52Z
            EndTime:   2023-05-03T20:20:52Z
            LookupAttributes:
              - AttributeKey: EventName
                AttributeValue: StartInstances
          - StartTime: 2023-05-01T20:20:52Z
            EndTime:   2023-05-03T20:20:52Z
            LookupAttributes:
              - AttributeKey: EventName
                AttributeValue: StopInstances

You'd have to repeat the start time/end time if you need this, but at least you could then fetch all the events you need in a single sync. Inside the resolver, we could loop over each entry in the list and do them one-by-one (probably in separate goroutines, but not necessarily).

What do you think?

@hermanschaaf - I like the idea of taking in an array of input arguments and then iterating over them! I think if we do it for a single table we should support it for all tables/APIs that support user provided inputs. One other thing that this will possibly generate will be duplicate data as the filters could theoretically overlap. Is that a concern?

Where do we stand on account-level filtering of these options? Each account gets its own list of TableOptions. But then you'd need catch-all etc. as well so it could get out of hand quickly. Maybe some kind of an account-id-match "metafield" can be included, which could take wildcards or multiple account ids. In a future iteration.

Each account gets its own list of TableOptions

At this time I think we should start with just a single "global" TableOptions if users need an account specific filters for specific accounts/regions they can use multiple source blocks

I will not be including the documentation in this PR. I am working on a follow up PR that will solely be the documentation

I had a few comments on the codegen, but hoping we can get away with a shorter solution that doesn't require any codegen; here's a branch that demonstrates the idea (and some other changes): https://github.com/cloudquery/cloudquery/compare/aws-inputs-simplified

The basic idea is to use a custom JSON Unmarshal function that first unmarshals to a map[string]any, then transforms all the key names from snake to camel, then marshalling the map back to JSON and calling the default config object unmarshal on that instead. It seems to work in my testing and it's relatively simple I think

@hermanschaaf - I am fine with your approach... In the end both approaches achieve the same thing with the exact same public interface. The difference that I see between the two solutions is the codegen solutions handles all of the name changing at build time rather than run time. Having access to the structs that define TableOptions might be important when it comes to documenting the object.

@bbernays I pushed a couple of commits to the branch:

• A test to validate the snake case JSON parsing is working as expected
• A fix for an edge case discovered by the above
• Renaming table_options package tableoptions (not great for readability, but more consistent with our other packages)
• A commit to use lists in the config like we discussed 600879c This was a bit trickier than I thought for Cloudtrail events, because there we have to carefully combine it with the state backend. I kept it backwards-compatible, but please have a look there and let me know whether you agree this is the right way to implement it. None of these are using additional goroutines yet. I think that's okay for now