Found 2 different approaches to solve this issue:

• Use Walkfunc with callback and expandEnvVars like in Nuclei.
• Create a temp config file and replace the words that start with $ and then decode using yaml library.

The problem I'm facing is that the yaml decoder does not set boolean values in any of the following cases:

• Set in environment variables.
• Hardcoded in the config file.

It always defaults to false.

I've tried using both the std lib pkg and the yamlutil used in Nuclei.
Any suggestions?
CC: @ehsandeep @Mzack9999 @tarunKoyalwar

we can go with second approach..

// utils.go
func readProviderConfig(filepath string) (io.Reader, error) {
	// Open the file
	file, err := os.Open(filepath)
	if err != nil {
		return nil, err
	}
	defer file.Close()

	// Create a scanner to read the file line by line
	scanner := bufio.NewScanner(file)

	// Create a string builder to accumulate the modified data
	var sb strings.Builder

	// Iterate over each line and do variable substitution
	for scanner.Scan() {
		line := scanner.Text()
		newLine := substituteEnvVars(line)
		sb.WriteString(newLine)
		sb.WriteString("\n")
	}
	// Check for errors
	if err := scanner.Err(); err != nil {
		return nil, err
	}

	return strings.NewReader(sb.String()), nil
}

func substituteEnvVars(line string) string {
	for _, word := range strings.Fields(line) {
		word = strings.Trim(word, " \"")
		if strings.HasPrefix(word, "$") {
			key := strings.TrimPrefix(word, "$")
			val := os.Getenv(key)
			line = strings.ReplaceAll(line, word, val)
		}
	}
	return line
}

runner.go

	reader, err := readProviderConfig(options.ProviderConfig)
	if err != nil {
		return nil, errors.Wrap(err, "could not open provider config file")
	}
	if parseErr := yaml.NewDecoder(reader).Decode(&providerOptions); parseErr != nil {
		return nil, errors.Wrap(parseErr, "could not parse provider config file")
	}

@ShubhamRasal

• Thank you for the suggestion. It does solve the problem I was facing.
• All datatypes that are part of the providerOptions struct can be parsed.

@iamargus95 , since this is a generic implementation what do you think of creating new PR to move this function to https://github.com/projectdiscovery/utils/tree/main/file . that way we can extend this support to other projects

Thanks

@tarunKoyalwar
Kindly review this PR in utils.
I've added this functionality as a helper.

• Add ability to set config from env vars utils#149

@iamargus95 , since upstream pr projectdiscovery/utils#149 is merged now can you update this PR to use those helper utils

@tarunKoyalwar I think the latest release (v0.0.30) does not have the newly added helper function. Waiting for it to be added.

@iamargus95 , since upstream pr projectdiscovery/utils#149 is merged now can you update this PR to use those helper utils

@tarunKoyalwar I think the latest release (v0.0.30) does not have the newly added helper function. Waiting for it to be added.

https://github.com/projectdiscovery/utils/releases/tag/v0.0.31

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication