Added the dont-merge/blocked label, because of the upcoming conflicts with the following PR:

• Add an option to disable L3/L4 network policy correlation of flows #37986

Since this one is newer and pure refactoring I think the conflicts will be easier to resolve if we merge #37986 first.

Sounds good to me, I will wait for it to be merged and then rebase my changes with updated commit msg.

Added the dont-merge/blocked label, because of the upcoming conflicts with the following PR:

• Adding an option to disable L3/L4 network policy correlation of flows #37986

Since this one is newer and pure refactoring I think the conflicts will be easier to resolve if we merge #37986 first.

This is now merged!

• Cilium E2E Upgrade and Cilium IPsec upgrade: 1.1.1.1 blip
• Conformance Ginkgo: You have reached your unauthenticated pull rate limit (docker rate-limited)
• Conformance Runtime Runtime Test (privileged) and Integration Tests first time seeing this, could be related to the PR:

  FAIL: TestAgentCell
  cells_test.go:46: found unexpected goroutines
  

Conformance Runtime Runtime Test (privileged) and Integration Tests first time seeing this, could be related to the PR

They did fail again. I am trying to figure out the issue, will ping again if I find something, in the mean time it would be great help if you can give your thoughts the potential issue.

@ritwikranjan
Failing test:

• cilium/daemon/cmd/cells_test.go

  Lines 31 to 34 in ba1e34a

  	// TestAgentCell verifies that the Agent hive can be instantiated with
  	// default configuration and thus the Agent hive can be inspected with
  	// the hive commands and documentation can be generated from it.
  	func TestAgentCell(t *testing.T) {

  
  Now the full test error message is:

    cells_test.go:46: found unexpected goroutines:
        [Goroutine 11 in state select, with github.com/cilium/cilium/pkg/controller.(*controller).runController on top of the stack:
        github.com/cilium/cilium/pkg/controller.(*controller).runController(0xc000df6a00, {{{0x5127a7a, 0xa}}, {0x0, 0x0}, 0x5368e10, 0x0, 0x53663f0, 0x37e11d600, 0x0, ...})
        	/host/pkg/controller/controller.go:319 +0x939
        created by github.com/cilium/cilium/pkg/controller.(*Manager).createControllerLocked in goroutine 6
        	/host/pkg/controller/manager.go:116 +0x438
        ]

So the initial inspection hint at an unexpected goroutine launched on Hive instantiation. The PR introduces:

parser.New(params.Log, g, g, g, params.Ipcache, g, link.NewLinkCache(), params.CGroupManager, params.Config.SkipUnknownCGroupIDs, parserOpts...)

in newPayloadParser — i.e. at instantiation — which was previously in (*hubbleIntegration) launch() (part of the hubble job lifecycle). Going down link.NewLinkCache():

• cilium/pkg/datapath/link/link.go

  Line 88 in ba1e34a

  controller.NewManager().UpdateController("link-cache",

• cilium/pkg/controller/manager.go

  Line 53 in ba1e34a

  m.updateController(name, params)

• cilium/pkg/controller/manager.go

  Line 89 in ba1e34a

  return m.createControllerLocked(name, params)

• cilium/pkg/controller/manager.go

  Line 116 in ba1e34a

  go ctrl.runController(ctrl.params)

The cleanest way to address this would be to refactor linkcache into a Cell with proper lifecycle cc @mhofstetter

@kaworu I have done some refactoring with link cache, can you retry the tests? Also think I need another review from datapath sig.

/test

Hey @kaworu @gentoo-root can you also take a look as well and merge the changes if all looks okay?

@kaworu is it okay to merge the PR?

It looks like this change made the --hubble-network-policy-correlation-enabled flag ineffective, i.e. network policy correlation can no longer be disabled. I've opened #39142 to fix this.