Build nplb_evergreen_compat_tests with default toolchain - #5691

@andrewsavage1 we have 2 options here i think :
Build nplb_evergreen_compat_tests as a

1. starboard_level_final_executable_type with starboard toolchain
   or
2. gtest_target_type with default_toolchain

actually 2) is already done in main branch https://source.corp.google.com/h/github/youtube/cobalt/+/main:starboard/nplb/BUILD.gn;l=320
So im sticking to 2)

@andrewsavage1 we have 2 options here i think : Build nplb_evergreen_compat_tests as a

1. starboard_level_final_executable_type with starboard toolchain
   or
2. gtest_target_type with default_toolchain

actually 2) is already done in main branch https://source.corp.google.com/h/github/youtube/cobalt/+/main:starboard/nplb/BUILD.gn;l=320 So im sticking to 2)

Yeah seems reasonable

Oh sorry, I didn't realize this was in 25.lts.1+ and not the main branch—in that case, I worry that this would change the pattern partners are following in the middle of their cert cycles, which would be confusing. I assume there's already documentation they follow to run these compat tests, and I don't want to affect that in an LTS branch.

Is there an RDK reason this needs to be done?

Just a note that I think I rolled these tests up into "just" nplb, on ToT. So this PR is Co25 specific.

Oh sorry, I didn't realize this was in 25.lts.1+ and not the main branch—in that case, I worry that this would change the pattern partners are following in the middle of their cert cycles, which would be confusing. I assume there's already documentation they follow to run these compat tests, and I don't want to affect that in an LTS branch.

Is there an RDK reason this needs to be done?

there's a bunch of choices here. Context: we want to have nplb_evergreen_compat_tests running on CI in the evergreen config- along with nplb - those are the 2 tests required by partners to pass cert.

first off i can't find any explicit partner documentation which says build nplb_evergreen_compat_tests monolithically/ using the default monolithic config.
There is documentation on how to run nplb : https://developers.google.com/youtube/cobalt/docs/development/setup-linux#build_nplb_library_using_cobalt_toolchain
I'm trying to make the process for running nplb_eg_compat_tests similar to nplb, otherwise you have 2 configs where:

1. nplb - runs in the evergreen way built as a shared lib with cobalt toolchain
2. compat_tests: which runs in the monolithic way - built as an executable with sb toolchain.

there's a bunch of choices here. Context: we want to have nplb_evergreen_compat_tests running on CI in the evergreen config- along with nplb - those are the 2 tests required by partners to pass cert.

first off i can't find any explicit partner documentation which says build nplb_evergreen_compat_tests monolithically/ using the default monolithic config. There is documentation on how to run nplb : https://developers.google.com/youtube/cobalt/docs/development/setup-linux#build_nplb_library_using_cobalt_toolchain I'm trying to make the process for running nplb_eg_compat_tests similar to nplb, otherwise you have 2 configs where:

1. nplb - runs in the evergreen way built as a shared lib with cobalt toolchain
2. compat_tests: which runs in the monolithic way - built as an executable with sb toolchain.

I'll leave this choice to @hlwarriner who knows more about these tests and what we've been telling partners than I do

there's a bunch of choices here. Context: we want to have nplb_evergreen_compat_tests running on CI in the evergreen config- along with nplb - those are the 2 tests required by partners to pass cert.
first off i can't find any explicit partner documentation which says build nplb_evergreen_compat_tests monolithically/ using the default monolithic config. There is documentation on how to run nplb : https://developers.google.com/youtube/cobalt/docs/development/setup-linux#build_nplb_library_using_cobalt_toolchain I'm trying to make the process for running nplb_eg_compat_tests similar to nplb, otherwise you have 2 configs where:

1. nplb - runs in the evergreen way built as a shared lib with cobalt toolchain
2. compat_tests: which runs in the monolithic way - built as an executable with sb toolchain.

I'll leave this choice to @hlwarriner who knows more about these tests and what we've been telling partners than I do

I support building nplb_evergreen_compat_tests as an evergreen binary that can be loaded and actually suggested this exact code change in https://b.corp.google.com/issues/378586326#comment2. But I have the same question Andrew had and still don't quite understand the value of making this change in 25.lts.1+ at this point.

In reality I think it's unlikely we'd break or disrupt any partners with this change. I'm pretty sure we just ask them to manually run the nplb_evergreen_compat_tests tests and submit the results, so it would probably be fine for them to continue to run them either way - as a standalone executable (old way) or as an evergreen shared library that's loaded (new way).

It sounds like the main reason for this change is to get these tests running in CI? I think that would be good in theory but you should be aware the tests weren't designed to run in CI and there may be some issues: https://b.corp.google.com/issues/292138589#comment2. Some of these issues aren't relevant if we are just talking about a few select reference platforms on 25.lts.1+, but it's possible there are issues with some of the tests.

So in summary:

• I'd prefer we keep the status quo in 25.lts.1+ and make the change in chrobalt (I think already done, in a different way, by starboard/nplb: Fold nplb_evergreen_compat_tests into nplb proper #4946?)
• If you really want to make this change in 25.lts.1+, I'm ok with it, but you may need to resolve some test failures or filter some tests out in CI

there's a bunch of choices here. Context: we want to have nplb_evergreen_compat_tests running on CI in the evergreen config- along with nplb - those are the 2 tests required by partners to pass cert.
first off i can't find any explicit partner documentation which says build nplb_evergreen_compat_tests monolithically/ using the default monolithic config. There is documentation on how to run nplb : https://developers.google.com/youtube/cobalt/docs/development/setup-linux#build_nplb_library_using_cobalt_toolchain I'm trying to make the process for running nplb_eg_compat_tests similar to nplb, otherwise you have 2 configs where:

1. nplb - runs in the evergreen way built as a shared lib with cobalt toolchain
2. compat_tests: which runs in the monolithic way - built as an executable with sb toolchain.

I'll leave this choice to @hlwarriner who knows more about these tests and what we've been telling partners than I do

I support building nplb_evergreen_compat_tests as an evergreen binary that can be loaded and actually suggested this exact code change in https://b.corp.google.com/issues/378586326#comment2. But I have the same question Andrew had and still don't quite understand the value of making this change in 25.lts.1+ at this point.

In reality I think it's unlikely we'd break or disrupt any partners with this change. I'm pretty sure we just ask them to manually run the nplb_evergreen_compat_tests tests and submit the results, so it would probably be fine for them to continue to run them either way - as a standalone executable (old way) or as an evergreen shared library that's loaded (new way).

It sounds like the main reason for this change is to get these tests running in CI? I think that would be good in theory but you should be aware the tests weren't designed to run in CI and there may be some issues: https://b.corp.google.com/issues/292138589#comment2. Some of these issues aren't relevant if we are just talking about a few select reference platforms on 25.lts.1+, but it's possible there are issues with some of the tests.

So in summary:

• I'd prefer we keep the status quo in 25.lts.1+ and make the change in chrobalt (I think already done, in a different way, by starboard/nplb: Fold nplb_evergreen_compat_tests into nplb proper #4946?)
• If you really want to make this change in 25.lts.1+, I'm ok with it, but you may need to resolve some test failures or filter some tests out in CI

it turns out my previous change was not enough to get it building with evergreen. iadded a new change.
with the new change - the target is buildable the old way(sb toolchain) and the new way(cobalt toolchain) .
This whole change is int he context of RDK reference platform - Andre (from Collabora) is adding CI builders to run nplb_evergreen_compat_tests . I would prefer if he uses the same way of running tests as nplb for evergreen.
The current condition would make it both forward and backward compatible - does that sg ?
I would prefer only the newer way of building the target(building with cobalt tc) but if you prefer having the flexibility of having both ways- that's fine too. wdyt ?

there's a bunch of choices here. Context: we want to have nplb_evergreen_compat_tests running on CI in the evergreen config- along with nplb - those are the 2 tests required by partners to pass cert.
first off i can't find any explicit partner documentation which says build nplb_evergreen_compat_tests monolithically/ using the default monolithic config. There is documentation on how to run nplb : https://developers.google.com/youtube/cobalt/docs/development/setup-linux#build_nplb_library_using_cobalt_toolchain I'm trying to make the process for running nplb_eg_compat_tests similar to nplb, otherwise you have 2 configs where:

1. nplb - runs in the evergreen way built as a shared lib with cobalt toolchain
2. compat_tests: which runs in the monolithic way - built as an executable with sb toolchain.

I'll leave this choice to @hlwarriner who knows more about these tests and what we've been telling partners than I do

I support building nplb_evergreen_compat_tests as an evergreen binary that can be loaded and actually suggested this exact code change in https://b.corp.google.com/issues/378586326#comment2. But I have the same question Andrew had and still don't quite understand the value of making this change in 25.lts.1+ at this point.
In reality I think it's unlikely we'd break or disrupt any partners with this change. I'm pretty sure we just ask them to manually run the nplb_evergreen_compat_tests tests and submit the results, so it would probably be fine for them to continue to run them either way - as a standalone executable (old way) or as an evergreen shared library that's loaded (new way).
It sounds like the main reason for this change is to get these tests running in CI? I think that would be good in theory but you should be aware the tests weren't designed to run in CI and there may be some issues: https://b.corp.google.com/issues/292138589#comment2. Some of these issues aren't relevant if we are just talking about a few select reference platforms on 25.lts.1+, but it's possible there are issues with some of the tests.
So in summary:

• I'd prefer we keep the status quo in 25.lts.1+ and make the change in chrobalt (I think already done, in a different way, by starboard/nplb: Fold nplb_evergreen_compat_tests into nplb proper #4946?)
• If you really want to make this change in 25.lts.1+, I'm ok with it, but you may need to resolve some test failures or filter some tests out in CI

it turns out my previous change was not enough to get it building with evergreen. iadded a new change. with the new change - the target is buildable the old way(sb toolchain) and the new way(cobalt toolchain) . This whole change is int he context of RDK reference platform - Andre (from Collabora) is adding CI builders to run nplb_evergreen_compat_tests . I would prefer if he uses the same way of running tests as nplb for evergreen. The current condition would make it both forward and backward compatible - does that sg ? I would prefer only the newer way of building the target(building with cobalt tc) but if you prefer having the flexibility of having both ways- that's fine too. wdyt ?

Resolved offline - I'd prefer to keep the change as you have it now to support both modes of building the target in c25. Thanks!

failure in nplb tests on android looks unrelated to this PR http://shortn/_SSJJhDb3Ow