HI @WarmBeer, I'm reviewing this PR, but my first impressions are:

• The "primitives" package is actually the core tracker configuration. I would rename it to "core-tracker-configuration".
• The "test-helpers" package contains helpers only for the core tracker configuration. Besides, even though the function ephemeral is used only for testing, I would move it the the "core-tracker-configuration" package. In fact, it's implemented like a production feature because now you can run the tracker in production using port "0" and letting the OS assign a free port. For me, that's a production feature and we are taking advantage of it for testing. Besides, that would simplify packages, we would only need one.

I suppose you want to add more stuff in the "test-helpers" package in the future, but we should carefully think about the structure we want in the future, otherwise, It could a pain to reorganize the packages. I think in the future, I could end up having at least 4 main components:

• The "core-tracer" with its configuration.
• The "udp-tracker" with the "core-tracker" dependency.
• The "http-tracker" with the "core-tracker" dependency.
• The "tracker-api" with the "core-tracker" dependency.

If we end up having those packages (independently released), we would need to extract things from "primitives" and "test-helpers" packages. And that could be a big effort if those packages become too big.

• The "test-helpers" package contains helpers only for the core tracker configuration. Besides, even though the function ephemeral is used only for testing, I would move it the the "core-tracker-configuration" package. In fact, it's implemented like a production feature because now you can run the tracker in production using port "0" and letting the OS assign a free port. For me, that's a production feature and we are taking advantage of it for testing. Besides, that would simplify packages, we would only need one.

I suppose you want to add more stuff in the "test-helpers" package in the future, but we should carefully think about the structure we want in the future, otherwise, It could a pain to reorganize the packages. I think in the future, I could end up having at least 4 main components:

• The "core-tracer" with its configuration.
• The "udp-tracker" with the "core-tracker" dependency.
• The "http-tracker" with the "core-tracker" dependency.
• The "tracker-api" with the "core-tracker" dependency.

If we end up having those packages (independently released), we would need to extract things from "primitives" and "test-helpers" packages. And that could be a big effort if those packages become too big.

Hey @josecelano ,

Thank you for reviewing!

HI @WarmBeer, I'm reviewing this PR, but my first impressions are:

• The "primitives" package is actually the core tracker configuration. I would rename it to "core-tracker-configuration".

The "primitives" crate is supposed to be the crate that holds all the common types and functions that don't fit anywhere else. Like https://github.com/paritytech/polkadot/blob/master/core-primitives/src/lib.rs .

In this PR, I have only moved the DatabaseDriver and TrackerMode enums to it, to avoid a circular dependency issue (as the configuration crate depends on these enums).

If we isolate the database logic and tracker logic into their own crates, we could move these enums to those and might not even need a primitives crate. Although it might not be a bad idea to have a light primitives crate that we can import into our other crates. Having to depend on a big database crate when you only need one type is not very efficient.

The "test-helpers" package contains helpers only for the core tracker configuration. Besides, even though the function ephemeral is used only for testing, I would move it the the "core-tracker-configuration" package. In fact, it's implemented like a production feature because now you can run the tracker in production using port "0" and letting the OS assign a free port. For me, that's a production feature and we are taking advantage of it for testing. Besides, that would simplify packages, we would only need one.

Moving ephemeral to the configuration crate is fine by me. I just think that we should keep the test-helpers crate to easily share helper structs and functions between integration and unit tests. Even though it is quite empty now, I see it being used a lot in the future if we want to avoid adding test code to production.

If we end up having those packages (independently released), we would need to extract things from "primitives" and "test-helpers" packages. And that could be a big effort if those packages become too big.

The idea is that all our packages could be independently released as-is.

I think that we should focus on crate isolation rather sooner than later. I had to deal with a lot of merge conflicts for this PR. Isolating crates will also speed up the dreadful compilation time: https://endler.dev/2020/rust-compile-times/#use-cargo-workspaces

• The "test-helpers" package contains helpers only for the core tracker configuration. Besides, even though the function ephemeral is used only for testing, I would move it the the "core-tracker-configuration" package. In fact, it's implemented like a production feature because now you can run the tracker in production using port "0" and letting the OS assign a free port. For me, that's a production feature and we are taking advantage of it for testing. Besides, that would simplify packages, we would only need one.

I suppose you want to add more stuff in the "test-helpers" package in the future, but we should carefully think about the structure we want in the future, otherwise, It could a pain to reorganize the packages. I think in the future, I could end up having at least 4 main components:

• The "core-tracer" with its configuration.
• The "udp-tracker" with the "core-tracker" dependency.
• The "http-tracker" with the "core-tracker" dependency.
• The "tracker-api" with the "core-tracker" dependency.

If we end up having those packages (independently released), we would need to extract things from "primitives" and "test-helpers" packages. And that could be a big effort if those packages become too big.

Hey @josecelano ,

Thank you for reviewing!

HI @WarmBeer, I'm reviewing this PR, but my first impressions are:

• The "primitives" package is actually the core tracker configuration. I would rename it to "core-tracker-configuration".

The "primitives" crate is supposed to be the crate that holds all the common types and functions that don't fit anywhere else. Like https://github.com/paritytech/polkadot/blob/master/core-primitives/src/lib.rs .

In this PR, I have only moved the DatabaseDriver and TrackerMode enums to it, to avoid a circular dependency issue (as the configuration crate depends on these enums).

If we isolate the database logic and tracker logic into their own crates, we could move these enums to those and might not even need a primitives crate. Although it might not be a bad idea to have a light primitives crate that we can import into our other crates. Having to depend on a big database crate when you only need one type is not very efficient.

The "test-helpers" package contains helpers only for the core tracker configuration. Besides, even though the function ephemeral is used only for testing, I would move it the the "core-tracker-configuration" package. In fact, it's implemented like a production feature because now you can run the tracker in production using port "0" and letting the OS assign a free port. For me, that's a production feature and we are taking advantage of it for testing. Besides, that would simplify packages, we would only need one.

Moving ephemeral to the configuration crate is fine by me. I just think that we should keep the test-helpers crate to easily share helper structs and functions between integration and unit tests. Even though it is quite empty now, I see it being used a lot in the future if we want to avoid adding test code to production.

If we end up having those packages (independently released), we would need to extract things from "primitives" and "test-helpers" packages. And that could be a big effort if those packages become too big.

The idea is that all our packages could be independently released as-is.

I think that we should focus on crate isolation rather sooner than later. I had to deal with a lot of merge conflicts for this PR. Isolating crates will also speed up the dreadful compilation time: https://endler.dev/2020/rust-compile-times/#use-cargo-workspaces

hi, @WarmBeer have you considered having multiple primitive packages? I'm afraid of having all packaged coupled to that one. But I like the idea of extracting only things when we needed. Maybe we can continue with this approach but try to keep a well-defined structure for the primitives. For example, we could add a "http_tracker" mod from primitives that are only used in the context of HTTP trackers.

Having to depend on a big database crate when you only need one type is not very efficient.

Could we use cargo "features" to import only the primitives and keep them in their original package?

The idea is that all our packages could be independently released as-is.

I have not realized we could independently release those packages even if they are inside the same repo.

I think we can merge the PR, it's a very good improvement, but I would:

• Fix the failing tests.
• Add the state machine to the HTTP tracker too in production.
• Move the "ephemeral" function to the production code because it's a feature you can run on production. In fact, it's an end-user feature.

Moved back to #170