Skip to content

Commit 3b93db0

Browse files
committed
feat(infra): a script's curl / wget request HTTP_CALLS its Route
Shell scripts, Make recipes and CI steps download tools and call services with curl and wget; the graph had no edge for any of them, so neither what a repository fetches nor the scripts that call its own services were visible. Such a command now HTTP_CALLS the Route of each URL it states, with {via: curl | wget, method, url_path, host, line}: - the method from the flags: -X / --request / --method, a body (-d, -F, --data*, --json, --form*, wget --post-data / --post-file) POST, -T PUT, -I HEAD, -G back to GET; - the URL from constant words (VER=1.2.3 ... v${VER}), unknown parts `{}` as in a Route path -- also in an unquoted word tree-sitter-bash splits around its expansions; a URL whose host is not literal, or the value of a proxy / referer option, is no request; - never the user info, the query or the fragment: a token in a URL does not reach the graph; - the Route: path-only (__route__POST__/v1/items) when the host is a Kubernetes Service of the repository in a service DNS form, where that service's handler meets it as a code client's request does; any other host -- a download, localhost, a compose service -- keeps it (__route__GET__https://dl.k8s.io/release/{}/bin/linux/amd64/kubectl, props {method, host, external}), and route_nodes makes no path-only twin of it. Incremental indexing now drops a Route without a file that no edge touches any more -- its last caller's file changed or went. A full index never holds one; before, the Route of a deleted code client stayed behind too. Measured on 17 infrastructure repositories: 92 edges, every one correct by hand; an independent reading of the scripts and Makefiles finds 82 requests, 81 of them in the graph (the last is lost in a file tree-sitter-bash cannot parse past `[ "$v" \< "1.33" ]`). No property of any other node or edge changes. Signed-off-by: Martin Vogel <[email protected]>
1 parent bd928b3 commit 3b93db0

10 files changed

Lines changed: 572 additions & 23 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -247,7 +247,7 @@ The install script placed beside the binary is **reported, not deleted** — uni
247247
- `CALL_REFERENCE` — a callable is used at a supported reference site (for example, a direct value argument) and resolves to one exact target
248248
- `USAGE` — an identifier is used, but a unique callable target is not proven (including ambiguous or complex expressions)
249249
- `IMPORTS`, `DEFINES`, `IMPLEMENTS`, `INHERITS`
250-
- `HTTP_CALLS`, `ASYNC_CALLS` (cross-service)
250+
- `HTTP_CALLS`, `ASYNC_CALLS` (cross-service); a shell script, Make recipe or CI step running `curl` / `wget` on a URL it states (constant words resolved, unknown parts `{}`) to its `Route` with the method its flags give: a Kubernetes Service of the repository by the path, as code clients do, any other host (a download, localhost) with the host kept — the edge keeps `via`, `method`, `url_path`, `host`, never the user info or the query
251251
- `SPAWNS` — the caller starts another program (`subprocess.run`, `exec.Command`, `posix_spawn`, `Process.Start`, a shell command or Make recipe running a program by path, ...); the target is a `Process` node named after the program, or `<dynamic>` when the program is not a literal. When a shell script, Make recipe, GitHub Actions step or Prow job runs a script of the repository (`./hack/x.sh`, `bash hack/x.sh`), the target is that script's `Module` (else its `File`); `make TARGET` (also `$(MAKE)`, `-C`, `-f`) targets the rule of that target, followed through included makefiles; edges from CI carry `ci` and `job`. A Dockerfile's last stage `ENTRYPOINT` (else `CMD`) spawns what the container runs (`api: dockerfile`): the repository script or source copied into the image, the Go `main` of a binary a stage builds with `go build -o`, else a `Process`
252252
- `INFRA_MAPS` — a Kubernetes Service to each workload whose pod labels carry every `key: value` of the Service's selector, within one namespace
253253
- `CONNECTS_TO` — a Kubernetes workload or compose service to the Service its containers' environment addresses (`PRODUCT_CATALOG_SERVICE_ADDR: productcatalogservice:3550`, a URL's host, `<svc>.<ns>.svc...`, a bare name on an `*_ADDR` / `*_HOST` / `*_URL` / ... key; compose: the project's `.env` interpolated): a Kubernetes Service of the same bundle and namespace, or a compose service of the same file. The edge keeps `via: env`, `key` and `container` — never the value

‎src/graph_buffer/graph_buffer.c‎

Lines changed: 43 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1550,10 +1550,10 @@ static bool gbuf_node_touched(const cbm_gbuf_t *gb, int64_t id, const char *cons
15501550
return false;
15511551
}
15521552

1553-
int cbm_gbuf_delete_unreferenced(cbm_gbuf_t *gb, const char *label, const char *const *edge_types) {
1554-
if (!gb || !label || !edge_types || !edge_types[0]) {
1555-
return CBM_NOT_FOUND;
1556-
}
1553+
/* Delete the `label` nodes no edge of `types` (NULL-terminated) touches --
1554+
* with `fileless`, only those carrying no file; returns how many. */
1555+
static int gbuf_delete_untouched(cbm_gbuf_t *gb, const char *label, const char *const *types,
1556+
bool fileless) {
15571557
node_ptr_array_t *arr = cbm_ht_get(gb->nodes_by_label, label);
15581558
if (!arr || arr->count == 0) {
15591559
return 0;
@@ -1568,7 +1568,8 @@ int cbm_gbuf_delete_unreferenced(cbm_gbuf_t *gb, const char *label, const char *
15681568
for (int i = 0; i < arr->count; i++) {
15691569
const char *qn = arr->items[i]->qualified_name;
15701570
cbm_gbuf_node_t *node = qn ? cbm_ht_get(gb->node_by_qn, qn) : NULL;
1571-
if (node && !gbuf_node_touched(gb, node->id, edge_types)) {
1571+
if (node && !(fileless && node->file_path && node->file_path[0]) &&
1572+
!gbuf_node_touched(gb, node->id, types)) {
15721573
victims[n++] = node;
15731574
}
15741575
}
@@ -1604,6 +1605,43 @@ int cbm_gbuf_delete_unreferenced(cbm_gbuf_t *gb, const char *label, const char *
16041605
return n;
16051606
}
16061607

1608+
int cbm_gbuf_delete_unreferenced(cbm_gbuf_t *gb, const char *label, const char *const *edge_types) {
1609+
if (!gb || !label || !edge_types || !edge_types[0]) {
1610+
return CBM_NOT_FOUND;
1611+
}
1612+
return gbuf_delete_untouched(gb, label, edge_types, false);
1613+
}
1614+
1615+
typedef struct {
1616+
const char **items;
1617+
uint32_t count;
1618+
} gb_type_list_t;
1619+
1620+
static void collect_edge_type(const char *key, void *value, void *ud) {
1621+
(void)value;
1622+
gb_type_list_t *list = ud;
1623+
list->items[list->count++] = key;
1624+
}
1625+
1626+
int cbm_gbuf_delete_edgeless(cbm_gbuf_t *gb, const char *label) {
1627+
if (!gb || !label) {
1628+
return CBM_NOT_FOUND;
1629+
}
1630+
/* Edges are indexed by (node, type) only: "any edge" is an edge of any
1631+
* type the buffer holds (NULL-terminated by the calloc). */
1632+
gb_type_list_t list = {
1633+
cbm_calloc(CBM_MEM_CLASS_GBUF_INDEX,
1634+
((size_t)cbm_ht_count(gb->edges_by_type) + 1) * sizeof(*list.items)),
1635+
0};
1636+
if (!list.items) {
1637+
return CBM_NOT_FOUND;
1638+
}
1639+
cbm_ht_foreach(gb->edges_by_type, collect_edge_type, &list);
1640+
int n = gbuf_delete_untouched(gb, label, list.items, true);
1641+
cbm_free(CBM_MEM_CLASS_GBUF_INDEX, list.items);
1642+
return n;
1643+
}
1644+
16071645
int cbm_gbuf_load_from_db(cbm_gbuf_t *gb, const char *db_path, const char *project) {
16081646
if (!gb || !db_path || !project) {
16091647
return CBM_NOT_FOUND;

‎src/graph_buffer/graph_buffer.h‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,13 @@ int cbm_gbuf_delete_by_file(cbm_gbuf_t *gb, const char *file_path);
124124
* is gone, as a full index never makes them. */
125125
int cbm_gbuf_delete_unreferenced(cbm_gbuf_t *gb, const char *label, const char *const *edge_types);
126126

127+
/* Delete every `label` node that carries no file and that no edge of any
128+
* type touches; returns how many. For a label made by edges of many kinds
129+
* (Route: HTTP_CALLS, ASYNC_CALLS, HANDLES, ...): incremental indexing drops
130+
* the file-less one whose last edge went with its files, as a full index
131+
* never makes it. */
132+
int cbm_gbuf_delete_edgeless(cbm_gbuf_t *gb, const char *label);
133+
127134
/* Bulk-load all nodes and edges for a project from an existing SQLite DB
128135
* into this graph buffer. Returns 0 on success. */
129136
int cbm_gbuf_load_from_db(cbm_gbuf_t *gb, const char *db_path, const char *project);

‎src/pipeline/pass_route_nodes.c‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,17 @@ static void route_edge_visitor(const cbm_gbuf_edge_t *edge, void *userdata) {
7878
if (strcmp(edge->type, "HTTP_CALLS") != 0 && strcmp(edge->type, "ASYNC_CALLS") != 0) {
7979
return;
8080
}
81+
/* A script's request (pass_shell.c) names its host and already ends at
82+
* its Route, which keeps the host for one outside the repository: a
83+
* path-only Route made from its url_path would meet a handler it never
84+
* calls. A code client's edge names none and refreshes its Route here. */
85+
char host_buf[CBM_SZ_256];
86+
if (json_extract(edge->properties_json, "host", host_buf, sizeof(host_buf))) {
87+
const cbm_gbuf_node_t *target = cbm_gbuf_find_by_id(ctx->gb, edge->target_id);
88+
if (target && target->label && strcmp(target->label, "Route") == 0) {
89+
return;
90+
}
91+
}
8192

8293
/* Extract url_path from properties */
8394
char url_buf[CBM_SZ_512];

0 commit comments

Comments
 (0)