Skip to content

Commit bbf739a

Browse files
feat: Add robots.txt to site (apache#24333)
## Which issue does this PR close? - Closes apache#23258 ## What changes are included in this PR? There are 2 PR's for this issue. This one contains the changes for _robots.txt_ - added the robots.txt , added license header and allow rules. ## Are these changes tested? Yes. Tested by building the docs and verifying the files getting generated in the build/ ## Are there any user-facing changes? No. --------- Co-authored-by: Jeffrey Vo <[email protected]>
1 parent 110a90a commit bbf739a

3 files changed

Lines changed: 6 additions & 1 deletion

File tree

‎dev/release/rat_exclude_files.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,7 @@ datafusion/proto-common/src/generated/prost.rs
6161
.github/ISSUE_TEMPLATE/feature_request.yml
6262
.github/workflows/docs.yaml
6363
docs/source/llms.txt
64+
docs/source/robots.txt
6465
**/node_modules/*
6566
datafusion/wasmtest/pkg/*
6667
clippy.toml

‎docs/source/conf.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -112,7 +112,7 @@
112112

113113
# Copy agent-facing files (llms.txt) verbatim to the site root so they
114114
# resolve at the conventional URL `https://datafusion.apache.org/llms.txt`.
115-
html_extra_path = ["llms.txt"]
115+
html_extra_path = ["llms.txt", "robots.txt"]
116116

117117
# Base URL of the published site. Required by sphinx-sitemap to build
118118
# absolute <loc> entries in the generated sitemap.xml.

‎docs/source/robots.txt‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
User-agent: *
2+
Allow: /
3+
4+
Sitemap: https://datafusion.apache.org/sitemap.xml

0 commit comments

Comments
 (0)