Repository navigation
feat: Add robots.txt to site - #24333
Conversation
github-actions
Bot
added
the
documentation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #24333 +/- ##
==========================================
- Coverage 81.23% 81.23% -0.01%
==========================================
Files 1111 1111
Lines 390208 390208
Branches 390208 390208
==========================================
- Hits 316991 316983 -8
- Misses 54592 54594 +2
- Partials 18625 18631 +6 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| @@ -0,0 +1,19 @@ | |||
| # Licensed to the Apache Software Foundation (ASF) under one | |||
There was a problem hiding this comment.
not sure if we're supposed to have this license txt in a robots.txt?
There was a problem hiding this comment.
I added this because , '#' will be ignored by crawlers also I had to add the robots.txt file name to rat_exclude_ files.txt. Im happy to remove it , fine either way .
There was a problem hiding this comment.
for reference, airflow and spark dont include the license
There was a problem hiding this comment.
Thanks. I have added a commit removing license and following what llms.txt followed.
github-actions
Bot
added
the
development-process
Jefffrey
changed the title
|
wooohooo. first commit . @Jefffrey Thanks for the help. |
Which issue does this PR close?
What changes are included in this PR?
There are 2 PR's for this issue. This one contains the changes for robots.txt
Are these changes tested?
Yes. Tested by building the docs and verifying the files getting generated in the build/
Are there any user-facing changes?
No.