{"@attributes":{"version":"2.0"},"channel":{"title":"elmiko's notes","description":"Notes from the front lines of development","link":"https:\/\/notes.elmiko.dev","item":[{"title":"KubeCon Europe 2026 Retrospective","description":"<p>As is becoming tradition, here is my bi-annual update to this blog with my thoughts and perceptions\nfrom another KubeCon. This time for the Spring 2026 KubeCon in Europe. TL;DR Lots of\nexcitement for Kubernetes, more people than Atlanta, AI fatigue is setting in, scheduling is getting\nbigger, and as always excellent hallway track conversations.<\/p>\n\n<p>KubeCon was held at the RAI center in Amsterdam Netherlands this year, the same venue we were at for\nKubeCon in 2023. The RAI is a large venue and KubeCon filled it nicely, but not overly so. This year\nmy schedule was quite packed with the Maintainer Summit happening on Sunday, the Workload Aware Scheduling\nDesign Summit on Monday, and then KubeCon proper from Tuesday through Thursday. I spoke on two panels\nduring KubeCon and was generally quite exhausted by the end of the week, but I had a great time and KubeCon\ncontinues to be a great investment of time and energy for me.<\/p>\n\n<p>As is becoming a sad tradition, the political world is lighting itself on fire as KubeCon is happening, with\nthe war in Iran being a constant note in the background. Thankfully I did not experience any travel delays\ndue to the conflict or the continued government shutdowns in the United States. From what I could gather, the\nattendance seemed higher than in Atlanta and I have to attribute some of that to people\u2019s comfort with\ntravelling to Europe.<\/p>\n\n<p><img src=\"\/img\/kubecon-eu-2026-selfie.jpg\" class=\"img-responsive center-block\" alt=\"Selfie out front of RAI\" \/><\/p>\n\n<h2 id=\"kubernetes-maintainer-summit-sunday-march-22\">Kubernetes Maintainer Summit, Sunday March 22<\/h2>\n\n<p>I started the week with my favorite activity, the Maintainer Summit. I knew going in that the summit was sold out\nthis year, and it sure felt like it. Every session was packed, there were many hallway conversations happening, and\nthere was generally a great vibe in the air. Folks were excited to see each other, talk about their projects, and\nstart doing some face-to-face collaboration. As in the previous edition, the conference committee really listened\nto the feedback and we once again had hot meals for lunch and more unconference sessions (although it\u2019s always a\nsort of mad scramble to get the sessions in and voted on before the afternoon).<\/p>\n\n<p>After the keynotes, I attended the <a href=\"https:\/\/maintainersummiteu2026.sched.com\/event\/2EWeU\/ask-the-experts-kubernetes-steering-committee-kat-cosgrove-minimus-maciej-szulik-defense-unicorns-antonio-ojea-google\">Ask the Experts: Kubernetes Steering Committee<\/a> session. I\u2019ve been\nkeenly curious about the steering committee and these sessions really help to cement my understanding and interest.\nAs might be expected there were many questions for the representatives about the use of AI technologies in the\nKubernetes development process. I was encouraged by the responses to the questions about AI, in general protecting the humans is\nof paramount priority. To support this, there must always be a human in the chain of responsibility when evaluating\nissues and pull requests. Further, there was acknowledgement of more and more projects being inundated with generative\nLLM contributions. These contributions don\u2019t just come in the form of code, they also appear as comments and desciptions\nin the various communications channels. There is broad support for any human who feels that they are being\noverwhelmed by generative content. I was happy to see such a strong focus on the human element.<\/p>\n\n<p><img src=\"\/img\/kubecon-eu-2026-capi.jpg\" class=\"img-responsive center-block\" alt=\"Cluster API meetup\" \/><\/p>\n\n<p>I have been a contributor to the Cluster API project for many years now, mostly as the maintainer for the Cluster\nAutoscaler and Karpenter providers, so it was a natural fit that I would attend the <a href=\"https:\/\/maintainersummiteu2026.sched.com\/event\/2EWev\/project-meeting-cluster-api\">Cluster API Project Meeting<\/a>\nnext. This was a great discussion with the other maintainers and community members where we talked a lot about\nCI and a kubetest2 deployer for Cluster API, and also about the low level mechanics of node joining and how that might\nbecome more structured in the future. We talked about what a \u201cconformance\u201d with Cluster API would mean for various\nproviders and how that relates to the API contracts that the project defines for provider implementers. All in all this\nwas a great discussion and I always love getting a chance to hang out with the other Cluster API maintainers.<\/p>\n\n<p><img src=\"\/img\/kubecon-eu-2026-unconference.jpg\" class=\"img-responsive center-block\" alt=\"Maintainer summit unconference voting\" \/><\/p>\n\n<p>Next up on my schedule, after lunch, were the unconference sessions. There was a little confusion about these sessions\nas we vote on them the day of the summit and then the conference committee has to plan where and when they will be.<\/p>\n\n<p>After a few hiccups, I had found a popular session on <a href=\"https:\/\/maintainersummiteu2026.sched.com\/event\/2JTp2\/unconference-session-how-should-maintainers-navigate-and-review-ai-based-prs\">How Should Maintainers Navigate (and Review) AI-based PRs?<\/a>.\nThis session had a lot of great back and forth between the attendees about how we handle the flood of AI-based\ncontributions that are hitting CNCF projects. There were many people who shared stories about how their projects are being\naffected and it is clear that the focus of contributions is not even across the CNCF landscape. Some of the highlights\nfrom this session were:<\/p>\n<ul>\n  <li>How to handle the honesty issue, are people being truthful about their use of AI?<\/li>\n  <li>Security vulnerabilities introduced by bots, and conversely the attention gained from bug bounty programs.<\/li>\n  <li>Encouraging people to speak in their own voice instead of using an AI for translation.<\/li>\n  <li>Protecting people\u2019s time from being focused on overly verbose contributions with low value.<\/li>\n  <li>The possibility of using AI-based tools to help with the volume of PRs.<\/li>\n<\/ul>\n\n<p>The discussion was very lively and I was happy to see that many of the points made by the steering committee were being\nreinforced organically by the community. Many people are still learning how to navigate this new world of software\ndevelopment. Some people have had great success and others have experienced terrible failures, but it is clear that as\na community we want to work towards a world where these tools can be used safely and consciously by contributors.<\/p>\n\n<p><img src=\"\/img\/kubecon-eu-2026-jack-kuba.jpg\" class=\"img-responsive center-block\" alt=\"Maintainer summit unconference voting\" \/><\/p>\n\n<p>As I mentioned previously, I have been maintaining the Cluster API provider for the Cluster Autoscaler for several years now.\nSo I was not going to miss Jack and Kuba\u2019s session on <a href=\"https:\/\/maintainersummiteu2026.sched.com\/event\/2EWf1\/cluster-autoscaler-evolution-kuba-tuznik-google-jack-francis-microsoft\">Cluster Autoscaler Evolution<\/a>. In this session they described\nhow the Cluster Autoscaler is going to transform to a code model that more closely aligns with how Karpenter is distributed,\nnamely as a library. I think this is an exciting step forward for the autoscaling community as it will allow the maintainers\nto separate the concerns of specific providers from the core behavior. In addition to the library migration there are also\nseveral new features that will be coming to the autoscaler that I think will be eagerly awaited by the community, for one\nthe ability to defragement clusters. More on these features to come in the months ahead.<\/p>\n\n<p>With my SIG Cloud Provider hat on, I have been following the Node Lifecycle Working Group\u2019s efforts since it began last year.\nWhile I\u2019m not yet convinced there is much work needed to be done by the SIG, I do think it\u2019s important to keep touch on\nthis work as there might be ways that the SIG can help support the new interactions being proposed. To support that, I\nwent to the unconference session <a href=\"https:\/\/maintainersummiteu2026.sched.com\/event\/2EWhP\/unconference-session-node-lifecycle-state-needs-a-real-api\">Node Lifecycle State Needs a Real API<\/a>, which was a good discussion from the\nmaintainers of the working group about how we might identify the states needed to improve lifecycle awareness for nodes. It\u2019s\na complex issue and I have a feel it will be solved incrementally by addressing the sub-problems, such as eviction, first.<\/p>\n\n<p>To wrap up the \u201cbusiness\u201d part of my maintainer summit day (as opposed to the after hours party), I attended the SIG Autoscaling\nmeetup. I was a little late to this session as I had gotten into a deep hallway track discussion about improved ignition\nintegration in Cluster API. Regardless, I arrived in time to hear the continued discussion about the new architecture for the\nautoscaler. This initiative is going to be big for this year and I expect that by the next summit in Salt Lake City much of the\ninitial work will be complete. Another big topic of discussion for the autoscaling community is the decoupling of the scheduler\nfrom the core of the Cluster Autoscaler and Karpenter projects. There is a new API being developed to help with the workload and\ntopology aware scheduling that will help this effort and I am very curious to see how it develops.<\/p>\n\n<p>After the end keynotes, we all gathered for the <a href=\"https:\/\/www.flickr.com\/photos\/143247548@N03\/55163324677\/in\/album-72177720332630036\">maintainer summit group photo<\/a> and then on to the socializing with refreshements!\nThe wandering photographers managed to catch Justin and I deep in a conversation about AI workflows and what the future might\nhold.<\/p>\n\n<p><img src=\"\/img\/kubecon-eu-2026-justin-me.jpg\" class=\"img-responsive center-block\" alt=\"Justin and me talking\" \/><\/p>\n\n<h2 id=\"workload-aware-scheduling-design-summit-at-google-monday-march-23\">Workload Aware Scheduling Design Summit at Google, Monday March 23<\/h2>\n\n<p>Coming close on the heels of the Dynamic Resource Allocation (DRA) designs that have helped to improve the state\nof advanced hardware utilization in Kubernetes, comes the next big design work: workload aware scheduling. I am thankful to\nJohn Belamaric and Wojciech Tyczynski from Google who organized this design summit where many SIG tech leads and\nchairs, as well as interested parties, were invited to participate in architecting the future of Kubernetes scheduling.\nThis design summit was focused on the myriad problems that need to be addressed so that we can improve the story\naround workload awareness during scheduling.<\/p>\n\n<p><img src=\"\/img\/kubecon-eu-2026-was-summit.jpg\" class=\"img-responsive center-block\" alt=\"John and Wojciech\" \/><\/p>\n\n<p>Workload aware scheduling speaks to the notion that there are workloads which have greater scheduling constraints than Kubernetes currently\nsupport today. Think about applications where network topology and rack location have an impact on the results. For example, the\nmachine learning workloads that we see today can often request to have hardware that shares a network segment or in the most\ndemanding cases might need multiple racks of hardware physically located near each other. The design summit was an opportunity to\nhave people in one place to discuss the various issues around how the Kubernetes community can deliver these features for users.<\/p>\n\n<p>The day was separted into a few sessions and was organized in a very unconference fashion. There were two general themes that emerged\nfrom the topics: scheduling and autoscaling. It was difficult for me to pick which tracks I wanted to participate with since I am\nrepresenting SIG Cloud Provider to see where we might help with lower level provider interactions, but I also want to keep track of what \nis happening with the autoscaler as I am keenly interested in ensuring that the Cluster API intergrations are as full featured as possible.\nIn the end, I chose to participate in the scheduling tracks as there were many autoscaling experts in the room and I generally was curious to\nsee what solutions they agreed on. I was rewarded with some deep discussions around the mechanics of how workload aware scheduling will\nneed to work from the hardware pespective. This ended up being very fruitful for me as I walked away with some clear ideas about how\nSIG Cloud Provider might support activities where scheduling plugins will need to make calls to the underlying infrastructure provider\nto learn about the hardware configurations and topologies.<\/p>\n\n<p>For anyone interested in this exciting new area of development, I would start by reviewing the DRA related mechanisms that exist in\nKubernetes today. These primitives are inspiring how the future of workload aware scheduling will be architected. Then I would study\nthe <a href=\"https:\/\/github.com\/kubernetes\/enhancements\/issues?q=label%3Aarea%2Fworkload-aware\">workload aware related KEPs that are currently under review<\/a>.<\/p>\n\n<p>Lastly, I am grateful for the opportunity to participate in this design summit. Although I don\u2019t think I had much to contribute, I\ncertainly learned a great deal and will hopefully be able to support the effort through SIG Cloud Provider. I\u2019ve heard that quote\nabout \u201cif you are the smartest person in the room, find a new room\u201d, and I felt humbled by the assembled brain power at the summit.\nSome of the brightest and sharpest minds in the Kubernetes community were there and it was inspiring to watch the thoughtful discussions\nthat arose.<\/p>\n\n<h2 id=\"kubecon-day-1-tuesday-march-24\">KubeCon Day 1, Tuesday March 24<\/h2>\n\n<p>After two solid days of work already, KubeCon proper starts. I briefly cruised the keynotes and the early talks, but my mind was focused\non the two big activities I had for day 1: the <a href=\"https:\/\/kccnceu2026.sched.com\/event\/2ITlD\/kubernetes-contribution-101\">Kubernetes Contribution 101<\/a> session, and the panel\n<a href=\"https:\/\/kccnceu2026.sched.com\/event\/2EoKz\/from-static-tokens-to-attestation-the-evolution-of-secure-node-joining-ciprian-hacman-jack-francis-microsoft-michael-mccune-josephine-pfeiffer-red-hat-justin-santa-barbara-google\">From Static Tokens to Attestation: The Evolution of Secure Node Joining<\/a>. I was a participant in both activities and I was quite\nexcited for the opportunities.<\/p>\n\n<p><img src=\"\/img\/kubecon-eu-2026-communityhub.jpg\" class=\"img-responsive center-block\" alt=\"Community hub placard\" \/><\/p>\n\n<p>The Community Hub is one of my favorite parts of KubeCon and when the call went out for SIG chairs and tech leads to participate in the\nContribution 101 session I jumped at the chance. It was a nearly two hour session where we had a presentation about the mechanics of\ncontribution and the landscape of the CNCF projects, and then a question and answer portion where the SIG leads got to interact with\nthe audience. It was truly inspiring to see so many people interested in contributing and asking thoughtful questions about how to get\ninvolved. Whenever I have time, I will always participate in these sessions as I love helping new folks get a boost into open source. Passing\non the open source values and ethics is a large part of what I do these days and I want to ensure that we have a healthy community for\nfuture generations.<\/p>\n\n<p>After the 101 session I had to find the panel where I was a particpant, unfortunately the room was on the opposite end of the convention\ncenter and I had to make haste. Thankfully, I made it with time to spare. The panel went great and we had about 150 people in the room to\nhear us talk about a possible future for secure booting and attestation in Kubernetes. In some respects this is new territory for the\nKubernetes community to tackle, although there are bespoke implementations of this style of booting, we would like to develop best practices\nthat the community as a whole can rely on. This will be important if we want to include secure boot activities in projects like kOps and\nCluster API. While there is more work to be done, we had an excellent conversation and got many good ideas from interacting with the audience.\nI\u2019m hopeful that some of the work I\u2019m doing with the Cluster API community to improve ignition support will also help with the workflows\naround attestation.<\/p>\n\n<p>I spent the remainder of my time on day 1 cruising the solutions showcase to see what people were talking about with respect to the product\nside of Kubernetes. The solutions showcase was a good time and there was plenty of room and natural light penetrating the room, making it feel\nmuch better to walk around and get lost amongst the technology. As always, LEGO raffles were huge. I also noted the return of the curling\narena.<\/p>\n\n<p><img src=\"\/img\/kubecon-eu-2026-curling.jpg\" class=\"img-responsive center-block\" alt=\"Curling Skills Arean signage\" \/><\/p>\n\n<h2 id=\"kubecon-day-2-wednesday-march-25\">KubeCon Day 2, Wednesday March 25<\/h2>\n\n<p>In many ways, day 2 was a repeat of day 1 for me. I had plans to participate in the <a href=\"https:\/\/kccnceu2026.sched.com\/event\/2H66n\/kubernetes-meet-+-greet\">Kubernetes Meet + Greet<\/a> and then join the\n<a href=\"https:\/\/kccnceu2026.sched.com\/event\/2EF68\/how-will-customized-kubernetes-distributions-work-for-you-a-discussion-on-options-and-use-cases-michael-mccune-joel-speed-red-hat-bridget-kromhout-microsoft-jesse-butler-aws-bowei-du-google\">How Will Customized Kubernetes Distributions Work for You? A Discussion on Options and Use Cases<\/a> panel. It should go\nwithout saying that I love the meet and greet. It\u2019s been a high point for me for several KubeCons now and I volunteered to work the\nfirst hour as a greeter, and then would join the second hour in my SIG Cloud Provider capacity to talk with attendees.<\/p>\n\n<p>The meet and greet went great, I was able to meet many new folks and help them to find the communities where they could learn and connect\nwith others. After my official duties had ended, I got a good slice of time to talk with people about SIG Cloud Provider and how they could\nget involved. One inspiring discussion that came out of this was with a gentleman who had done documentation work across three languages!\nHe was connected with developer communities in Vietnam and was curious how he could connect those communities with the wider Kubernetes\ncommunity. It was great to talk about how Vietnamese cloud providers will be able to join our efforts and really how they can gain\nthe benefits of the common cloud provider framework we have developed. It made me feel good to know that the work we are doing in the\nopen source can truly reach all communities so that they can join the great activity that is happening around Kubernetes.<\/p>\n\n<p>I had to run at the end of the meet and greet to make the second panel I was doing for this KubeCon. I made it back to the panel room with a\nlittle time to spare, but as it turned out the panel before ours (which was on Node Lifecycle APIs) was packed to the rafters and we had to wait\noutside for the room to empty.<\/p>\n\n<p><img src=\"\/img\/kubecon-eu-2026-cloudprovider-panel.jpg\" class=\"img-responsive center-block\" alt=\"Cloud Provider panel\" \/><\/p>\n\n<p>The panel went off well and we had about 150 people show up to hear us talk about the idea of Kubernetes distributions. I wasn\u2019t sure\nwhat to expect when we proposed this panel. SIG Cloud Provider has been interested in this notion to help us achieve some testing goals, but\nalso to work towards the future where there is a clear method for cloud provider specific bits to be included with a Kubernetes installation.\nWhether that installation happens from kOps, Cluster API, or some other tooling, we would like there to be a common set of guides to follow.\nI was touched by how many people approached me afterwards to thank the SIG for diving in to this topic. We\u2019ll see how things progress.<\/p>\n\n<p>In a similar fashion as day 1, I spent some time at the end of the day in the solutions showcase checking out the project pavilion and connecting\nwith friends at different companies who were working booth duty.<\/p>\n\n<h2 id=\"kubecon-day-3-thursday-march-26\">KubeCon Day 3, Thursday March 26<\/h2>\n\n<p><img src=\"\/img\/kubecon-eu-2026-bootc.jpg\" class=\"img-responsive center-block\" alt=\"Bootc demo\" \/><\/p>\n\n<p>By Thursday my tanks were running on near empty, but I wanted to get around to the project pavilion again to check out the projects I was less\nfamiliar with, and I was rewarded richly. I spent time talking with my colleague Thilo at the Flatcar Linux booth. We are attempting to improve\nthe state of ignition in Cluster API and I\u2019m hopeful that we\u2019ve made some solid designs for work that we can do this year. I also spent time talking\nwith the Bootc project team as well. <a href=\"https:\/\/bootc.dev\/bootc\/\">Bootc<\/a> is a really cool project that unlocks a great deal of potential for managing and upgrading\nostree style operating system images. I\u2019m sure it can do so much more than that, but the demo that my colleagues from Red Hat gave showed how\nyou could perform rolling upgrades and downgrades to an application embedded as part of an ostree image. I am absolutely planning to setup a\nhome lab for playing around with bootc, especially since I think it will help with the ignition\/Cluster API work I\u2019d like to do.<\/p>\n\n<h3 id=\"a-note-on-ai\">A note on AI<\/h3>\n\n<p><img src=\"\/img\/kubecon-eu-2026-ai.jpg\" class=\"img-responsive center-block\" alt=\"ai doggo\" \/><\/p>\n\n<p>I would be neglectful if I didn\u2019t mention the presence of AI at KubeCon. There were plenty of talks and demonstrations with people showing off\nhow they are using AI to build value and also integrate AI into their products. Additionally there was no shortage of talks about how AI\nis affecting, and will continue to influence, the software development process. I found the discussion good and the amount of AI related talks\nonly a little oppressive. One thing I noted that is increasing from the hallway track perspective is that many people are getting tired of\nseeing AI related talks and want to see more technical Kubernetes information. I saw several non-AI sessions that were packed to capacity,\nand my informal discussions with people seemed to indicate a weariness of AI and also a desire to find spaces where AI was not being\ndiscussed. Hopefully this is a sign that the community is large enough that we need more equal represenatation of ideas at KubeCon. AI has\nmade its mark and will continue to evolve, but it is not the Ur-solution for all problems.<\/p>\n\n<h2 id=\"thoughts-and-takeaways\">Thoughts and takeaways<\/h2>\n\n<p>KubeCon Amsterdam was a tremendous success for me. I had an excellent time connecting with friends and peers from across the industry\nand got to participate in some amazing discussions. I am optimistic about the future of Kubernetes and I look forward to how the\ncommunity will continue to grow and evolve.<\/p>\n\n<p>I am hesitant about the continued usage of AI in our software development processes. I can see the benefits but I also see the harm that\nit can do to the people involved in the process. I think the most important thing for us to remember is that the people who make up this\ncommunity are the most important element, and we should ensure that regardless of the technological changes happening we continue to \nrespect and protect the community members.<\/p>\n\n<p>In brief, here are some thoughts I had:<\/p>\n\n<ul>\n  <li>Scheduling continues to be a giant topic and it is only becoming more complex.<\/li>\n  <li>AI continues to grow and we are now dealing with how to incorporate this style of development into our open source processes.<\/li>\n  <li>Digital sovereignty and on-premises computing are growing in popularity once again, Kubernetes represents a game changer for people wanting open source solutions that they can own.<\/li>\n  <li>Even with the global economy in an uncertain state, there continues to be investment in cloud services and platform engineering.<\/li>\n  <li>Focus on community health and safety has never been higher, and I fully endorse this activity.<\/li>\n<\/ul>\n\n<p>Whelp, that\u2019s about it for my thoughts on this retrospective. I hope I can continue to attend KubeCons and be part of this community.\nIt brings me great joy and satisfaction to know that what we are building, in the open, can help make the world a better place. It seems\nchallenging to remember this given the state of the world these days, but I count myself as an eternal optimist in this respect. If you\nmade it this far, thank you, be safe out there, and as always happy hacking!<\/p>\n\n<p><img src=\"\/img\/kubecon-eu-2026-outro.jpg\" class=\"img-responsive center-block\" alt=\"rainy RAI\" \/><\/p>\n\n","pubDate":"Wed, 08 Apr 2026 00:00:00 +0000","link":"https:\/\/notes.elmiko.dev\/2026\/04\/08\/kubecon-eu-2026-retrospective.html","guid":"https:\/\/notes.elmiko.dev\/2026\/04\/08\/kubecon-eu-2026-retrospective.html"},{"title":"KubeCon North America 2025 Retrospective","description":"<p>Well, it\u2019s been about a year since I updated this blog, so why not get back into things by giving my\nretrospective on this fall\u2019s KubeCon North America. TL;DR fewer people, slightly less AI hype, and\nloads of good conversations.<\/p>\n\n<p>KubeCon North America for 2025 was hosted in Atlanta Georgia in the United States. The offical \u201cKubeCon\u201d\nwas on the 11th through the 13th of November, with the CNCF co-located events happening on Monday the\n10th, and the Kubernetes Maintainer Summit happening on Sunday the 9th. I was lucky enough to travel to\nAtlanta for all the events, but the start of my journey was slightly delayed due to the government shutdown\nin the United States. It\u2019s worth mentioning the political activity in the United States, without going into\ntoo much detail, as I believe the current political climate contributed to my perception of a smaller population\nat this KubeCon.<\/p>\n\n<p>But, even though my flight was delayed by 6 hours, I was able to finally make it to Atlanta. I heard stories\nfrom many friends and colleagues who had flights delayed, and in some cases cancelled, which led to a fun situation\nwhere most of us were asking each other \u201chow long was your delay?\u201d as we met at the conference.<\/p>\n\n<p><img src=\"\/img\/kubecon-na-2025-selfie.jpg\" class=\"img-responsive center-block\" alt=\"Selfie at Red Hat booth\" \/><\/p>\n\n<h2 id=\"kubernetes-maintainer-summit-sunday-november-9\">Kubernetes Maintainer Summit, Sunday November 9<\/h2>\n\n<p>As is tradition, maintainer summit is my favorite part of the conference. This year, we started with the summit\noccurring before any other event. There was a small gathering at the conference hall and we had a day of\nconversations, presentations, and unconference sessions; this year including a hot meal for lunch! (thank you con team &lt;3)<\/p>\n\n<p>We started with the keynotes and getting warmed up for the day, it\u2019s always nice starting out with some laughs\nand looking around the room to see who you might recognize. As I noted, the attendance seemed a little lower\nthan previous summits and some of that was due to the travel challenges, and the rest I believe was due to folks\nnot wanting to engage with the United States customs and border control.<\/p>\n\n<p>I started my summit by attending the <a href=\"https:\/\/sched.co\/2B5Lm\">TAG Workshop<\/a> hosted by Karena Angell, Mario Fahlandt, Brandt Keller,\nand Dylan Page. TAG stands for \u201cTechnical Advisory Groups\u201d and this was my first experience learning how these groups\noperate and what function they serve in the community. I was glad to learn that they help with cross-project efforts\nand in areas where greater coordination is needed for initiatives that will affect several technology groups. A large\npart of the reason I wanted to learn more about the TAGs is that <a href=\"https:\/\/www.youtube.com\/watch?v=WeWQqQM6kjM\">the testing work I am interested in doing<\/a>\nis going to involve some level of standardizing around how Kubernetes is deployed for different cloud providers, something\nwe have been calling \u201cdistributions\u201d in SIG Cloud Provider.<\/p>\n\n<p>As happens at these events, the discussion I was having about kubernetes distributions continued past the end time for\nthe TAG Workshop and I ended up spending my next session in the <em>hallway track<\/em> talking with folks about the idea. Basically,\nit would be very convenient for cloud provider testing to have the notion of a <em>distribution<\/em> of kubernetes. This would mean\nincluding a reference topology (eg 3 control plane nodes instead of 1, etc) and also including provider-specific components\ndepending on the distribution. For example, on OpenStack, the kubernetes distribution would include the cloud controller manager\nfor OpenStack, plus any other storage or networking components required for that platform. This would help us in arriving\nat a destination where tests could more easily select the platform and also the components that need testing. In the end\nwe will have provider agnostic tests that can operate on any provider, but which will also exercise provider-specific behavior\nthrough provider interfaces.<\/p>\n\n<p>After lunch, I went to some of the unconference sessions starting with\n<a href=\"https:\/\/sched.co\/28aDq\">Better Together, Strengthening Inter-Project Collaboration &amp; Developer Experience Across the CNCF Ecosystem<\/a>\nproposed by Yacine Kheddache and Colin Griffin. This was a fun session where we talked about how to better share\ninformation in the CNCF community, especially for the purposes of helping increase cross-project collaboration and knowledge.\nAs with many unconference sessions, the discussion really got going once the time was almost out, but I had a revelation while\nparticipating in the discussion. Essentially, we could use more project manager community members. This is something I have\nstruggled with in the past, but it was purely based on people reaching out to me for mentorship as project managers. I never\nhad a good answer for this style of collaboration in open source projects, but in this Kubernetes community I am now seeing\na clear place for these roles. In the future I am going to guide project managers to become more involved in the <em>social glue<\/em>\nthat holds our community together. I think there is a strong place for contribution in the form of people who can go between\nprojects to help advocate for projects needs in differing venues. <em>Perhaps another blog post topic for the future ;)<\/em><\/p>\n\n<p>The last session I attended was the one on <a href=\"https:\/\/sched.co\/28aE2\">Making the Kubernetes CI\/Infra better<\/a>. Now, this wasn\u2019t technically the\nlast session of the day, but the discussion I got into towards the end kept going into the hallway track, and by the time we\nwere done it was time for the group picture. This session was great though, especially given my desire to chase the idea of Kubernetes\ndistributions. I learned a lot in this session and came away with a new excitement about building a Cluster API deployer for\n<a href=\"https:\/\/github.com\/kubernetes-sigs\/kubetest2\">kubetest2<\/a>. We\u2019ll see where it goes, but I\u2019m optimistic about the future of Kubernetes cloud provider testing.<\/p>\n\n<p>After a day of visiting with colleagues and talking about the future of Kubernetes we adjourned for some much needed relaxation.<\/p>\n\n<h2 id=\"openshift-commons-and-co-located-events-monday-november-10\">OpenShift Commons and co-located events, Monday November 10<\/h2>\n\n<p>It\u2019s going to be a big week, so I need to start with a solid breakfast. We don\u2019t have Waffle House in my state (Michigan), and\nit is a national institution so I had to visit. <em>(I actually went almost every day XD)<\/em>\n<img src=\"\/img\/kubecon-na-2025-wh.jpg\" class=\"img-responsive center-block\" alt=\"Waffle House\" \/><\/p>\n\n<p>On Monday, I had the pleasure of visiting <a href=\"https:\/\/commons.openshift.org\/\">OpenShift Commons<\/a> as an attendee instead of a speaker or employee, which\nwas a change from previous years. I love the Commons and as a Red Hat event I always feel a little like a celebrity when\nI visit. I was able to spend some face time with colleagues and even got to talk with a customer or two. It is very gratifying\nto hear about our customers\u2019 journeys and how our efforts become solutions for them.<\/p>\n\n<p>After spending the morning at Commons, I returned to the convention center to watch my teammate Mansi Kulkarni give\na talk on <a href=\"https:\/\/sched.co\/28D71\">Windows Container Monitoring Demystified: OpenTelemetry in Action<\/a> with her peer Ritika Gupta. As\nI don\u2019t normally work on this area of Kubernetes, it was interesting to hear how Windows containers work and also\nhow they are monitored by users. It\u2019s a wild world combining container technology with the Windows kernel.<\/p>\n\n<p>I didn\u2019t have a ton going on Monday aside from catching up with folks, so I spent some time reading code and planning\nfor KubeCon proper.<\/p>\n\n<h2 id=\"kubecon-day-1-tuesday-november-11\">KubeCon Day 1, Tuesday November 11<\/h2>\n\n<p>After the keynotes, I availed myself of a walk through the exhibitor hall and then found some talks to watch. I also\nstopped many times in the hallway track to catch up with folks I had not seen in several months.<\/p>\n\n<p>One of the big talks on Tuesday was Corey Quinn\u2019s <a href=\"https:\/\/sched.co\/27FVz\">The Myth of Portability: Why Your Cloud Native App Is Married To Your Provider<\/a>.\nI was only a little familiar with Corey\u2019s work and I was not at all ready for his presentation style. This talk was well polished, and\nwell delivered. Corey has a unique style that trends more towards comedy than analysis. His message was astute though, about the\nintracacies and challenges of delivering applications in hybrid environments and what it means to make an application truly portable\nacross clouds. I found his analysis of the problem to be spot on, his solutions left me wanting more though, and his delivery was downright\nmean-spirited. While he made good points, I couldn\u2019t help but feel insulted as he emphasized again and again how the work of myself and\nmy peers was \u201cshit\u201d.<\/p>\n\n<p>Thankfully, after the mean-spirited pillory of the previous talk, I was delighted to see Taylor Dolezal and Erica Hughberg\u2019s talk\n<a href=\"https:\/\/sched.co\/27FVV\">The Missing Manual for Open Source Community Sustainability<\/a>. I found this talk energizing and informative about how we\nbuild more sustainable processes into our communities. I really enjoyed how Taylor and Erica broke down the personas within software\ncommunities, and then provided techniques for engaging with those people. I left with a full page of notes from this session.<\/p>\n\n<p>The last session I attended on Tuesday was <a href=\"https:\/\/sched.co\/27NmS\">Beyond the Code: How the Kubernetes Steering Committee Tackles the Hard, Non-Technical Problems<\/a>\nhosted by Antonio Ojea, Benjamin Elder, and Maciej Szulik. I have been curious about the steering committee and how they work; this\nsession delivered exactly what I desired. I learned a great deal about how the committee is structured in relation to the SIGs and WGs,\nand also how they operate. It was nice to hear stories from the current committee members about how they ran for election and what the\nexperience has been like for them. Also, very cool to hear how the kubernetes community keeps itself healthy and on-track.<\/p>\n\n<p><img src=\"\/img\/kubecon-na-2025-powertrio.jpg\" class=\"img-responsive center-block\" alt=\"Antonio, Ben, and Maciej\" \/>\nAntonio, Ben, and Maciej, what a kubernetes power trio!<\/p>\n\n<h2 id=\"kubecon-day-2-wednesday-november-12\">KubeCon Day 2, Wednesday November 12<\/h2>\n\n<p>Wednesday was a big day for me as I was part of a talk, so I spent the morning preparing. At 11:30am it was time for us to present\n<a href=\"https:\/\/sched.co\/27FZc\">Maximizing Global Potential: Cost-Optimized, High-Availability Workloads Across Regions<\/a>. I joined Praseeda Sathaye and\nJingkang Jiang, with shoutouts to Wei Jiang, to present this talk and we had an absolute blast. We talked about how kubernetes can\nbe deployed globally across regions and providers to deliver highly available infrastructure. We demonstrated how the Karmada,\nCluster API, and Karpenter projects can be utilized to build a single multi-cluster workload pipeline. We had about 60-70 people in the room and got\nsome good questions.<\/p>\n\n<p><img src=\"\/img\/kubecon-na-2025-ourtalk.jpg\" class=\"img-responsive center-block\" alt=\"JK, me, Praseeda\" \/>\nJingkang, myself, and Praseeda, another power trio!<\/p>\n\n<p>After our talk, I quickly moved to the <a href=\"https:\/\/sched.co\/28xeb\">SIG\/WG Meet &amp; Greet<\/a>. The meet and greet is another of my favorite activities at\nKubeCon. It\u2019s a great chance to hang out with maintainers and learn about kubernetes and the community, and to pick people\u2019s\nbrains about what is next, or how to get involved. I absolutely recommend it to anyone who wants to learn more about the\nmaintainer community.<\/p>\n\n<p><img src=\"\/img\/kubecon-na-2025-mng.jpg\" class=\"img-responsive center-block\" alt=\"SIG meet n greet\" \/>\nIt was so good that the CNCF photographers caught us! (<a href=\"https:\/\/www.flickr.com\/photos\/143247548@N03\/54921155691\/in\/album-72177720330018728\">source<\/a>)<\/p>\n\n<p>I was talking with Josh Berkus at the end of the meet and greet and we both noted how the line for the puppy petting event\nwas very long and also right outside the meet and greet. This could perhaps be a good technique for bringing more folks to the meet\nand greet next time. Just put the puppy pit at the back of the meet and greet. XD<\/p>\n\n<p><img src=\"\/img\/kubecon-na-2025-puppy.jpg\" class=\"img-responsive center-block\" alt=\"Puppy pit at kubecon\" \/><\/p>\n\n<p>Next it was on to see my colleague Jose Valdes co-present with Mark Rosetti on the <a href=\"https:\/\/sched.co\/27NoC\">Kubernetes SIG Windows Updates<\/a>. As I\nnoted earlier, I don\u2019t normally work on Windows (although I did in a previous life), but I learned some cool stuff in this presentation\nand I got to support my teammate. I find it interesting to learn how Windows is able to meet the OCI standard through its various\nprocess APIs. I\u2019m not sure I\u2019m ready to return to that world (I\u2019m not), but I enjoyed learning more about this corner of the\nkubernetes ecosystem.<\/p>\n\n<p>After all that, I still had some gas left in the tanks and I was eagerly waiting for Justin Santa Barbara and Ciprian Hacman\u2019s talk\n<a href=\"https:\/\/sched.co\/27Nlp\">The Next Decoupling: From Monolithic Cluster, To Control-Plane With Nodes<\/a>. In this talk, Justin and Ciprian were\ndiscussing some experimentation they would like to do in <a href=\"https:\/\/github.com\/kubernetes\/kops\">kOps<\/a> to add more support for Cluster API and Karpenter. I find these\nideas to be exciting and a good path for improving our testing efforts across platforms. It filled my head with more\nthoughts about extending kubetest2 and building a proof of concept on top of their work. Also, they produced a logo for Karpenter\nthat I absolutely adore, and I want to make it the Karpenter Cluster API logo:<\/p>\n\n<p><img src=\"\/img\/kubecon-na-2025-karpcapi.jpg\" class=\"img-responsive center-block\" alt=\"Alternate karpenter logo\" \/><\/p>\n\n<p>wow, what a Wednesday!<\/p>\n\n<h2 id=\"kubecon-day-3-thursday-november-13\">KubeCon Day 3, Thursday November 13<\/h2>\n\n<p>By Thursday I was nearly completely wiped out, but I was part of a panel that would be the last session of the day. So, I pushed forward\nand made some connections. I spent most of my day talking with various colleagues about the work we are doing and what we would like\nto do in the next 6-12 months.<\/p>\n\n<p>Then, at the end of the day, I was delighted to join my fellow SIG Cloud Provider co-chair Bridget Kromhout, as well Joel Speed, Walter\nFender, and Jesse Butler to have a panel about <a href=\"https:\/\/sched.co\/27NoX\">SIG Cloud Provider Deep Dive: Expanding Our Mission<\/a>. I had a great time and I\u2019m\nfairly sure the other speakers did as well. We had a nice chance to talk about how the SIG is working to create cross-platform building\nblocks for the future of kubernetes. We had a nice attendence for our event and a good discussion among the panelists. I\u2019m optimistic about\nthe future, and it seemed like our discussion and energy permeated into the audience with several nice comments directed towards the SIG at\nthe end.<\/p>\n\n<p><img src=\"\/img\/kubecon-na-2025-sigcp.jpg\" class=\"img-responsive center-block\" alt=\"SIG Cloud Provider\" \/><\/p>\n\n<h2 id=\"thoughts-and-takeaways\">Thoughts and takeaways<\/h2>\n\n<p>This KubeCon seemed smaller than years past. I can only assume this was due to travel and the behavior of the government of the\nUnited States. I was nervous to travel, especially given the shutdown and delays, but I am glad I made it. I love seeing my\nfriends in the community and I generally had a positive experience in Atlanta. I also walked away with many new ideas and that\nfamiliar sense of excitment that percolates when I\u2019ve got plans a\u2019brewing.<\/p>\n\n<p>Some thoughts I had:<\/p>\n<ul>\n  <li>While \u201cAI\u201d was still big, it seemed less big than previous KubeCons.<\/li>\n  <li>Many of my peers are now using LLM-based solutions for the smaller tasks. Nearly all of these efforts are being forced\nby their employers and the results seem mixed at best.<\/li>\n  <li>Resource allocation and placement is still a <em>big deal<\/em>. There continues to be more and more work done on exposing better\nmetadata for workload placement and optimization.<\/li>\n  <li>There is a wealth of younger people looking to become involved with Kubernetes. We need to hire more people!<\/li>\n  <li>Kubernetes is not slowing down, nor does it appear to be entering the maintenance mode just yet.<\/li>\n<\/ul>\n\n<p>That\u2019s about it from me. Another KubeCon in the books. Hopefully this small report has helped to give one window into KubeCon.\nI hope to make to Amsterdam, if you see me come say hi (and don\u2019t be surprised if I give a quizical look at first). I\u2019ll try\nto wear my fedora again, it seemed to be a good way to find me lol. As always, stay safe out there and happy hacking!<\/p>\n\n<p><img src=\"\/img\/kubecon-na-2025-atlanta.jpg\" class=\"img-responsive center-block\" alt=\"Atlanta\" \/><\/p>\n\n","pubDate":"Mon, 17 Nov 2025 00:00:00 +0000","link":"https:\/\/notes.elmiko.dev\/2025\/11\/17\/kubecon-na-2025-retrospective.html","guid":"https:\/\/notes.elmiko.dev\/2025\/11\/17\/kubecon-na-2025-retrospective.html"},{"title":"KubeCon North America 2024 Retrospective","description":"<p>I attended KubeCon North America 2024 in Salt Lake City last week and this is my retrospective on the trip.<\/p>\n\n<p>KubeCon North America for 2024 was hosted in Salt Lake City from November 13th through the 15th, with three days of events before the conference. In addition to KubeCon, I attended the Cloud Native Rejekts on Sunday the 10th, the Kubernetes Contributor Summit on Monday the 11th, and OpenShift Commons on Tuesday the 12th. The weather was mostly nice, with a little rain on Tuesday, with temperatures generally cool in the mid 30s to high 40s Fahrenheit. Salt Lake City is easy to get around, it has generally large blocks and roads in the downtown area, there is also public transportation, and an amazing view of the mountains from just about everywhere.<\/p>\n\n<p><img src=\"\/img\/kubecon-na-2024-outfront.jpg\" class=\"img-responsive center-block\" alt=\"Selfie out front of kubecon\" \/><\/p>\n\n<h2 id=\"cloud-native-rejekts-sunday-november-10\">Cloud Native Rejekts, Sunday November 10<\/h2>\n\n<p>Cloud Native Rejekts is a conference that bills itself as \u201c\u2026 the b-side conference giving a second chance to the many wonderful, but rejected, talks leading into KubeCon + CloudNativeCon\u201d. It is usually a great place to see high quality talks in a smaller setting than KubeCon, and an opportunity to meet up with other people to talk about Kubernetes before the <em>main event<\/em>.<\/p>\n\n<p>This year, I was presenting a talk with colleague David Morrison from the Applied Computing Research Labs titled \u201cKarpenter and Cluster Autoscaler: A data-driven comparison\u201d. We had about 20-25 people in the audience, and it went fairly well considering we had some issues with latency on the slide display.<\/p>\n\n<ul>\n  <li><a href=\"https:\/\/cfp.cloud-native.rejekts.io\/cloud-native-rejekts-na-salt-lake-city-2024\/talk\/CZ9VGR\/\">Read the abstract<\/a><\/li>\n  <li><a href=\"https:\/\/www.youtube.com\/live\/M1R05c1pWmc?si=H5jQnHiflrMDRuaO&amp;t=14609\">Watch the replay<\/a><\/li>\n<\/ul>\n\n<p>On Sunday, there were probably around 200 people in attendance at Rejekts. There were 2 presentation areas: the main theater, and the flex room. Most talks saw people filling the rooms and I noted several with crowds standing in the hall to catch the topics (the eBPF talks were popular). I met several old friends during the social time for Rejekts and it was a great \u201cpre-game\u201d for KubeCon. If you are ever coming to a KubeCon and have the inclination and opportunity to arrive early, Cloud Native Rejekts is a free event and the talks are focused on technical topics, I recommend checking it out at least once.<\/p>\n\n<h2 id=\"kubernetes-contributor-summit-monday-november-11\">Kubernetes Contributor Summit, Monday November 11<\/h2>\n\n<p>The contributor summit is my favorite part of KubeCon. The amount of high-bandwidth conversations and learning that I do during the summit is unparalleled for me at any other Kubernetes event. This year was no different. I\u2019m not sure on the total attendance numbers for the contributor summit, but I would not be surprised if there were a few hundred people there throughout the day. We had several general sessions, including an awards ceremony, and breakout rooms with pre-planned and unconference topics.<\/p>\n\n<p>I felt that one of the big topics for this summit was getting to know and understand the steering committee. We had a good panel session with the committee and it generally seemed like they wanted the contributor community to understand what the steering committee does, and how we can lift more voices to join the committee. I found this to be a good topic of self-awareness for the Kubernetes contributor community, and I continue to be impressed at the thoughtfulness of this community in terms of attempting to keep itself healthy, active, and inclusive.<\/p>\n\n<p>There were 2 talks I attended during the summit that I would like to call attention to:<\/p>\n\n<ul>\n  <li><a href=\"https:\/\/kcsna2024.sched.com\/event\/1nSgo\/official-kubernetes-crds-where-to-from-here\">Official Kubernetes CRDs: Where to from here ? - Nick Young, Isovalent, Rob Scott, Google<\/a><\/li>\n  <li><a href=\"https:\/\/kcsna2024.sched.com\/event\/1nSjo\/unified-framework-for-unit-integration-and-e2e-testing\">Unified framework for unit, integration, and E2E testing - Patrick Ohly, Intel<\/a><\/li>\n<\/ul>\n\n<p>I found the talk on CRDs interesting because for many years now the Kubernetes community has been attempting to address several issues around CRDs (versioning and migrating to name two) and this talk was moving the ball forward on those issues. I also found the talk on testing to be interesting because the author is proposing a test framework that should be built specifically for Kubernetes to address the specific needs of test writers. There seems to be a good opportunity to help push the Kubernetes tests towards a more unified approach, and bring in more contributors who might want to join the testing efforts.<\/p>\n\n<p>Aside from the talks, a big part of attending the contributor summit are the project meetings. I am an active contributor to the Cluster API project and I attended their project update session. This year was not as well-attended as previous years with only around 10-20 people in attendance, but we had a great discussion about the future of the project. Some of the topics we covered during the discussion were the in-place upgrades proposal and status, and etcd resiliency in Cluster API clusters.<\/p>\n\n<h2 id=\"openshift-commons-tuesday-november-12\">OpenShift Commons, Tuesday November 12<\/h2>\n\n<p>OpenShift Commons is a Red Hat sponsored event where we talk about all things Red Hat and OpenShift. I volunteered to help with operations and also to be part of the round table discussions representing the OKD community with my peers Amy Marrich and Jaime Magiera.<\/p>\n\n<p>I love attending the OpenShift Commons, not only because I work on OpenShift and it\u2019s a great opportunity to meet with users, customers, and partners, but also because it\u2019s a great chance for me to spend some time with other Red Hatters. It was a really nice event and it seemed like we were nearly packed to capacity. I\u2019m not sure how many people were there but it had to be a few hundred.<\/p>\n\n<p>The round table discussions were interesting and I ended up talking with a few folks about how using OKD (the community supported version of OpenShift) could help their operations team in <em>sketching out<\/em> features they would like to see in OpenShift someday. The idea being that if you would like to see a feature in OpenShift, and you have some technical understanding around OpenShift and the feature, then you could use OKD to add the feature and demonstrate a proof of concept. This seemed to resonate with the people I talked with, although we all admitted that there is a significant engineering resource demand to create these types of demonstrations.<\/p>\n\n<p>Commons also had one of the best pieces of OpenShift swag I have ever had the pleasure of snagging: OpenShift keycaps!<\/p>\n\n<p><img src=\"\/img\/kubecon-na-2024-keycaps.jpg\" class=\"img-responsive center-block\" alt=\"openshift keycaps in a bucket\" \/><\/p>\n\n<h2 id=\"kubecon-day-1-wednesday-november-13\">KubeCon Day 1, Wednesday November 13<\/h2>\n\n<p>Wednesday brought the official start of KubeCon. I followed the herd (we were told attendance reached 9,200 people this year) to the first day keynotes and with that, KubeCon had begun. Two of the big themes for this KubeCon were AI, and security, with the first two days being dedicated to those topics respectively. The first day keynotes seemed well attended and there was a buzz in the air that was familiar. Immediately after the keynotes was the stampede to the exhibit hall.<\/p>\n\n<p><img src=\"\/img\/kubecon-na-2024-curling.jpg\" class=\"img-responsive center-block\" alt=\"inflatable curling strip\" \/><\/p>\n\n<p>The exhibitor hall was fun, as usual, with some special highlights being the pickleball court and the inflatable curling. It didn\u2019t feel over-crowded or empty, and the times I visited there were plenty of people walking around. The project pavilion was also in the exhibitor hall and I ended up doing an unscheduled impromptu demonstration of the Karpenter Cluster API project, which was fun but also chaotic. This demonstration led to several <em>hallway track<\/em> conversations on Wednesday about Karpenter and Cluster API. While AI and security were the conference themes, one of my themes for the week was definitely Karpenter as I got into many more conversations about it than I anticipated.<\/p>\n\n<p>I saw a bunch of talks on Wednesday, and I would like to call attention to a couple that I think are worth watching when they are posted:<\/p>\n\n<ul>\n  <li><a href=\"https:\/\/sched.co\/1i7ke\">Architecting Tomorrow: The Heterogeneous Compute Resources for New Types of Workloads - Alexander Kanevskiy, Intel Finland<\/a> \u2013 <a href=\"https:\/\/www.youtube.com\/watch?v=jyovyLafMOs\">Watch on YouTube<\/a><\/li>\n  <li><a href=\"https:\/\/sched.co\/1i7mE\">From Observability to Performance - Nadia Pinaeva, Red Hat &amp; Antonio Ojea, Google<\/a> \u2013 <a href=\"https:\/\/www.youtube.com\/watch?v=uYo2O3jbJLk\">Watch on YouTube<\/a><\/li>\n<\/ul>\n\n<p>The first talk on heterogeneous compute resources gave a window into the future of exposing more details about processors to help make scheduling and processing more efficient. Alexander gave a nice overview of how different workloads can be adversely affected by some processor architectures. In a world where nanoseconds can make a difference, this is a great talk to understand how to identify and minimize those bottlenecks.<\/p>\n\n<p>The second talk was a deep breakdown of how networking metrics can be used to identify performance limiters and speed bumps in your infrastructure. Understanding networking failures is difficult enough without all the layers that cloud native infrastructures add. Being able to see real world uses of the metrics alongside the methodology for understanding the implications of those metrics was enlightening for me.<\/p>\n\n<p>Near the end of the day, I ended up getting into a deep discussion with some friends from SIG Storage (shoutouts to Hemant, Jan, and Michelle). They had been chatting all day about a storage issue related to CSI drivers and the Cluster Autoscaler. It was an interesting discussion and I think after about an hour or two we had made some progress on possible solutions, now if only we could find the time to work on them!<\/p>\n\n<p>Wednesday evening was also the \u201cBooth Crawl\u201d at the exhibitor hall, but as I had dinner plans with my colleagues from Red Hat, I skipped out for the evening.<\/p>\n\n<p><img src=\"\/img\/kubecon-na-2024-booth.jpg\" class=\"img-responsive center-block\" alt=\"red hat booth\" \/><\/p>\n\n<h2 id=\"kubecon-day-2-thursday-november-14\">KubeCon Day 2, Thursday November 14<\/h2>\n\n<p>Day 2, I\u2019m exhausted but driven forward by excitement and the goal of delivering a talk on Friday.<\/p>\n\n<p>For the first time in my KubeCon history, I attended a workshop on the DEI track. I saw this title in the schedule and, given my interest in building healthy open source communities, I felt that I had to attend:<\/p>\n\n<ul>\n  <li><a href=\"https:\/\/sched.co\/1pee5\">Be Part of the Solution: Cultivating Inclusion in Open Source - Allyship Workshop<\/a><\/li>\n<\/ul>\n\n<p>I was rewarded with an hour of discussions and group activities where I got to hear about some common issues related to inclusion, and then got to spend time talking with people about their experiences. I had a good time and learned a little more about what I can do to help build more inclusive communities, and what to look for when things aren\u2019t quite going right.<\/p>\n\n<p>I do a lot of work with the Cluster Autoscaler and Karpenter projects, so I was keenly interested to see the SIG Autoscaling update, delivered by my friend from the Cluster API community, Jack Francis. It was a nice overview covering all the projects that the SIG sponsors, how users can get involved, and some crystal ball gazing about what is coming next for the SIG. If you are interested in node or pod autoscaling, definitely watch the recording.<\/p>\n\n<ul>\n  <li><a href=\"https:\/\/kccncna2024.sched.com\/event\/1howV\/sig-autoscaling-projects-update-jack-francis-microsoft\">SIG Autoscaling Projects Update - Jack Francis, Microsoft<\/a> \u2013 <a href=\"https:\/\/www.youtube.com\/watch?v=3fr2J3G1s1U\">Watch on YouTube<\/a><\/li>\n<\/ul>\n\n<p>Something I love being a part of at KubeCon is the SIG Meet n\u2019 Greet. It is an occasion for the Kubernetes SIGs (Special Interest Groups) to make some space and do a little self-promotion. I represented SIG Cloud Provider and had a number of interesting discussions with people who are interested in what the SIG does, and also how they can build cloud controllers for their own infrastructure offerings. I also had a few people approach me about Karpenter related topics, which was nice.<\/p>\n\n<p>My friend and teammate Joel Speed also had a talk on Thursday about CEL validation budgets. If you are using CEL with your Kubernetes API designs, I think it\u2019s worth watching the recording to learn the deep details about how validation budgets are calculated.<\/p>\n\n<ul>\n  <li><a href=\"https:\/\/sched.co\/1i7nv\">Exceeded Your Validation Cost Budget? Now What? - Joel Speed, Red Hat<\/a> \u2013 <a href=\"https:\/\/www.youtube.com\/watch?v=IfaPAqDfJHk\">Watch on YouTube<\/a><\/li>\n<\/ul>\n\n<p>I ended my day at the conference by attending a talk on the OpenCost project. I am keenly interested in exploring how the Cluster API project could expose information about instance pricing. I\u2019m not quite sure what the best way to do this is yet, but I have been wanting to explore OpenCost to see if it might be appropriate for Cluster API. I\u2019m still not sure, but the talk I watched did help me understand some basics about OpenCost. It wasn\u2019t quite what I was expecting, but I found it interesting as a primer.<\/p>\n\n<ul>\n  <li><a href=\"https:\/\/sched.co\/1i7oQ\">Measuring All the Costs with OpenCost Plugins - Alex Meijer, Stackwatch<\/a> \u2013 <a href=\"https:\/\/www.youtube.com\/watch?v=yLAx2z4FqSk\">Watch on YouTube<\/a><\/li>\n<\/ul>\n\n<h2 id=\"kubecon-day-3-friday-november-15\">KubeCon Day 3, Friday November 15<\/h2>\n\n<p>Last day of KubeCon, for now.<\/p>\n\n<p>I didn\u2019t have a lot on my agenda for Friday aside from co-presenting the SIG Cloud Provider maintainer track talk. But, I was happily surprised to see this talk about flaky tests and continuous integration. It gave me a better window into how the Kubernetes testing infrastructure is configured and deployed. I am keenly interested in this because I would like to improve the state of testing for cloud controllers. And, as luck would have it, this talk about flaky CI was happening a few slots before our talk related to testing. I highly recommend watching the recording if you have an interest in Kubernetes continuous integration testing.<\/p>\n\n<ul>\n  <li><a href=\"https:\/\/sched.co\/1hoxc\">Achieving and Maintaining a Healthy CI with Zero Test Flakes - Antonio Ojea, Michelle Shepardson &amp; Benjamin Elder, Google<\/a> \u2013 <a href=\"https:\/\/www.youtube.com\/watch?v=hl3jjCTTL50\">Watch on YouTube<\/a><\/li>\n<\/ul>\n\n<p>Fridays at KubeCon are always tough for me as I am usually exhausted but also want to catch up with people before we all leave. This year was no different. I spent much of my time walking around and having discussions with people (shoutout to Kevin on our talks of what will disrupt Kubernetes and the state of homelab clusters). But, as the day, and con, were winding down it was time to deliver our talk for SIG Cloud Provider:<\/p>\n\n<ul>\n  <li><a href=\"https:\/\/sched.co\/1hoyJ\">Building a More Resilient Future with Advanced Cloud Provider Testing - Michael McCune, Red Hat &amp; Bridget Kromhout, Microsoft<\/a> \u2013 <a href=\"https:\/\/www.youtube.com\/watch?v=5FKMFlooC6c\">Watch on YouTube<\/a><\/li>\n<\/ul>\n\n<p>I\u2019m happy and proud to say that the talk went well despite the standard technical shenanigans at the beginning. I also want to say a big thank you to Bridget as well. I have had the pleasure of co-presenting with Bridget a few times now and she is a talented and amazing person to be on stage with. I was humbled to hear the audience\u2019s reactions to our presentation and I\u2019m so happy that people enjoyed themselves and appreciated our delivery. Looking forward to having another opportunity like this in the future.<\/p>\n\n<p>And with that, I started to make my way out of the building for the last time at this KubeCon. I did see a few people on the way out, and got my last hugs and well-wishes in. It was an amazing and exhausting experience as always, and I\u2019m hopeful I can attend in London.<\/p>\n\n<p><img src=\"\/img\/kubecon-na-2024-leaving.jpg\" class=\"img-responsive center-block\" alt=\"looking down an empty hall\" \/><\/p>\n\n<h2 id=\"thoughts-and-takeaways\">Thoughts and takeaways<\/h2>\n\n<p>KubeCon is a bustling place filled with ideas and excitement, and this edition was no different. Here are some of my unvarnished thoughts about my experiences:<\/p>\n\n<ul>\n  <li>\u201cAI\u201d workloads are still white hot, but I do hear people wondering aloud when, and if, we will see profitable business models emerge from this trend.<\/li>\n  <li>There are more clouds coming. As a SIG Cloud Provider co-chair, and Red Hat engineer on our cluster infrastructure team, I have a decent vantage point for viewing cloud integrations with Kubernetes. I had several direct conversations with people who are interested in writing cloud controllers for their clouds, and talked about wanting to integrate with Cluster API. This made me feel good about the health of that ecosystem.<\/li>\n  <li>More people talked to me about Karpenter Cluster API than I expected. I had several in-depth conversations with people about this project and how it works. There was a genuine excitement that Karpenter could work with Cluster API to unlock its features on all the available platforms. I was happy to hear this and tried to get a sense for what people are wanting. If we can get accurate cost information available in Cluster API, I have a feeling the Karpenter provider will get more attention.<\/li>\n  <li>Dynamic Resource Allocation (DRA) is almost as hot as \u201cAI\u201d. The push for more GPU-centric applications of DRA was present all throughout the con, with several talks being dedicated to DRA and GPU workloads. I think this just speaks to the popularity of workloads that require specific resources. And while today the talk is mostly about GPUs, I look forward to the day when we are talking about DRA for everything from CPUs and memory to customized hardware accelerators.<\/li>\n<\/ul>\n\n<p>So, that\u2019s it, another KubeCon in the books. I hope this retrospective gave you a taste of what the excitement is all about. And I sincerely hope that if you find this exciting, that you also may attend some day. I look forward to seeing you out there on the road, and as always, stay safe out there, and happy hacking =)<\/p>\n\n","pubDate":"Tue, 19 Nov 2024 00:00:00 +0000","link":"https:\/\/notes.elmiko.dev\/2024\/11\/19\/kubecon-na-2024-retrospective.html","guid":"https:\/\/notes.elmiko.dev\/2024\/11\/19\/kubecon-na-2024-retrospective.html"},{"title":"Developing the Karpenter Cluster API Provider","description":"<p>Since October 2023 I\u2019ve been working with the <a href=\"https:\/\/cluster-api.sigs.k8s.io\">Kubernetes Cluster API<\/a> community\nto develop a native <a href=\"https:\/\/karpenter.sh\">Karpenter<\/a> provider so that we can explore the behavior\nof these projects together. Karpenter is an exciting node auto-provisioner that has features\nfor configurable cluster consolidation and deep cloud inventory awareness, and Cluster API is\na declarative infrastructure API for Kubernetes with coverage on nearly 2 dozen providers.\nIf these projects can work well together, it would give the community an excellent way to run\nKarpenter on many cloud providers. As of last week, we have reached the minimally\nviable implementation for a proof-of-concept, and we are in the\n<a href=\"https:\/\/github.com\/kubernetes\/org\/issues\/5097\">process of donating the repository<\/a> to the Kubernetes SIGs community for wider\nexperimentation and development.<\/p>\n\n<p>Although we have reached a nice milestone for the project, there is still much work to do\nas we attempt to reach feature parity with the native Karpenter implementations. We will also\nneed to learn how best to integrate Karpenter with Cluster API, where the bottlenecks might be,\nand how we can create the best user experience possible.<\/p>\n\n<p>With the first phase of work complete, it\u2019s a nice time to reflect on what we have done and\nhow we got here. This is especially relevant given that the developer community around Karpenter\nis still growing and there are many people interested in implementing providers for other\nplatforms. In this post, I am going to share the process we used to develop the Cluster API\nprovider in hopes that it will help others who are making the same journey.<\/p>\n\n<p><img src=\"\/img\/karpcapi-logo.svg\" class=\"img-responsive center-block\" alt=\"Karpenter Cluster API logo\" \/><\/p>\n\n<h2 id=\"planning-the-foundation\">Planning the foundation<\/h2>\n\n<p>To help us solve problems of architectural constraints, project goals, and user experience options,\nwe started a <a href=\"https:\/\/github.com\/kubernetes-sigs\/cluster-api\/blob\/main\/docs\/community\/20231018-karpenter-integration.md\">Karpenter feature group<\/a> in the Cluster API community where we had a regular time\nand space to have and record our discussions and decisions. It took us a few months to arrive at a\ndesign and initial plan for how we would create the proof-of-concept Karpenter. This process was hugely beneficial\nas it gave us ample opportunity to talk through the software constraints that the Cluster API provider\nplaced on this implementation. In specific, the asynchronous nature of infrastructure creation in\nCluster API is slightly at odds with the synchronous nature of Karpenter\u2019s cloud interface.<\/p>\n\n<p>One of the challenges in writing a Karpenter provider for Cluster API is the limitation of needing\nto use the Kubernetes API for making changes to the infrastructure. This means that any time we\nwant to learn about the inventory or status of the infrastructure resources, we need to query the\nKubernetes API and potentially exercise reconciliation loops to ensure that asynchronous behavior\nis captured accurately. This is in stark contrast to infrastructure providers where there is\ndirect access to a metadata service with synchronous responses.<\/p>\n\n<p>In addition to the engineering concerns around designing an integration between Cluster API and\nKarpenter, there is also a necessary focus on how to expose the API features of both projects. Cluster API\nand Karpenter are both provisioning tools built on top of Kubernetes, and this means that they\nhave some overlap in the features they expose. A point of discussion that we spent several meetings\nexploring was the idea of where on the spectrum of \u201cCluster API to Karpenter\u201d does the community want the\nuser experience. Given the nature of Cluster API, I feel this specific design concern will most\nlikely not be an issue for other provider implementers, unless those providers have a user community\nthat expects a deep interaction with the platform APIs.<\/p>\n\n<h2 id=\"initial-code-and-trajectory\">Initial code and trajectory<\/h2>\n\n<p>With the plans solidified and the community in consensus about our initial direction, I started to\nbuild a skeleton repository for the project. I did this by copying the <a href=\"https:\/\/github.com\/kubernetes-sigs\/karpenter\/tree\/main\/kwok\">Kwok provider<\/a> from\nthe <a href=\"https:\/\/github.com\/kubernetes-sigs\/karpenter\">Karpenter repository<\/a> into a new repository, and then building a simple Makefile and\nthe necessary Go files to build the project. At this point I had a basic buildable project and could\nbegin the next step of defining the boundaries for the code changes that would be needed.<\/p>\n\n<p>The Karpenter developers have created a straightforward interface that providers must implement. In\ngeneral it defines functions that provide information about the infrastructure inventory and resources,\nand functions to manage the nodes of the cluster and the resources. It is small enough to reproduce\nhere:<\/p>\n\n<div class=\"language-go highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"c\">\/\/ Create launches a NodeClaim with the given resource requests and requirements and returns a hydrated<\/span>\n\t<span class=\"c\">\/\/ NodeClaim back with resolved NodeClaim labels for the launched NodeClaim<\/span>\n\t<span class=\"n\">Create<\/span><span class=\"p\">(<\/span><span class=\"n\">context<\/span><span class=\"o\">.<\/span><span class=\"n\">Context<\/span><span class=\"p\">,<\/span> <span class=\"o\">*<\/span><span class=\"n\">v1<\/span><span class=\"o\">.<\/span><span class=\"n\">NodeClaim<\/span><span class=\"p\">)<\/span> <span class=\"p\">(<\/span><span class=\"o\">*<\/span><span class=\"n\">v1<\/span><span class=\"o\">.<\/span><span class=\"n\">NodeClaim<\/span><span class=\"p\">,<\/span> <span class=\"kt\">error<\/span><span class=\"p\">)<\/span>\n\t<span class=\"c\">\/\/ Delete removes a NodeClaim from the cloudprovider by its provider id<\/span>\n\t<span class=\"n\">Delete<\/span><span class=\"p\">(<\/span><span class=\"n\">context<\/span><span class=\"o\">.<\/span><span class=\"n\">Context<\/span><span class=\"p\">,<\/span> <span class=\"o\">*<\/span><span class=\"n\">v1<\/span><span class=\"o\">.<\/span><span class=\"n\">NodeClaim<\/span><span class=\"p\">)<\/span> <span class=\"kt\">error<\/span>\n\t<span class=\"c\">\/\/ Get retrieves a NodeClaim from the cloudprovider by its provider id<\/span>\n\t<span class=\"n\">Get<\/span><span class=\"p\">(<\/span><span class=\"n\">context<\/span><span class=\"o\">.<\/span><span class=\"n\">Context<\/span><span class=\"p\">,<\/span> <span class=\"kt\">string<\/span><span class=\"p\">)<\/span> <span class=\"p\">(<\/span><span class=\"o\">*<\/span><span class=\"n\">v1<\/span><span class=\"o\">.<\/span><span class=\"n\">NodeClaim<\/span><span class=\"p\">,<\/span> <span class=\"kt\">error<\/span><span class=\"p\">)<\/span>\n\t<span class=\"c\">\/\/ List retrieves all NodeClaims from the cloudprovider<\/span>\n\t<span class=\"n\">List<\/span><span class=\"p\">(<\/span><span class=\"n\">context<\/span><span class=\"o\">.<\/span><span class=\"n\">Context<\/span><span class=\"p\">)<\/span> <span class=\"p\">([]<\/span><span class=\"o\">*<\/span><span class=\"n\">v1<\/span><span class=\"o\">.<\/span><span class=\"n\">NodeClaim<\/span><span class=\"p\">,<\/span> <span class=\"kt\">error<\/span><span class=\"p\">)<\/span>\n\t<span class=\"c\">\/\/ GetInstanceTypes returns instance types supported by the cloudprovider.<\/span>\n\t<span class=\"c\">\/\/ Availability of types or zone may vary by nodepool or over time.  Regardless of<\/span>\n\t<span class=\"c\">\/\/ availability, the GetInstanceTypes method should always return all instance types,<\/span>\n\t<span class=\"c\">\/\/ even those with no offerings available.<\/span>\n\t<span class=\"n\">GetInstanceTypes<\/span><span class=\"p\">(<\/span><span class=\"n\">context<\/span><span class=\"o\">.<\/span><span class=\"n\">Context<\/span><span class=\"p\">,<\/span> <span class=\"o\">*<\/span><span class=\"n\">v1<\/span><span class=\"o\">.<\/span><span class=\"n\">NodePool<\/span><span class=\"p\">)<\/span> <span class=\"p\">([]<\/span><span class=\"o\">*<\/span><span class=\"n\">InstanceType<\/span><span class=\"p\">,<\/span> <span class=\"kt\">error<\/span><span class=\"p\">)<\/span>\n\t<span class=\"c\">\/\/ DisruptionReasons is for CloudProviders to hook into the Disruption Controller.<\/span>\n\t<span class=\"c\">\/\/ Reasons will show up as StatusConditions on the NodeClaim.<\/span>\n\t<span class=\"n\">DisruptionReasons<\/span><span class=\"p\">()<\/span> <span class=\"p\">[]<\/span><span class=\"n\">v1<\/span><span class=\"o\">.<\/span><span class=\"n\">DisruptionReason<\/span>\n\t<span class=\"c\">\/\/ IsDrifted returns whether a NodeClaim has drifted from the provisioning requirements<\/span>\n\t<span class=\"c\">\/\/ it is tied to.<\/span>\n\t<span class=\"n\">IsDrifted<\/span><span class=\"p\">(<\/span><span class=\"n\">context<\/span><span class=\"o\">.<\/span><span class=\"n\">Context<\/span><span class=\"p\">,<\/span> <span class=\"o\">*<\/span><span class=\"n\">v1<\/span><span class=\"o\">.<\/span><span class=\"n\">NodeClaim<\/span><span class=\"p\">)<\/span> <span class=\"p\">(<\/span><span class=\"n\">DriftReason<\/span><span class=\"p\">,<\/span> <span class=\"kt\">error<\/span><span class=\"p\">)<\/span>\n\t<span class=\"c\">\/\/ Name returns the CloudProvider implementation name.<\/span>\n\t<span class=\"n\">Name<\/span><span class=\"p\">()<\/span> <span class=\"kt\">string<\/span>\n\t<span class=\"c\">\/\/ GetSupportedNodeClasses returns CloudProvider NodeClass that implements status.Object<\/span>\n\t<span class=\"c\">\/\/ NOTE: It returns a list where the first element should be the default NodeClass<\/span>\n\t<span class=\"n\">GetSupportedNodeClasses<\/span><span class=\"p\">()<\/span> <span class=\"p\">[]<\/span><span class=\"n\">status<\/span><span class=\"o\">.<\/span><span class=\"n\">Object<\/span>\n<span class=\"p\">}<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>The first thing I did after getting the project building was to stub out these functions and have them\nall return the equivalent of a zero response or an error. In this way, anyone looking at the code\ncould easily seen what was implemented and what was not. I also added a README file to the project with\na checklist showing which interfaces were implemented. I then set off on the task of implementing the\nindividual functions.<\/p>\n\n<p>In the case of Cluster API, one of the things I spent a significant amount of time on was figuring out\nhow to translate capacity, geographic, and pricing data from the instance types to the Karpenter API types.\nThis might not be as big a problem on platforms with direct access to a metadata service, but given the\nabstracted nature of Cluster API it posed a challenge in writing some of the functions. I am sure this will\nbe a point of investigation and perhaps expansion in the Cluster API project as we learn more about how\nto expose deep infrastructure metadata.<\/p>\n\n<p>One of the biggest challenges for Cluster API is the implementation of the <code class=\"language-plaintext highlighter-rouge\">Create<\/code> method. In Cluster API\nthe creation of new Machines (the API type associated with a Node\u2019s instance) is controlled by a scale\nsubresource on another API type. This relationship is similar to that of a Pod to a ReplicationController or\nDeployment in that an increase in a replica count will trigger the creation of new resources. This means\nthat we must increase the replica count, and then wait for the Machine resource to be created before we\nknow the identifying information about the infrastructure resource. Karpenter, in contrast, would like to\nknow that identifying information when <code class=\"language-plaintext highlighter-rouge\">Create<\/code> returns, and this causes the synchronicity issue between the two.<\/p>\n\n<p>When learning about how to implement the interface functions and what Karpenter expected from them, I spent\nmuch time exploring the <a href=\"https:\/\/github.com\/kubernetes-sigs\/karpenter\">core Karpenter repository<\/a>, but I also studied the way that the\n<a href=\"https:\/\/github.com\/aws\/karpenter-provider-aws\/blob\/main\/pkg\/cloudprovider\/cloudprovider.go\">AWS provider<\/a> and <a href=\"https:\/\/github.com\/Azure\/karpenter-provider-azure\/blob\/main\/pkg\/cloudprovider\/cloudprovider.go\">Azure provider<\/a> implemented their cloud provider interface. This study\nhelped me immensely to understand what values would be absolutely required and which might be more supplemental.\nI also used the code of the <a href=\"https:\/\/github.com\/kubernetes-sigs\/karpenter\/tree\/main\/kwok\">Kwok provider<\/a> as an example, especially early on in development, but as\nI got further into testing and debugging I found the AWS and Azure providers to be more useful examples.<\/p>\n\n<p>Another very helpful resource was the Karpenter community itself. There are several ways to contact\nand participate with the <a href=\"https:\/\/karpenter.sh\/docs\/contributing\/\">Karpenter working group<\/a>, and I highly recommend reaching out if you\nhave questions. I was able to connect with several wonderful and helpful people in the Karpenter community\nby asking questions on their Kubernetes Slack channel (#karpenter-dev) and by attending their meetings with\nquestions and announcements on their agenda.<\/p>\n\n<h3 id=\"a-note-on-code-structure\">A note on code structure<\/h3>\n\n<p>Something that I found helpful when sketching out the initial structure for the code of the project was\nconsidering how the AWS and Azure providers were constructed. Looking at their designs from the cloud provider\ninterface down, it was quick to see that they were both following a similar pattern. In specific the\nabstraction between the cloud provider interface functions and the various resource and data provider\nfunctions is worthy of note for designing new implementations.<\/p>\n\n<h2 id=\"handling-custom-resources\">Handling custom resources<\/h2>\n\n<p>Karpenter requires a few custom resource definitions in order to operate. Notably the NodePool and NodeClaim\ndefinitions, but it will also need some sort of NodeClass implementation for the provider. I found the\npattern for inclusion to be straightforward in the AWS and Azure projects, namely including the YAML manifests\nin a subdirectory of the API code. Following this pattern seemed like the easiest way to keep consistency\nfor developers who might be inspecting the Cluster API provider in the future. To make it even easier, I added\nsome scripting to the makefile to make the generation of those manifests a little easier by rendering them\ndirectly from the vendored dependencies.<\/p>\n\n<p>Checking the custom resource definition manifests in to the code repository makes automating the testing\nmuch easier and also including things like helm charts and other examples. In addition\nto creating the core Karpenter manifests for NodePool and NodeClaim, the repository will also need to contain\nany platform specific manifests, such as the NodeClass implementation.<\/p>\n\n<p>A design question that came up during development was around the lifecycle of Karpenter\u2019s NodePool and NodeClaim\nresources, and how they could be used to carry provider specific information. To be clear, both the NodePool and NodeClaim objects\nare created and reconciled types within the Kubernetes API, and as such you can build other controllers to\ninteract with them. Both the NodePool and NodeClaim are also free to carry provider specific information in the\nform of annotations and labels, as you might expect from any other API object. In my experience, I did not\nhave to implement any specific functions for controlling the lifecycle of the Karpenter resources, they are\nall handled by the core Karpenter controllers.<\/p>\n\n<h2 id=\"testing\">Testing<\/h2>\n\n<p>For Cluster API, testing gave me tremendous confidence in the code that we were building. A big advantage\nfor Cluster API in this area is that we could use <a href=\"https:\/\/book.kubebuilder.io\/reference\/envtest.html\">kubebuilder\u2019s envtest<\/a> package to great effect\nsince most of the platform interactions would happen through Kubernetes resources.<\/p>\n\n<p>Using examples from kubebuilder and some other projects, I was able to quickly configure a test suite that\ncould exercise all of the cloud provider interface functions. This ultimately became a foundation point for\nthe project as it gave me great confidence when bringing the code out of testing and on to a live cluster\nenvironment.<\/p>\n\n<p>Testing should be a core of whatever provider is being built, but depending on the provider, and perhaps the\nmaturity of tooling, it may be more difficult to mock out the infrastructure specific parts of instance\nmanagement. This is something to consider and plan for when building a new provider implementation.<\/p>\n\n<h2 id=\"proof-of-concept\">Proof of concept<\/h2>\n\n<p>After nearly 3 months of design and coding, we achieved the initial proof-of-concept version of the Karpenter\nCluster API provider. There were a few minor bumps to smooth out as integration testing began, but thanks to\nthe unit testing the fixes were quick to implement and before long I was able to demonstrate the application\nin action.<\/p>\n\n<iframe class=\"center-block\" width=\"560\" height=\"315\" src=\"https:\/\/www.youtube.com\/embed\/BZz5ibGP7ZQ?si=DlwNO2O8-nuGNti7\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen=\"\"><\/iframe>\n\n<h2 id=\"future-plans\">Future plans<\/h2>\n\n<p>This is just the beginning. The most basic functionality is working but there are still several open questions:<\/p>\n\n<ul>\n  <li>will we need a more reactive, asynchronous, design for Machine creation?<\/li>\n  <li>how will price data be exposed on Cluster API objects?<\/li>\n  <li>can we implement the opt-in scale from zero capacity information for each provider?<\/li>\n  <li>is it possible to use efficient native interfaces like EC2Fleet and VMSS?<\/li>\n  <li>how will kubelet and user data configuration changes be handled?<\/li>\n  <li>can drift be implemented?<\/li>\n<\/ul>\n\n<p>And those are just a few of the top issues, there will be plenty more as we start to explore the\nbehavior of this integration. I am quite happy and curious to see how things go from here and what the\ncommunity would like to see from this project.<\/p>\n\n<p>If you would like to get involved, please see the <a href=\"https:\/\/cluster-api.sigs.k8s.io\/introduction.html?highlight=meeting#-community-discussion-contribution-and-support\">Cluster API community book<\/a> for more\ninformation on how to contact people and where to follow. Also, please follow the\n<a href=\"https:\/\/github.com\/kubernetes\/org\/issues\/5097\">migration pull request for the repository<\/a> to see when the\nrepository will be adopted into the Kubernetes SIGs organization. Assuming the migration goes through, we\nwill have the common Kubernetes contribution process setup there with plenty of issues to share.<\/p>\n\n<p>Although there are many members of the Cluster API and Karpenter communities who participated in the\ndesign and development of this project, I would like to give a special mention to GitHub user @daimaxiaxie who\ncontributed some very timely patches and collaborated on some tricky code issues. Thank you, I am grateful\nfor your help!<\/p>\n\n<p>Hopefully this has been helpful for those of you who might be building your own Karpenter provider, or who are just\ninterested in open source software development. The biggest takeaway from this experience that I can share is\nthat reading the other providers\u2019 code as examples helped me tremendously, it took time to study their sources but I\nfeel it made my understanding of Karpenter better and gave me more confidence about what we were building.\nHave fun out there and as always, happy hacking =)<\/p>\n\n","pubDate":"Sun, 18 Aug 2024 00:00:00 +0000","link":"https:\/\/notes.elmiko.dev\/2024\/08\/18\/developing-the-karpenter-cluster-api-provider.html","guid":"https:\/\/notes.elmiko.dev\/2024\/08\/18\/developing-the-karpenter-cluster-api-provider.html"},{"title":"Comparing the Kubernetes Cluster Autoscaler and Karpenter","description":"<p>Over the last couple years, the <a href=\"https:\/\/karpenter.sh\">Karpenter project<\/a> has been gaining\npopularity and momentum in the Kubernetes community. It is often spoken about\nin the same breath as the <a href=\"https:\/\/github.com\/kubernetes\/autoscaler\/tree\/master\/cluster-autoscaler\">Cluster Autoscaler<\/a>(CAS), and is commonly\nviewed as a node autoscaler. There are nuanced differences between the two\nprojects that make this equivalence slightly inaccurate and I\u2019d like to\nexplore and highlight those differences.<\/p>\n\n<h2 id=\"a-little-background-for-context\">A little background for context<\/h2>\n\n<p>To start with though, why am I even talking about this?<\/p>\n\n<p>As part of my <code class=\"language-plaintext highlighter-rouge\">$JOB<\/code> at Red Hat I spend a great deal of time working on the\nCAS; mostly doing maintenance and ensuring that the <a href=\"https:\/\/cluster-api.sigs.k8s.io\">Cluster API<\/a>\nprovider has as few bugs and as many features as possible. I am also working\nwith the Cluster API communuity\u2019s <a href=\"https:\/\/github.com\/kubernetes-sigs\/cluster-api\/blob\/main\/docs\/community\/20231018-karpenter-integration.md\">Karpenter feature group<\/a> to understand\nhow we can integrate these projects while preserving the core features from\nboth. On top of that, I have been interested in distributed systems and\ncloud infrastructures for many years and find the topic of autoscaling to be\nfascinating.<\/p>\n\n<p>I get asked a lot of questions about CAS and Karpenter and I thought it would\nbe worthwhile to write something a little more durable and public to help\nshare my perspectives and opinions. That said, what is written here is my\nopinion based on reading documentation and source code, and by operating\nthe projects. I highly recommend reviewing the\n<a href=\"https:\/\/github.com\/kubernetes\/autoscaler\/blob\/master\/cluster-autoscaler\/FAQ.md\">Kubernetes Cluster Autoscaler FAQ<\/a> and\n<a href=\"https:\/\/karpenter.sh\/docs\/concepts\/\">Karpenter Concepts<\/a> documentation pages as many details can\nbe gleaned from these sources. I will also note that my bias is solidly rooted\nin the maintenance of the Cluster API provider for CAS and building a Cluster\nAPI integration for Karpenter, <em>caveat emptor<\/em>.<\/p>\n\n<h2 id=\"whats-in-a-name\">What\u2019s in a name?<\/h2>\n\n<p>To start with, let\u2019s look at the names of the projects: Cluster Autoscaler,\nand Karpenter. A Cluster Autoscaler is something that will automatically scale\nyour cluster. This clearly implies that I have a cluster and somehow the nodes\nin it will be scaled. Meaning that I can have more, or fewer, of the nodes that\nI already have based on some sort of calculation. Fairly straightforward.<\/p>\n\n<p>On the other hand, a Karpenter (carpenter) is someone who builds things out\nof wood. I interpret this to mean that the project implies it will be building\nthings, in this case Kubernetes nodes. So, instead of viewing Karpenter as\nan application that will scale the existing nodes in my cluster, it might be\nmore accurate to view it as a node builder that can provision new nodes in\nthe cluster based on what the workloads of the cluster need.<\/p>\n\n<h2 id=\"scaling-and-provisioning\">Scaling and provisioning<\/h2>\n\n<p>What does it mean that the CAS will scale things and Karpenter will provision\nthings?<\/p>\n\n<p>When configuring the CAS for use, one thing the user must do is to configure\nthe node groups that will be available for CAS to manipulate. The\nconfiguration is highly provider specific and is required for the CAS to\nunderstand what types of nodes it can scale. For example, with the Cluster API\nprovider this process involves the user adding specific annotations to their\nscalable resources (<code class=\"language-plaintext highlighter-rouge\">MachinePool<\/code>, <code class=\"language-plaintext highlighter-rouge\">MachineDeployment<\/code>, <code class=\"language-plaintext highlighter-rouge\">MachineSet<\/code>) to instruct\nthe CAS about scaling inclusion and limits.<\/p>\n\n<p>By contrast, when configuring Karpenter the user must specify what constraints\nwill apply when determining how pods will fit onto specific instance types\nand categories. This means that Karpenter can understand what types of\ninstances will become nodes that a specific pod can be scheduled to, and then\nit can check the infrastructure inventory to determine if an instance can be\ncreated to contain the pod. An example of this can be seen in how the\n<a href=\"https:\/\/karpenter.sh\/v0.33\/faq\/#how-does-karpenter-dynamically-select-instance-types\">Karpenter AWS provider can use EC2 Fleet<\/a> to find many\ninstances which might fit the pod (or group of pods) and then choose the best\noption based on user preferences.<\/p>\n\n<p>Allowing the application to understand provisioning instead of scaling also\nlends itself to more dynamic discovery of instance types and categories at run\ntime. This means that as cloud inventory changes, Karpenter can react to those\nchanges automatically and make moment-to-moment decisions about market\navailability and pricing. By contrast, to emulate this behavior using the CAS\nwith Cluster API would require the user to be updating the\n<code class=\"language-plaintext highlighter-rouge\">InfrastructureMachineTemplate<\/code> resources in their cluster as well as updating\nthe associated scalable types on a continual basis.<\/p>\n\n<h2 id=\"when-to-activate-pending-or-unscheduled\">When to activate, pending or unscheduled?<\/h2>\n\n<p>Another subtle point of difference that flows from the notion of scaling\nversus provisioning is the conditions under which these applications will jump\ninto action.<\/p>\n\n<p>When there are pending pods in a cluster, this means that there\nis a node available which could receive the pod but that it currently does not\nhave allocatable capacity. In these cases, when configured, the CAS and\nKarpenter both have the ability to  add more nodes of the type that will accept\nthe workload.<\/p>\n\n<p>When there are unschedulable pods in a cluster, this means that there are no\nnodes that can satisfy the requirements of the pod. In these cases the CAS\nwill only act if it has a node group which could possibly make a node to\nschedule that pod. This scenario can happen when the CAS is configured to\nhave node groups of size zero and thus there are no nodes in the cluster which\ncould schedule the pod, but the CAS knows how to make that type of node.<\/p>\n\n<p>Karpenter, by contrast, is configured by the user to instruct the provisioning\nof nodes based on pod constraints. In the case of an unschedulable pod,\nKarpenter will refer to its provisioners, and will then request several\ninstance types which could satisfy the pod requirements. Karpenter can then\nchoose the best (by cost, resource, availability, etc) instance type to create\nas the pod is being requested.<\/p>\n\n<p>This is not to say that the CAS won\u2019t also look at several instance types when\ndeciding which to make, but we need to look at the details a little closer to\nunderstand that choice. In Karpenter, having the ability to calculate instance\ntypes based on the workload constraints means that it can request a broad\nrange of instances (limited by user configuration) from the infrastructure. By\ncontrast, CAS can be configured with many instance types and when presented\nwith a decision about which type of instance to request for a specific workload\nit will use the user\u2019s preference (up to and including <a href=\"https:\/\/github.com\/kubernetes\/autoscaler\/tree\/master\/cluster-autoscaler\/expander\/grpcplugin\">custom code<\/a>)\nto make that choice. The CAS is limited by its node group configurations,\nwhich are different on each platform, and may or may not support dynamic\ninstance discovery and full resource and cost data.<\/p>\n\n<p>In both cases, CAS and Karpenter, the user has control over the options for\nhow nodes are chosen, with preferences and priorities across several ranges\nsuch as pricing and resource consumption.<\/p>\n\n<h2 id=\"disrupting-behavior-and-bin-packing\">Disrupting behavior and bin packing<\/h2>\n\n<p>Another topic that comes up frequently when talking about Karpenter is\n<a href=\"https:\/\/en.wikipedia.org\/wiki\/Bin_packing_problem\">bin packing<\/a>. Bin packing in this context refers to the\nalgorithms that a program uses to fit items into a set of groups (bins)\nbased on arbitrary constraints. Both CAS and Karpenter use bin packing when\ncalculating how pods could fit into the possible node choices available. What\npeople are most frequently referring to when talking about bin packing and\nKarpenter is its <a href=\"https:\/\/karpenter.sh\/docs\/concepts\/disruption\/\">consolidation and disruption features<\/a>, which the\nCAS does not replicate.<\/p>\n\n<p>A frequent problem with distributed systems that have workloads which come and\ngo over time, is that the cluster can become sparsely populated. As workloads\nare removed and not replaced they leave <em>holes<\/em> in the nodes. This in turn\ncauses nodes within a cluster to become underutilized. In\nthese situations an activity is required to rebalance the workloads and\nresources in the cluster to optimize usage. CAS and Karpenter both have\nfeatures to help address this problem, with Karpenter providing a more active\napproach.<\/p>\n\n<p>When using the CAS, users have the ability to configure node resource\nutilization thresholds and inactivity timers. Resource usage is calculated\nbased on summed pod resource requests compared to node allocatable capacities.\nWhen utilization falls below the threshold, a node is cordoned and drained, to\nallow for graceful termination before being removed. The inactivity timer\nprovides the mechanism for dictating how long underutilized nodes should\npersist in the cluster. CAS is not doing any explicit rebalancing of workloads\nduring this scale down, it is only removing underutilized nodes. Any pods\ndisrupted during a node removal will be rescheduled by Kubernetes.<\/p>\n\n<p>Karpenter provides users with more options than CAS for defining how the cluster\nwill remove and replace nodes, it refers to these events as consolidation and\ndisruption. In the example of sparse workloads from above, Karpenter can\nconsolidate the cluster on a user defined schedule, preferring to choose instances that\nare cheaper or more resource efficient. This consolidation activity will repack\npods within the cluster (using the Kubernetes scheduler) to replace inefficient\nnode configurations. Karpenter also allows users to replace\nnodes based on cluster configuration skew, age of node in the cluster, and manual\nintervention. Aside from manual intervention, the CAS does not provide an interface\nfor configuring arbitrary node replacement based on user defined conditions.\nIn all cases, when Karpenter removes nodes it follows an orderly eviction process\nto allow for graceful node termination.<\/p>\n\n<h2 id=\"community-and-sig-engagement\">Community and SIG engagement<\/h2>\n\n<p>You might have heard around the Kubernetes community that Karpenter has joined\nthe SIG Autoscaling community. This true!<\/p>\n\n<p>As of last December, the <a href=\"https:\/\/github.com\/kubernetes-sigs\/karpenter\">Karpenter project core<\/a> has been donated to\nthe Kubernetes Autoscaling SIG for maintenance and contribution. This package\nis meant to be used as a library for providers to implement on their platforms.\nCurrently there are <a href=\"https:\/\/github.com\/aws\/karpenter-provider-aws\">AWS<\/a> and <a href=\"https:\/\/github.com\/Azure\/karpenter\">Azure<\/a> implementations.\nHopefully in the future we will have a Cluster API version as well ;)<\/p>\n\n<p>I think it\u2019s important to highlight the provider implementation details in a\nlittle more detail. I am optimistic about building a generic Cluster API\nprovider that would unlock Karpenter on all the Cluster API platforms, but I\nalso acknowledge that this might not provide the best experience and there will\nbe challenges to implementation. Karpenter would like to act as a provisioner\nin the cluster, but Cluster API also wants to perform this role. To preserve the Cluster\nAPI experience for users, the Karpenter and Cluster API controllers will need\nto cooperate on the provisioning front. Making this interface generic might mean\nlosing access to some of the powerful cloud interfaces, like EC2 Fleet, which\nhelp to make Karpenter powerful by extension. I\u2019m sure there will be solutions\nto these problems, but it\u2019s a point of concern that I think about frequently.<\/p>\n\n<p>In addition to code, the joining of the communities has lead to an advancement\nof defining common APIs around cluster lifecycle in relation to node\nprovisioning and removal. See the\n<a href=\"https:\/\/docs.google.com\/document\/d\/1rHhltfLV5V1kcnKr_mKRKDC4ZFPYGP4Tde2Zy-LE72w\/edit\">Cluster Autoscaler\/Karpenter API Alignment AEP draft<\/a> for more\ninformation.<\/p>\n\n<h2 id=\"is-one-better-than-the-other\">Is one \u201cbetter\u201d than the other?<\/h2>\n\n<p>This is a question that I don\u2019t think is quite right. Both CAS and Karpenter\nare tremendous software applications with many satisfied users. They are\ndifferent in approach and features, and I don\u2019t think it\u2019s fair to proclaim\nthat one is greater or lesser than the other. I think it\u2019s more appropriate to\nask, <em>which one is better for your use case?<\/em><\/p>\n\n<p>There is tremendous overlap between the two applications but they do have\ndifferent, opinionated, approaches to solving the problem of having\njust-in-time resources in Kubernetes. In many respects, Karpenter can be\nconfigured in a manner to perform the same task as CAS, and that cannot be\nsaid in the converse. In this respect Karpenter\u2019s features might be seen as\na superset CAS\u2019s features.<\/p>\n\n<p>On the other hand, in clusters where swings in cluster size might be\nlower and choice of instance type is not required to be as dynamic, operating\nthe CAS could prove to be a less complex task. Especially if you have been\nusing the CAS for years already.<\/p>\n\n<p>Additionally, CAS currently runs on 28 providers listed in the repository, with\nat least 2 of those (Cluster API and OpenStack) being platforms that run on a\nmultitude of other platforms. Karpenter currently only supports AWS and Azure.<\/p>\n\n<h2 id=\"caution-strong-opinions-ahead\">Caution, strong opinions ahead<\/h2>\n\n<p>Karpenter seems ideal for situations where you want to manage larger\nheterogeneous clusters that have high amounts of workload churn. Where resource\nmaximization and cost reduction are primary drivers to configuring and\noptimizing the cluster. In these scenarios, Karpenter\u2019s ability to consolidate\nand choose from a wide variety of instance types will be very beneficial.<\/p>\n\n<p>The CAS seems well suited in cases where your cluster has a very reliable rate\nof growth, and the instance types are more homogeneous across the cluster. This\napplies well in smaller clusters, especially if the need for scaling is only a\nsingle node or two over a short period of time (e.g. bursting for an evening).\nIt also applies well in clusters where the instances are treated as a large pool\nand the specific types are less important, or in situations where having some\noverprovisioned capacity is preferred.<\/p>\n\n<p>I find Karpenter\u2019s configuration options to be more complicated than the CAS,\nand I\u2019m not sure how hard it is to debug issues with Karpenter as I have much\nmore time on the CAS. I really like that Karpenter has taken the approach to\nuse API resources (<code class=\"language-plaintext highlighter-rouge\">NodePool<\/code>, <code class=\"language-plaintext highlighter-rouge\">NodeClaim<\/code>, etc.) to drive its behavior as I\nthink it makes it easier to reckon about functionality and exposes more context\nto users.<\/p>\n\n<p>Given the similarities and differences, I think it\u2019s really difficult to make\nan \u201capples to apples\u201d comparison between the two. Karpenter really seems to\nbe positioned as an application that can open the doors on an infrastructure\nwith a large dynamic inventory. I suppose I am not surprised that this\ntechnology originated at AWS since they specialize in having just such an\ninventory. I have a feeling the differences with CAS would be much smaller if Karpenter\nwere implemented on a platform with a smaller or less dynamic inventory. The\nconsolidation and disruption features are very nice, and big features for\nKarpenter, but I believe some of these activities could be replicated by\nwell crafted CAS and Kubernetes configurations, use of projects like the\n<a href=\"https:\/\/github.com\/kubernetes-sigs\/descheduler\">descheduler<\/a>, and custom automation. At this point though, these are\njust my theories.<\/p>\n\n<p>I hope my impressions, interpretations, and opinions expressed here have helped\nyou to figure out what is most beneficial for your needs. If you\u2019ve\nmade it this far, I really appreciate it, and as always happy hacking o\/<\/p>\n\n","pubDate":"Sun, 21 Jan 2024 00:00:00 +0000","link":"https:\/\/notes.elmiko.dev\/2024\/01\/21\/comparing-cas-and-karpenter.html","guid":"https:\/\/notes.elmiko.dev\/2024\/01\/21\/comparing-cas-and-karpenter.html"},{"title":"From Dodging to Shooting in Godot","description":"<p>As a child video games fascinated me, and that inspiration was a large\npart of what drove me to learn about computers and programming. Over the\nyears I\u2019ve spent many wonderful hours playing, designing, and implementing\ngames. I\u2019ve even managed to get paid for it a few times in my life. Making\ngames is a hobby that I love to dabble with when the time, and for those\ntimes I\u2019ve found the <a href=\"https:\/\/godotengine.org\">Godot game engine<\/a> to be a powerful and license\nfriendly toolkit to use.<\/p>\n\n<p>Learning Godot can be frustrating at times, but it can also be very relaxing\nto explore the IDE that the community has created. I\u2019ve done several\ntutorials and have even created a few simple games\n(<a href=\"https:\/\/wmd.opbstudios.com\">Warfare, Magic, Divinity<\/a>, and <a href=\"https:\/\/mom.opbstudios.com\">Maps of Mnemos<\/a>), but recently\nI\u2019ve wanted to learn more about shooting style games. I thought that converting\nthe \u201cDodge the Creeps\u201d game from the basic tutorial into a \u201cShoot the Creeps\u201d game\nwould make for a nice exercise and I could reuse lessons learned in the original.<\/p>\n\n<p>So, here is my take on \u201cShoot the Creeps\u201d. I\u2019ve made some inherent design choices\nsuch as fixing the player at the bottom of the screen and shooting upward, likewise\nmy code choices might not be best idiomatic Godot but I\u2019ve tried to follow the\npatterns established in the original tutorial. <a href=\"https:\/\/gitlab.com\/elmiko\/shoot-the-creeps\">Shoot the Creeps source<\/a>.<\/p>\n\n<p><img src=\"\/img\/shoot-the-creeps.gif\" class=\"img-responsive center-block\" alt=\"shoot the creeps screenshot\" \/><\/p>\n\n<h2 id=\"a-note-on-language\">A note on language<\/h2>\n\n<p>I\u2019ve chosen to use GDScript for all my examples, apologies ahead of time to folks\nwho aren\u2019t using that language. Perhaps in the future I will get into the other\nlanguage options, but given the similarity between Python and GDScript, coupled with\n<a href=\"https:\/\/notes.elmiko.dev\/2022\/12\/18\/why-i-keep-python-in-the-tool-box.html\">my love of python<\/a>, it has been the path of least resistance for me.<\/p>\n\n<h2 id=\"step-1-do-the-tutorial\">Step 1, do the tutorial!<\/h2>\n\n<p>No, seriously, that\u2019s what I did to start this exercise =)<\/p>\n\n<p><a href=\"https:\/\/docs.godotengine.org\/en\/stable\/getting_started\/first_2d_game\/index.html\">Your first 2D game \u2013 Godot Engine (stable) documenation in English<\/a><\/p>\n\n<h2 id=\"move-the-player-to-the-bottom\">Move the player to the bottom<\/h2>\n\n<p>The first choice I have made for this shooter is that the player will be at\nthe bottom of the screen and only move left and right, akin to classics such\nas <a href=\"https:\/\/en.wikipedia.org\/wiki\/Space_Invaders\">Space Invaders<\/a> and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Galaga\">Galaga<\/a>. Because the game uses a node to\ndetermine the starting position for the player, we can change that value directly.\nIn the node inspector, adjust the starting <code class=\"language-plaintext highlighter-rouge\">y<\/code> position for the <code class=\"language-plaintext highlighter-rouge\">Main\/StartPosition<\/code>\nnode to <code class=\"language-plaintext highlighter-rouge\">660<\/code>, as follows:<\/p>\n\n<p><img src=\"\/img\/stc-ss2.png\" class=\"img-responsive center-block\" alt=\"set the player y position\" \/><\/p>\n\n<p>To restrict the player movement, we need to update the <code class=\"language-plaintext highlighter-rouge\">Player.gd<\/code> script file, mainly\nby removing the options for up and down movement and also fixing up the resting\nanimation frame (we want the player to look upward).<\/p>\n\n<div class=\"language-diff highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"gd\">--- a\/Player.gd\n<\/span><span class=\"gi\">+++ b\/Player.gd\n<\/span><span class=\"p\">@@ -19,10 +19,6 @@<\/span> func _process(delta):\n\t\t\t\tvelocity.x += 1\n\t\tif Input.is_action_pressed(\"move_left\"):\n\t\t\t\tvelocity.x -= 1\n<span class=\"gd\">-       if Input.is_action_pressed(\"move_down\"):\n-               velocity.y += 1\n-       if Input.is_action_pressed(\"move_up\"):\n-               velocity.y -= 1\n<\/span>\n\t\tif velocity.length() &gt; 0:\n\t\t\t\tvelocity = velocity.normalized() * speed\n<span class=\"p\">@@ -38,10 +34,8 @@<\/span> func _process(delta):\n\t\t\t\t$AnimatedSprite2D.flip_v = false\n\t\t\t\t# See the note below about boolean assignment.\n\t\t\t\t$AnimatedSprite2D.flip_h = velocity.x &lt; 0\n<span class=\"gd\">-       elif velocity.y != 0:\n<\/span><span class=\"gi\">+       else:\n<\/span>\t\t\t\t$AnimatedSprite2D.animation = \"up\"\n<span class=\"gd\">-               $AnimatedSprite2D.flip_v = velocity.y &gt; 0\n-\n<\/span>\n func _on_body_entered(body):\n\t\thide() # Player disappears after being hit.\n<\/code><\/pre><\/div><\/div>\n<h2 id=\"make-mobs-fly-downwards\">Make mobs fly downwards<\/h2>\n\n<p>Another big design choice I\u2019ve made is to make the mobs only fly downwards. To\naccomplish this, we want to restrict where they spawn to only originate from the\ntop border, and also make them only face downward with the same directional\nvelocity.<\/p>\n\n<p><em>Side note: I think there might be a better way to do this than to use RigidBody2D,\nthe main I reason I chose this is that the physics of the sprites can be affected\nby other objects in the system. It seems like a good challenge for designing a\nshooter would be to use non-physics based sprites.<\/em><\/p>\n\n<p>Removing the extra spawn points is basically a reversal of the steps followed in\nthe <a href=\"https:\/\/docs.godotengine.org\/en\/stable\/getting_started\/first_2d_game\/05.the_main_game_scene.html#spawning-mobs\">Spawning mobs section of the original tutorial<\/a>. By using the red\ndelete point tool on the <code class=\"language-plaintext highlighter-rouge\">Main\/MobPath<\/code> node, we remove the last 3 points (top\nleft, bottom left, and bottom right). This restricts the spawn path to only\noccur along the top border.<\/p>\n\n<p><img src=\"\/img\/stc-ss3.png\" class=\"img-responsive center-block\" alt=\"path tool\" \/><\/p>\n\n<p>Making the mobs face downwards and travel that direction is a short code change.\nWhenever a new mob is spawned, we want to set the direction to a fixed point (<code class=\"language-plaintext highlighter-rouge\">PI \/ 2<\/code>),\nand then let the physics to the rest.<\/p>\n\n<div class=\"language-diff highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"gd\">--- a\/Main.gd\n<\/span><span class=\"gi\">+++ b\/Main.gd\n<\/span><span class=\"p\">@@ -31,14 +31,11 @@<\/span> func _on_mob_timer_timeout():\n\t\tvar mob_spawn_location = get_node(\"MobPath\/MobSpawnLocation\")\n\t\tmob_spawn_location.progress_ratio = randf()\n\n-       # Set the mob's direction perpendicular to the path direction.\n<span class=\"gd\">-       var direction = mob_spawn_location.rotation + PI \/ 2\n-\n<\/span>\t\t# Set the mob's position to a random location.\n\t\tmob.position = mob_spawn_location.position\n\n-       # Add some randomness to the direction\n<span class=\"gd\">-       direction += randf_range(-PI \/ 4, PI \/ 4)\n<\/span><span class=\"gi\">+       # Set the mob's rotation to face the bottom of the window\n+       var direction = PI \/ 2\n<\/span>\t\tmob.rotation = direction\n\n\t\t# Choose the velocity for the mob.\n<\/code><\/pre><\/div><\/div>\n\n<h2 id=\"create-a-bullet-scene\">Create a bullet scene<\/h2>\n\n<p>Following in the patterns from the tutorial, we will make a Bullet scene so that\nwe can efficiently spawn new bullets whenever the fire button is pressed. Bullet\nbehavior is relatively straightforward; they should appear at the same X coordinate\nas the player, and then travel towards the top of the screen. They also need to\nhave a collision box for intersecting with mobs.<\/p>\n\n<p>Make a new scene similar to how the Player and Mob scenes were created.<\/p>\n\n<h3 id=\"node-setup\">Node setup<\/h3>\n\n<p>Click Scene -&gt; New Scene from the top menu of the IDE and add the following nodes:<\/p>\n\n<ul>\n  <li><a href=\"https:\/\/docs.godotengine.org\/en\/stable\/classes\/class_area2d.html\">Area2D<\/a> (named <code class=\"language-plaintext highlighter-rouge\">Bullet<\/code>)\n    <ul>\n      <li><a href=\"https:\/\/docs.godotengine.org\/en\/stable\/classes\/class_colorrect.html\">ColorRect<\/a><\/li>\n      <li><a href=\"https:\/\/docs.godotengine.org\/en\/stable\/classes\/class_collisionshape2d.html\">CollisionShape2D<\/a><\/li>\n      <li><a href=\"https:\/\/docs.godotengine.org\/en\/stable\/classes\/class_visibleonscreennotifier2d.html\">VisibleOnScreenNotifier2D<\/a><\/li>\n    <\/ul>\n  <\/li>\n<\/ul>\n\n<p>Set the children so they can\u2019t be selected, similar to the Player and Mob scenes.<\/p>\n\n<p>In the <a href=\"https:\/\/docs.godotengine.org\/en\/stable\/classes\/class_area2d.html\">Area2D<\/a> properties, under the <a href=\"https:\/\/docs.godotengine.org\/en\/stable\/classes\/class_collisionobject2d.html#class-collisionobject2d\">CollisionObject2D<\/a>\nsection, uncheck the <code class=\"language-plaintext highlighter-rouge\">1<\/code> inside the <code class=\"language-plaintext highlighter-rouge\">Layer<\/code> and <code class=\"language-plaintext highlighter-rouge\">Mask<\/code> properties, and check the\n<code class=\"language-plaintext highlighter-rouge\">2<\/code> inside both. This will make it so that we can have bullets collide with mobs\nbut not with the player.<\/p>\n\n<p><img src=\"\/img\/stc-ss4.png\" class=\"img-responsive center-block\" alt=\"collisionobject2d properties\" \/><\/p>\n\n<p>For the bullet shape, a simple square will suffice for now; a future task would be to\nmake a bullet sprite or animation instead. Set the <a href=\"https:\/\/docs.godotengine.org\/en\/stable\/classes\/class_colorrect.html\">ColorRect<\/a> <code class=\"language-plaintext highlighter-rouge\">Color<\/code> property\nto <code class=\"language-plaintext highlighter-rouge\">00ffff<\/code> (or any other preferable color). Then set the <code class=\"language-plaintext highlighter-rouge\">x<\/code> and <code class=\"language-plaintext highlighter-rouge\">y<\/code> properties to <code class=\"language-plaintext highlighter-rouge\">20<\/code> under\nthe <code class=\"language-plaintext highlighter-rouge\">Size<\/code> sub-section of the <code class=\"language-plaintext highlighter-rouge\">Transform<\/code> section. Lastly, set the <code class=\"language-plaintext highlighter-rouge\">x<\/code> and <code class=\"language-plaintext highlighter-rouge\">y<\/code> of the <code class=\"language-plaintext highlighter-rouge\">Position<\/code>\nproperty to <code class=\"language-plaintext highlighter-rouge\">-10<\/code>. This will make the bullet track from the middle of its area. The end\nresult should look like this:<\/p>\n\n<p><img src=\"\/img\/stc-ss5.png\" class=\"img-responsive center-block\" alt=\"colorrect properties\" \/><\/p>\n\n<p>To properly detect when bullets collide with mobs we need to set the\n<a href=\"https:\/\/docs.godotengine.org\/en\/stable\/classes\/class_collisionshape2d.html\">CollisionShape2D<\/a> <code class=\"language-plaintext highlighter-rouge\">Shape<\/code> property to <code class=\"language-plaintext highlighter-rouge\">RectangleShape2D<\/code>,\nand then expand it to the size of the <code class=\"language-plaintext highlighter-rouge\">ColorRect<\/code>. When everything is put\ntogether it should look like this:<\/p>\n\n<p><img src=\"\/img\/stc-ss6.png\" class=\"img-responsive center-block\" alt=\"bullet properties\" \/><\/p>\n\n<h3 id=\"bullet-script\">Bullet script<\/h3>\n\n<p>Add a script to the <code class=\"language-plaintext highlighter-rouge\">Bullet<\/code> scene as follows.<\/p>\n\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>extends Area2D\n<\/code><\/pre><\/div><\/div>\n\n<p>Whenever a bullet is on the screen we want it to travel towards the top, and register\na hit if it collides with a mob. To begin we want to make the bullet travel.<\/p>\n\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>const velocity = 200.0\n\nfunc _process(delta):\n\tposition.y -= velocity * delta\n<\/code><\/pre><\/div><\/div>\n\n<p>The <code class=\"language-plaintext highlighter-rouge\">_process<\/code> function (see the <a href=\"https:\/\/docs.godotengine.org\/en\/stable\/tutorials\/scripting\/idle_and_physics_processing.html#doc-idle-and-physics-processing\">Idle and Physics Processing docs<\/a>) is called\nwith a frame rate dependent frequency. The code simply uses a constant velocity\nmultiplied by the time delta between invocations to move the bullet.<\/p>\n\n<p>To register when a bullet collides with a mob, we will have the bullet emit a signal, <code class=\"language-plaintext highlighter-rouge\">bullet_hit<\/code>,\nwith an argument of the mob object. This will allow the receiver to update the score and remove mobs.\nWe also <code class=\"language-plaintext highlighter-rouge\">hide()<\/code> the bullet once it has collided with another entity, before finally emitting\nthe signal.<\/p>\n\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>signal bullet_hit(mob)\n\nfunc _on_body_entered(body):\n\thide()\n\tbullet_hit.emit(body)\n<\/code><\/pre><\/div><\/div>\n\n<p>To make this work we must connect the <code class=\"language-plaintext highlighter-rouge\">body_entered(body: Node2D)<\/code> signal from the <code class=\"language-plaintext highlighter-rouge\">Area2D<\/code> object of\nthe <code class=\"language-plaintext highlighter-rouge\">Bullet<\/code> scene to the <code class=\"language-plaintext highlighter-rouge\">_on_body_entered<\/code> function.<\/p>\n\n<p><img src=\"\/img\/stc-ss13.png\" class=\"img-responsive center-block\" alt=\"bullet signals\" \/><\/p>\n\n<p>Lastly, we want to ensure that any bullet that reaches the boundary of the screen is culled.<\/p>\n\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>func _on_visible_on_screen_notifier_2d_screen_exited():\n\tqueue_free()\n<\/code><\/pre><\/div><\/div>\n\n<p>This method needs to be connected to the <code class=\"language-plaintext highlighter-rouge\">screen_exited<\/code> signal of the <code class=\"language-plaintext highlighter-rouge\">VisibleOnScreenNotifier2D<\/code>\nnode of the bullet, similar to what is described in the <a href=\"https:\/\/docs.godotengine.org\/en\/stable\/getting_started\/first_2d_game\/04.creating_the_enemy.html#enemy-script\">\u201cEnemy script\u201d section of the main tutorial<\/a>.<\/p>\n\n<h2 id=\"create-a-fire-bullet-action\">Create a fire bullet action<\/h2>\n\n<p>We want the player to trigger bullet firing when they press the space bar. To do this\nwe create an event in a similar manner as the <a href=\"https:\/\/docs.godotengine.org\/en\/stable\/getting_started\/first_2d_game\/03.coding_the_player.html\">\u201cCoding the player\u201d section from the main tutorial<\/a>.\nClick on Project -&gt; Project Settings to open the project settings window and then\nclick on the Input Map tab at the top. Add a <code class=\"language-plaintext highlighter-rouge\">fire_bullet<\/code> action, like this:<\/p>\n\n<p><img src=\"\/img\/stc-ss7.png\" class=\"img-responsive center-block\" alt=\"event settings\" \/><\/p>\n\n<h3 id=\"player-script\">Player script<\/h3>\n\n<p>Since the <code class=\"language-plaintext highlighter-rouge\">Player<\/code> object currently contains logic for handling pressed buttons, we will\nadd code to the GDScript file for the player to detect when the <code class=\"language-plaintext highlighter-rouge\">fire_bullet<\/code> action\nis pressed.<\/p>\n\n<p>Update the <code class=\"language-plaintext highlighter-rouge\">Player.gd<\/code> file to look like this:<\/p>\n\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>signal fire\n\nfunc _process(delta):\n\tvar velocity = Vector2.ZERO # The player's movement vector.\n\tif Input.is_action_pressed(\"move_right\"):\n\t\tvelocity.x += 1\n\tif Input.is_action_pressed(\"move_left\"):\n\t\tvelocity.x -= 1\n\n\tif Input.is_action_pressed(\"fire_bullet\"):\n\t\tfire.emit()\n<\/code><\/pre><\/div><\/div>\n\n<p>This will emit the <code class=\"language-plaintext highlighter-rouge\">fire<\/code> signal from the <code class=\"language-plaintext highlighter-rouge\">Player<\/code> whenever the <code class=\"language-plaintext highlighter-rouge\">fire_bullet<\/code> action is\npressed.<\/p>\n\n<h3 id=\"connect-the-fire-signal-to-the-main-logic\">Connect the fire signal to the main logic<\/h3>\n\n<p>In the <code class=\"language-plaintext highlighter-rouge\">Main<\/code> scene we will now connect the <code class=\"language-plaintext highlighter-rouge\">fire<\/code> signal from the <code class=\"language-plaintext highlighter-rouge\">Player<\/code> to a\nfunction in the GDScript for the main game logic. Update the <code class=\"language-plaintext highlighter-rouge\">Player<\/code> child node of the\n<code class=\"language-plaintext highlighter-rouge\">Main<\/code> scene to connect the <code class=\"language-plaintext highlighter-rouge\">fire()<\/code> signal to a function in the main GDScript named\n<code class=\"language-plaintext highlighter-rouge\">_on_player_fire()<\/code>, like this:<\/p>\n\n<p><img src=\"\/img\/stc-ss8.png\" class=\"img-responsive center-block\" alt=\"player signal\" \/><\/p>\n\n<p>Create the <code class=\"language-plaintext highlighter-rouge\">_on_player_fire()<\/code> function with an empty body for now.<\/p>\n\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>func _on_player_fire():\n    pass\n<\/code><\/pre><\/div><\/div>\n\n<h2 id=\"create-bullets-and-collisions\">Create bullets and collisions<\/h2>\n\n<p>With the bullet scene and logic for firing in place, we will now add the last pieces\nto create bullets on the screen and the detect when they hit mobs.<\/p>\n\n<h3 id=\"main-script\">Main script<\/h3>\n\n<p>There are several changes which must be made to the main GDScript file. We will go\nthrough them in small pieces.<\/p>\n\n<p>We will need the <code class=\"language-plaintext highlighter-rouge\">Bullet<\/code> scene imported so that we can spawn new bullets when the\nplayer presses the fire action.<\/p>\n\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>@export var bullet_scene: PackedScene\n<\/code><\/pre><\/div><\/div>\n\n<p>Another piece we will want before create bullets is a way to add a cooldown period\nto the fire action so that the player does not create a stream of bullets (although you\ncould change this if you want a stream of bullets!). We need a variable to gate when\nwe are on cooldown, this variable will be used by our fire function and by a timer\nto be added later.<\/p>\n\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>var bullet_cooldown = false\n<\/code><\/pre><\/div><\/div>\n\n<p>Next we add the fire function, this contains a lot of commands and will also rely on\nthe creation of a timer and a followup function. Update the <code class=\"language-plaintext highlighter-rouge\">_on_player_fire()<\/code>\nfunction as follows:<\/p>\n\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>func _on_player_fire():\n\tif bullet_cooldown:\n\t\treturn\n\tbullet_cooldown = true\n\t$BulletTimer.start()\n\tvar bullet = bullet_scene.instantiate()\n\tbullet.position = $Player.position\n\tbullet.position.y -= 20\n\tbullet.bullet_hit.connect(_on_bullet_hit)\n\tadd_child(bullet)\n<\/code><\/pre><\/div><\/div>\n\n<p>If the player is on a bullet cooldown then this function will return immediately.\nIf not on cooldown, it will set the cooldown to true, restart the bullet cooldown\ntimer, instantiate a new <code class=\"language-plaintext highlighter-rouge\">Bullet<\/code> scene, set its position to the same as the player <code class=\"language-plaintext highlighter-rouge\">x<\/code>\nwith a <code class=\"language-plaintext highlighter-rouge\">y<\/code> value slightly above the player, connect the new object\u2019s\n<code class=\"language-plaintext highlighter-rouge\">bullet_hit<\/code> signal (which we added previously in the <code class=\"language-plaintext highlighter-rouge\">Bullet<\/code> scene) to a function\nname <code class=\"language-plaintext highlighter-rouge\">_on_bullet_hit<\/code> (which will be added next), and lastly we add the new object\nas a child node to the main scene.<\/p>\n\n<p>When we created the <code class=\"language-plaintext highlighter-rouge\">Bullet<\/code> scene we described a <code class=\"language-plaintext highlighter-rouge\">bullet_hit<\/code> signal to emit when\nthe bullet collides with another object. In the previous step we created a connection\nbetween the newly create bullet and a function named <code class=\"language-plaintext highlighter-rouge\">_on_bullet_hit<\/code>. We now add\nthat function with a single argument of the object that is hit.<\/p>\n\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>func _on_bullet_hit(mob):\n\tmob.hide()\n\tscore += 1\n\t$HUD.update_score(score)\n<\/code><\/pre><\/div><\/div>\n\n<p>When this function receives the signal, it first hides the mob from view, then adds\none to the score, and lastly updates the score.<\/p>\n\n<p>The last thing we need to do is create a function for the bullet cooldown timer that\nwill be created. It should simply reset the cooldown variable, as follows:<\/p>\n\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>func _on_bullet_timer_timeout():\n\tbullet_cooldown = false\n<\/code><\/pre><\/div><\/div>\n\n<h3 id=\"bullet-cooldown-timer\">Bullet cooldown timer<\/h3>\n\n<p>Because the player object examines the fire event input at the same frequency as the\nframerate, we need to restrict the number of bullets that are created. One way to do\nthat is to create a cooldown gate in the main fire logic, which we did in the previous\nstep. The final step to make that logic work is to add a <a href=\"https:\/\/docs.godotengine.org\/en\/stable\/classes\/class_timer.html\">Timer<\/a> that will gate\nbullet creation.<\/p>\n\n<p>Add a <a href=\"https:\/\/docs.godotengine.org\/en\/stable\/classes\/class_timer.html\">Timer<\/a> child node named <code class=\"language-plaintext highlighter-rouge\">BulletTimer<\/code> to the <code class=\"language-plaintext highlighter-rouge\">Main<\/code> node. Set it\u2019s <code class=\"language-plaintext highlighter-rouge\">Wait Time<\/code>\nproperty to <code class=\"language-plaintext highlighter-rouge\">0.25<\/code>, and check the <code class=\"language-plaintext highlighter-rouge\">One Shot<\/code> checkbox to the on state.<\/p>\n\n<p><img src=\"\/img\/stc-ss9.png\" class=\"img-responsive center-block\" alt=\"bullettimer properties\" \/><\/p>\n\n<p>Now connect the <code class=\"language-plaintext highlighter-rouge\">timeout()<\/code> signal from the <code class=\"language-plaintext highlighter-rouge\">BulletTimer<\/code> to the <code class=\"language-plaintext highlighter-rouge\">_on_bullet_timer_timeout()<\/code>\nfunction in the main GDScript.<\/p>\n\n<p><img src=\"\/img\/stc-ss10.png\" class=\"img-responsive center-block\" alt=\"bullettimer signals\" \/><\/p>\n\n<h3 id=\"update-mob-collision-mask\">Update Mob collision mask<\/h3>\n\n<p>To finalize the collision mechanics between the bullet scene and the mob scene, we want\nto put the <code class=\"language-plaintext highlighter-rouge\">Mob<\/code> scene into the layer <code class=\"language-plaintext highlighter-rouge\">2<\/code> collision mask.<\/p>\n\n<p>In the <a href=\"https:\/\/docs.godotengine.org\/en\/stable\/classes\/class_rigidbody2d.html\">RigidBody2D<\/a> properties, under the <a href=\"https:\/\/docs.godotengine.org\/en\/stable\/classes\/class_collisionobject2d.html#class-collisionobject2d\">CollisionObject2D<\/a>\nsection, and check the <code class=\"language-plaintext highlighter-rouge\">2<\/code> inside the <code class=\"language-plaintext highlighter-rouge\">Layer<\/code> property. This will make it so that mobs will mask\nthe same layer as bullets when processing collisions.<\/p>\n\n<p><img src=\"\/img\/stc-ss12.png\" class=\"img-responsive center-block\" alt=\"mob collision\" \/><\/p>\n\n<p>We keep the mob in layer <code class=\"language-plaintext highlighter-rouge\">1<\/code> as well so that collision with the player will cause the\ngame to end.<\/p>\n\n<h2 id=\"final-clean-up\">Final clean up<\/h2>\n\n<p>Let\u2019s go back and clean up a few details. To start we can remove the <code class=\"language-plaintext highlighter-rouge\">ScoreTimer<\/code> node as\nit will no longer be needed.<\/p>\n\n<div class=\"language-diff highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"gd\">--- a\/Main.gd\n<\/span><span class=\"gi\">+++ b\/Main.gd\n<\/span><span class=\"p\">@@ -9,7 +9,6 @@<\/span> var bullet_cooldown = false\n func game_over():\n \t$Music.stop()\n \t$DeathSound.play()\n<span class=\"gd\">-\t$ScoreTimer.stop()\n<\/span> \t$MobTimer.stop()\n \t$HUD.show_game_over()\n \t\n<span class=\"p\">@@ -47,13 +46,8 @@<\/span> func _on_mob_timer_timeout():\n \t# Spawn the mob by adding it to the Main scene.\n \tadd_child(mob)\n \n<span class=\"gd\">-func _on_score_timer_timeout():\n-\tscore += 1\n-\t$HUD.update_score(score)\n-\n<\/span> func _on_start_timer_timeout():\n \t$MobTimer.start()\n<span class=\"gd\">-\t$ScoreTimer.start()\n<\/span> \n func _on_player_fire():\n<\/code><\/pre><\/div><\/div>\n\n<p>As part of the final polish we will also change the heads up display message to read\n\u201cShoot the Creeps\u201d instead of \u201cDodge the Creeps\u201d. In the <code class=\"language-plaintext highlighter-rouge\">HUD<\/code> scene, change the\n<code class=\"language-plaintext highlighter-rouge\">Text<\/code> property of the <code class=\"language-plaintext highlighter-rouge\">Label<\/code> node to read \u201cShoot the Creeps!\u201d.<\/p>\n\n<p><img src=\"\/img\/stc-ss11.png\" class=\"img-responsive center-block\" alt=\"hud label\" \/><\/p>\n\n<p>Then change the HUD GDscript to update the message properly.<\/p>\n\n<div class=\"language-diff highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"gd\">--- a\/HUD.gd\n<\/span><span class=\"gi\">+++ b\/HUD.gd\n<\/span><span class=\"p\">@@ -12,7 +12,7 @@<\/span> func show_game_over():\n \t# Wait until the MessageTimer has counted down.\n \tawait $MessageTimer.timeout\n \t\n<span class=\"gd\">-\t$Message.text = \"Dodge the\\nCreeps!\"\n<\/span><span class=\"gi\">+\t$Message.text = \"Shoot the\\nCreeps!\"\n<\/span> \t$Message.show()\n \t# Make a one-shot timer and wait for it to finish.\n \tawait get_tree().create_timer(1.0).timeout\n<\/code><\/pre><\/div><\/div>\n\n<h2 id=\"trying-it-out\">Trying it out!<\/h2>\n\n<p>Hopefully, if you\u2019ve made it this far things are still working for you. If you are\nhaving troubles getting your version to run properly, you can find a reference\nthat was the inspiration for this tutorial at <a href=\"https:\/\/gitlab.com\/elmiko\/shoot-the-creeps\">gitlab.com\/elmiko\/shoot-the-creeps<\/a>.\nIf you find errors in this tutorial or the associated source code, please do not\nhesitate to <a href=\"https:\/\/gitlab.com\/elmiko\/shoot-the-creeps\/-\/issues\">open an issue in the repository<\/a>.<\/p>\n\n<p>The game is fairly simple with the defaults that I chose, you can have some fun by changing\nthe bullet timer, velocity, and sizes of the mobs, player, and bullets. I wrote this\nmodification of the main tutorial to learn more about how to build a shooter style game. I\nmay not have chosen the most idiomatic methods for this implementation but I learned a lot\nand have been inspired to try out my next project.<\/p>\n\n<p><a href=\"https:\/\/godotengine.org\">Godot<\/a> really is a tremendous platform for experimenting and developing games. I\u2019ve\nfound it to inspire and empower my own personal hobby pursuit of making computer games.\nI hope this tutorial has helped on your journey to building your passion games. Stay safe\nout there, and as always, happy hacking!<\/p>\n\n","pubDate":"Sat, 26 Aug 2023 00:00:00 +0000","link":"https:\/\/notes.elmiko.dev\/2023\/08\/26\/from-dodging-to-shooting.html","guid":"https:\/\/notes.elmiko.dev\/2023\/08\/26\/from-dodging-to-shooting.html"},{"title":"Exploring OpenShift Must Gather Data","description":"<p>One of the aspects of working for Red Hat, and on the OpenShift product, that I\nget tremendous joy from are the activities that we have focused on empowering\nassociates to spend time contributing to open source projects and our communities.\nMany companies have these types of agreements with their employees, Wikipedia\nrefers to it as <a href=\"https:\/\/en.wikipedia.org\/wiki\/Side_project_time\">Side project time<\/a>, most famously popularized by Google\u2019s\n<a href=\"https:\/\/builtin.com\/software-engineering-perspectives\/20-percent-time\">\u201c20 percent time\u201d<\/a>. At Red Hat we call these times \u201cHack n\u2019 Hustle\u201d or\n\u201cShift week\u201d depending on when they occur, but the primary goal is for us to\nhave time working on passion projects, or contributing to upstreams, or even\nto be spending time in our physical communities giving back to those around\nus.<\/p>\n\n<p>For awhile now, since at least April 2021 by GitHub\u2019s record, I have been working\non a project that I call <a href=\"https:\/\/github.com\/elmiko\/camgi.rs\">\u201ccamgi.rs\u201d<\/a> (originally <a href=\"https:\/\/github.com\/elmiko\/okd-camgi\">\u201cokd-camgi\u201d<\/a>). Which\noriginally stood for <strong>C<\/strong>luster <strong>A<\/strong>utoscaler <strong>M<\/strong>ust <strong>G<\/strong>ather <strong>I<\/strong>nvestigator\n(I guess it still does stand for that, we just don\u2019t talk about it that way anymore). It was a\ntool that I started to develop to help with the arduous process of understanding why the\n<a href=\"https:\/\/github.com\/kubernetes\/autoscaler\">cluster autoscaler<\/a> had failed in a given scenario on OpenShift.<\/p>\n\n<p>Now, to set the stage a little more, there is a debugging tool that we use heavily\nin OpenShift to help diagnose failures. That tool is called <a href=\"https:\/\/github.com\/openshift\/must-gather\">\u201cmust-gather\u201d<\/a>, and\nit produces a tarball full of all sorts of Kubernetes goodness; including log files,\nmanifests, and even audit logs from a subset of the cluster in question. Must gather\nis very flexible and can be extended in many ways to add all sorts of custom\ninformation, but I will save that for another post. The main point here is that\nI wanted a visual way to quickly diagnose what was happening without having to open\na dozen YAML and log files. So, camgi was born.<\/p>\n\n<h2 id=\"looking-at-a-must-gather-archive\">Looking at a must gather archive<\/h2>\n\n<p>At a very high level, the must gather archive contains a bunch of directories that\nall have various bits of information about the cluster and it looks something like this:<\/p>\n\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>$ ls -l\ntotal 6676\ndrwxr-xr-x. 1 mike mike     874 May 30 16:30 cluster-scoped-resources\ndrwxr-xr-x. 1 mike mike     142 May 30 16:30 etcd_info\n-rw-r--r--. 1 mike mike 6824435 May 30 16:30 event-filter.html\ndrwxr-xr-x. 1 mike mike      14 May 30 16:30 host_service_logs\ndrwxr-xr-x. 1 mike mike      14 May 30 16:30 ingress_controllers\ndrwxr-xr-x. 1 mike mike      34 May 30 16:30 insights-data\ndrwxr-xr-x. 1 mike mike      44 May 30 16:30 monitoring\ndrwxr-xr-x. 1 mike mike    3332 May 30 16:30 namespaces\ndrwxr-xr-x. 1 mike mike     188 May 30 16:30 network_logs\ndrwxr-xr-x. 1 mike mike      66 May 30 16:30 pod_network_connectivity_check\ndrwxr-xr-x. 1 mike mike      28 May 30 16:30 static-pods\n-rw-r--r--. 1 mike mike     550 May 30 16:30 timestamp\n-rw-r--r--. 1 mike mike      78 May 30 16:28 version\n<\/code><\/pre><\/div><\/div>\n\n<p>That\u2019s just the top directory, I ran a <code class=\"language-plaintext highlighter-rouge\">tree<\/code> command on the archive and it says:<\/p>\n\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>2745 directories, 4381 files\n<\/code><\/pre><\/div><\/div>\n\n<p>whoa! That is way too much to reproduce in this post, but i guarantee it\u2019s got a\nton of good stuff in there.<\/p>\n\n<p>To make things much simpler, I wanted a web page that I could use to browse around and\nget a \u201cbird\u2019s eye view\u201d of what is happening. So, that\u2019s what I did. Today the camgi\noutput looks like this on the summary page:<\/p>\n\n<p><img src=\"\/img\/camgi-1.png\" class=\"img-responsive center-block\" alt=\"summary page\" \/><\/p>\n\n<p>You can immediately see some information about the cluster state when the must gather\nwas created. In this case (from a CI run) we can see that 1 Machine is not in a running\nstate, and 2 ClusterOperators are having issues (one is not upgradeable, and another is\ndegraded or not available). This tool started to change the way I could debug things and\nmade it much quicker to find problems. It was also starting to have an affect on my\ncolleagues as they started to ask for more features and custom resources to be added.<\/p>\n\n<p>Diving a little deeper, we can see how this tool can be used to explore resource and log\ndata.<\/p>\n\n<div style=\"padding:56.34% 0 0 0;position:relative;\"><iframe src=\"https:\/\/player.vimeo.com\/video\/836441744?badge=0&amp;autopause=0&amp;player_id=0&amp;app_id=58479\" frameborder=\"0\" allow=\"autoplay; fullscreen; picture-in-picture\" allowfullscreen=\"\" style=\"position:absolute;top:0;left:0;width:100%;height:100%;\" title=\"camgi 0.9.0 demo\"><\/iframe><\/div>\n<script src=\"https:\/\/player.vimeo.com\/api\/player.js\"><\/script>\n\n<h2 id=\"project-history-and-design-goals\">Project history and design goals<\/h2>\n\n<p>When I first started planning this project I was inspired by the work of a colleague\nwho had started an HTML-based display (<em>shoutout to Mike G!<\/em>), and also by the inclusion of\na different HTML file in the must gather output. You might have noticed in the directory listing\nabove a file name <code class=\"language-plaintext highlighter-rouge\">event-filter.html<\/code>. This file is a static HTML file with all the data included\nwithin the page. You can use it to search and filter the events which were emitted from the cluster\nduring the capute period. I thought this was really cool, although I also acknowledge it\u2019s\nnot the most frugal way to create an HTML page (more on this later).<\/p>\n\n<p>So I went to my <a href=\"https:\/\/notes.elmiko.dev\/2022\/12\/18\/why-i-keep-python-in-the-tool-box.html\">old favorite tool Python<\/a> to begin hacking up a static page to contain\nall the data I wanted to highlight.  This allowed me to rapidly prototype as I was able to use\nmodules like Jinja, PyYaml, and the standard library to quickly manipulate the text data. But as\nI got requests to include the output into our build systems for continuous integration it\nbecame apparent that including all the necessary Python modules was going to be very\ndifficult. It was at this point that I decided to re-write the project in Rust so that it could\nbe built as a binary for distribution. I chose Rust because I wanted to learn more about the language\nand this seemed like a perfect opportunity.<\/p>\n\n<p>After several months of development, I was able to release the new tool and get it included in our\nCI infrastructure as I could now have it downloaded during runtime. This process was difficult\nas I had to ensure that my builds would be usable within the containers that our\nCI uses to generate must gather artifacts. This was a trip down memory lane as I was fighting with\nglibc incompatibilities that really brought me back to my early C days. But finally, it was done\nand I was able to have it included in the output, which you can see today if you find the Prow output\nfrom a CI run on our many repositories.<\/p>\n\n<p><img src=\"\/img\/camgi-2.png\" class=\"img-responsive center-block\" alt=\"camgi in prow\" \/><\/p>\n\n<p>Having a single output file from the tool makes it very simple to include the artifact in whatever\nformat we choose. Although it would be more efficient to have some sort of HTTP server hosting the\nfiles from the must gather, this adds a lot of overhead for how it is used and confines the way\nit can be included in other places. It does produce quite large files sometimes, especially when\ninvestigating clusters with many nodes that have been active for a long time. But usually the files are only\nbeing generated locally in those cases, so we aren\u2019t passing around 500Mb HTML files, <em>usually<\/em>\u2026 XD<\/p>\n\n<h2 id=\"operating-camgi\">Operating camgi<\/h2>\n\n<p>Camgi itself is quite easy to install and operate. You can either get a binary release for Linux x86_64\ntargets from the <a href=\"https:\/\/github.com\/elmiko\/camgi.rs\/releases\">releases page on GitHub<\/a>, install it directly from <a href=\"https:\/\/github.com\/elmiko\/camgi.rs\">source<\/a> by cloning\nthe repo and running <code class=\"language-plaintext highlighter-rouge\">cargo build<\/code>, or by installing from <a href=\"https:\/\/crates.io\/crates\/camgi\">crates.io<\/a> by running <code class=\"language-plaintext highlighter-rouge\">cargo install camgi<\/code>.<\/p>\n\n<p>Once installed simply run the <code class=\"language-plaintext highlighter-rouge\">camgi<\/code> command with your must gather archive as a target, such as:<\/p>\n\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>[mike@ultra] ~\/Downloads\/my-must-gather\n$ camgi must-gather.local\/ &gt; camgi.html\n<\/code><\/pre><\/div><\/div>\n\n<p>Then open the resulting file in the browser of your choice.<\/p>\n\n<h2 id=\"release-090-and-the-future\">Release 0.9.0 and the future\u2026<\/h2>\n\n<p>We are currently winding down the latest Shift week at Red Hat, and as part of my activities I\nhave added some new features and created the <a href=\"https:\/\/github.com\/elmiko\/camgi.rs\/releases\/tag\/v0.9.0\">0.9.0 release of camgi<\/a>.<\/p>\n\n<p>As part of my development process I have been <a href=\"https:\/\/github.com\/elmiko\/camgi.rs\/issues\">opening issues<\/a> and <a href=\"https:\/\/github.com\/elmiko\/camgi.rs\/graphs\/contributors\">inviting my colleagues<\/a>\nto help in the construction of camgi. Even though I don\u2019t spend every week working on camgi, creating\nissues and reaching out to my peers for advice and guidance has been a tremendous help. When people\nask for new features, find bugs, or identify areas of improvement I quickly open an issue to remember\nwhat has been asked. In this manner I help myself out for the future and maintain a nice queue of things\nto hack on, it\u2019s been a tremendous experience for me.<\/p>\n\n<p>I mentioned earlier that creating a giant static HTML file is not the most frugal way to handle this\nactivity. One thing that I would really like to solve for the future is reducing the size of the log\nfiles that are included as I notice that sometimes the browser really has to crunch to make things work.\nThis is one of my top goals for the future, but I still have some learning to do so that I can achieve\nit in a way that is convenient for people to access the full log files. We\u2019ll see how it goes.<\/p>\n\n<p>If you\u2019ve made it this far, I hope this tale has at least been interesting and perhaps even inspired you\nto build your own projects or get involved with other collaborators. For me, I will be coding away on the\n<em>way too many<\/em> side projects I have and looking for ways to contribute back and become more involved with\nthe open source community at large. And so, as always, stay safe out there and happy hacking =)<\/p>\n\n","pubDate":"Thu, 15 Jun 2023 00:00:00 +0000","link":"https:\/\/notes.elmiko.dev\/2023\/06\/15\/exploring-openshift-must-gather-data.html","guid":"https:\/\/notes.elmiko.dev\/2023\/06\/15\/exploring-openshift-must-gather-data.html"},{"title":"Diving Deeper into Cluster API Testing","description":"<p>Recently I had the opportunity to spend some time reviewing and deep diving\ninto the <a href=\"https:\/\/cluster-api.sigs.k8s.io\">Cluster API<\/a> end-to-end test suite with the guidance of\n<a href=\"https:\/\/github.com\/fabriziopandini\">Fabrizio Pandini<\/a>. He has been crafting a change to the\n<a href=\"https:\/\/github.com\/kubernetes-sigs\/cluster-api-provider-kubemark\">Kubemark provider<\/a> that will integrate the\n<a href=\"https:\/\/cluster-api.sigs.k8s.io\/developer\/e2e.html\">Cluster API E2E test framework<\/a> so that we can more easily\ndevelop tests that utilize the Kind + Cluster API + Kubemark configurations\nthat I have <a href=\"https:\/\/notes.elmiko.dev\/2021\/10\/11\/setup-dev-capi-kubemark.html\">mentioned<\/a> a <a href=\"https:\/\/notes.elmiko.dev\/2023\/01\/21\/automating-my-hollow-kubernetes-test-rig.html\">few times<\/a> in the past. We paired\nup so that I could better understand the test framework and to talk about\ndebugging the pull request.<\/p>\n\n<p>The integration patch can be seen here: <a href=\"https:\/\/github.com\/kubernetes-sigs\/cluster-api-provider-kubemark\/pull\/69\">kubernetes-sigs\/cluster-api-provider-kubemark#69<\/a>,\nand Farbizio was also kind enough to let me record our deep dive so that we could share it\nwith the wider community:<\/p>\n\n<div style=\"margin: 1em;\">\n<iframe class=\"center-block\" width=\"640\" height=\"480\" src=\"https:\/\/www.youtube-nocookie.com\/embed\/KU7i4TfD1tg\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen=\"\"><\/iframe>\n<\/div>\n\n<p>All the code that was shown in the video is available in the pull request linked above,\nthe <a href=\"https:\/\/github.com\/kubernetes-sigs\/cluster-api-provider-kubemark\">cluster-api-provider-kubemark<\/a> repository, and the <a href=\"https:\/\/github.com\/kubernetes-sigs\/cluster-api\">cluster-api<\/a>\nrepository.<\/p>\n\n<p>If you are curious about the Tilt configuration we talk about, please see the\n<a href=\"https:\/\/cluster-api.sigs.k8s.io\/developer\/tilt.html\">Developing Cluster API with Tilt<\/a> page of the documentation. And if you have\nbeen following my <a href=\"https:\/\/github.com\/elmiko\/cluster-api-kubemark-ansible\">Cluster API Kubemark Ansible<\/a> efforts that I mentioned in the\n<a href=\"https:\/\/notes.elmiko.dev\/2023\/01\/21\/automating-my-hollow-kubernetes-test-rig.html\">previous post<\/a>, I have also added a new playbook for installing the Tilt server\nas well.<\/p>\n\n<p>One of the things I love about open source software and the culture that has evolved with it\nare the people and the communities behind the monitors. I want to give my special thanks and\ngratitude to Fabrizio for being a great mentor and collaborator, and to the rest of the Cluster\nAPI community for being awesome in general and for creating a warm and welcoming place to\nshare a passion for technology.<\/p>\n\n<p>as always, happy hacking =)<\/p>\n\n","pubDate":"Tue, 28 Feb 2023 00:00:00 +0000","link":"https:\/\/notes.elmiko.dev\/2023\/02\/28\/diving-deeper-into-cluster-api-testing.html","guid":"https:\/\/notes.elmiko.dev\/2023\/02\/28\/diving-deeper-into-cluster-api-testing.html"},{"title":"Automating My Hollow Kubernetes Test Rig","description":"<p><em>special thanks to Jos\u00e9 Castillo Lema for helping me to improve these scripts<\/em><\/p>\n\n<p>Way back in October of 2021, I wrote a post about\n<a href=\"https:\/\/notes.elmiko.dev\/2021\/10\/11\/setup-dev-capi-kubemark.html\">Setting Up a Development Environment for the Cluster API Kubemark Provider<\/a>.\nIn that piece I explained how I\u2019m configuring <a href=\"https:\/\/kind.sigs.k8s.io\">Kind<\/a> with <a href=\"https:\/\/cluster-api.sigs.k8s.io\">Cluster API<\/a> and\nthe <a href=\"https:\/\/github.com\/kubernetes-sigs\/cluster-api-provider-kubemark\">Kubemark provider<\/a> to create \u201chollow\u201d <a href=\"https:\/\/kubernetes.io\">Kubernetes<\/a> clusters. In the time\nsince then, I\u2019ve converted those instructions into a set of <a href=\"https:\/\/www.ansible.com\/\">Ansible<\/a>\nplaybooks and helper scripts which make the automation of this process very easy.\nSo, without further ado, let\u2019s look at how to deploy a virtual server for running\nhollow Kubernetes scale tests.<\/p>\n\n<h2 id=\"versions\">Versions<\/h2>\n\n<p>Before we get rolling though, these are the versions I am using at the time of\nwriting. I cannot guarantee that these will work in the future, but for as long as I\ncontinue to maintain these repositories they should be updated over time. Reader be aware.<\/p>\n\n<ul>\n  <li>Kubernetes 1.25.3<\/li>\n  <li>Cluster API 1.3.1<\/li>\n  <li>Cluster API Kubemark Provider 0.5.0<\/li>\n  <li>Ubuntu 22.04 Server<\/li>\n<\/ul>\n\n<h2 id=\"process\">Process<\/h2>\n\n<p>I\u2019m going to walk through this from the ground up. I will start by creating a\nvirtual machine, using Ansible to update it and build the Kubernetes bits, and\nfinally how to work my helper scripts to create clusters.<\/p>\n\n<h3 id=\"creating-a-virtual-machine\">Creating a Virtual Machine<\/h3>\n\n<p>I\u2019m using <a href=\"https:\/\/fedoraproject.org\">Fedora<\/a> as my host operating system with the default Gnome\ndesktop environment installed. Gnome comes with <a href=\"https:\/\/help.gnome.org\/users\/gnome-boxes\/stable\/\">Boxes<\/a> as the main\ngraphical application for managing virtual machines. Although my host is Fedora,\nI like to use Ubuntu for the virtual machine because the Docker integration is\na little easier for me. Kind does work with Podman but I have had\nissues getting the Cluster API Docker provider to work with it. An improvement I would like\nto make to this process is to automate the virtual machine creation process by using a\nscript that talks directly to the host hypervisor, or perhaps using a cloud image.<\/p>\n\n<p><img src=\"\/img\/boxes-1.png\" class=\"img-responsive center-block\" alt=\"creating a vm with boxes\" \/><\/p>\n\n<p>I usually create a virtual machine with 16 GB of RAM and 64 GB of hard drive space,\nthis is enough for me to test small clusters with up to a few dozen nodes.<\/p>\n\n<p><img src=\"\/img\/boxes-2.png\" class=\"img-responsive center-block\" alt=\"selecting vm size\" \/><\/p>\n\n<p>One thing I find really convenient about the Ubuntu installer is the ability\nto pull my SSH keys from GitHub.<\/p>\n\n<p><img src=\"\/img\/ubuntu-install-1.png\" class=\"img-responsive center-block\" alt=\"ubuntu ssh key install\" \/><\/p>\n\n<p>At this point I like to stop the virtual machine and make a snapshot. This allows\nme to quickly reset the instance back to a semi-pristine state if I feel like\ninstalling different versions of the tooling, or just want a <em>blank slate<\/em>.<\/p>\n\n<p><img src=\"\/img\/boxes-3.png\" class=\"img-responsive center-block\" alt=\"making a snapshot\" \/><\/p>\n\n<h2 id=\"installing-the-tooling\">Installing the Tooling<\/h2>\n\n<p>Once the virtual machine is created and rebooted, and I ensure I can login with\nSSH, I clone my <a href=\"https:\/\/github.com\/elmiko\/cluster-api-kubemark-ansible\">Cluster API Kubemark Ansible<\/a> repository\nto my Fedora host. This repository contains a couple playbooks; one for installing\nthe toolchain, and another for building the Cluster API binaries. The first thing\nI do is copy the <code class=\"language-plaintext highlighter-rouge\">inventory<\/code> directory to <code class=\"language-plaintext highlighter-rouge\">inventory.local<\/code> and then edit the <code class=\"language-plaintext highlighter-rouge\">hosts<\/code>\nfile to look like this:<\/p>\n\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>all:\n  hosts:\n    192.168.122.165\n  vars:\n    devel_user: mike\n    cluster_api_repo: https:\/\/github.com\/kubernetes-sigs\/cluster-api.git\n    cluster_api_version: v1.3.1\n    provider_kubemark_repo: https:\/\/github.com\/kubernetes-sigs\/cluster-api-provider-kubemark.git\n    provider_kubemark_version: v0.5.0\n<\/code><\/pre><\/div><\/div>\n\n<p>This <code class=\"language-plaintext highlighter-rouge\">hosts<\/code> file shows that my virtual machine is at <code class=\"language-plaintext highlighter-rouge\">192.168.122.165<\/code>, I will login\nas <code class=\"language-plaintext highlighter-rouge\">mike<\/code>, and the playbooks will install Cluster API version <code class=\"language-plaintext highlighter-rouge\">v1.3.1<\/code> and Kubemark\nprovider version <code class=\"language-plaintext highlighter-rouge\">v0.5.0<\/code>.<\/p>\n\n<p>After updating the inventory, I run the command to execute the setup playbook. Keep\nin mind that the <code class=\"language-plaintext highlighter-rouge\">-K<\/code> command line flag will ask for a password to become root. The\ncommand looks like this:<\/p>\n\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>$ ansible-playbook -K -i inventory.local setup_devel_environment.yaml\n<\/code><\/pre><\/div><\/div>\n\n<p>This will run for 10-15 minutes depending on connection speeds and local variables,\nbut when it finishes it should look like this (yes, I need to investigate that\ndeprecation warning):<\/p>\n\n<p><img src=\"\/img\/ansible-1.png\" class=\"img-responsive center-block\" alt=\"ansible setup results\" \/><\/p>\n\n<p>Something to note about this step of the process is that a container will be started\non the virtual machine to host a Docker registry. This container is used by Kind\nso that local images can be quickly pushed into the running Kubernetes clusters.\nTo access it you need to tag container images as belonging to <code class=\"language-plaintext highlighter-rouge\">localhost:5000\/<\/code>.<\/p>\n\n<p><img src=\"\/img\/docker-1.png\" class=\"img-responsive center-block\" alt=\"docker container running\" \/><\/p>\n\n<h2 id=\"building-cluster-api\">Building Cluster API<\/h2>\n\n<p>Now that the toolchain is setup to build Go code and container images, I want to\ninstall the Cluster API project, Cluster API Kubemark provider, and then build\neverything. To start the process I use this Ansible command, note that this does\nnot need root access:<\/p>\n\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>$ ansible-playbook -i inventory.local build_clusterctl_and_images.yaml\n<\/code><\/pre><\/div><\/div>\n\n<p>Like the previous playbook, this could also take 10-15 minutes depending on resources.\nWhen finished it should look something like this:<\/p>\n\n<p><img src=\"\/img\/ansible-2.png\" class=\"img-responsive center-block\" alt=\"ansible build results\" \/><\/p>\n\n<h2 id=\"preparing-for-launch\">Preparing for Launch<\/h2>\n\n<p>The last step in my process is to install my <a href=\"https:\/\/github.com\/elmiko\/capi-hacks\">CAPI Hacks<\/a> on the virtual\nmachine. These are a set of convenience scripts and Kubernetes manifests that I\nuse regularly to make the process of starting new clusters easier. Let\u2019s look at\nthe files I use most frequently.<\/p>\n\n<p><strong><code class=\"language-plaintext highlighter-rouge\">01-kind-mgmt-config.yaml<\/code><\/strong><\/p>\n\n<p>This is the configuration file for the Kind management cluster, it sets up a couple\nthings including the Kubernetes version, local Docker socket location, and patch\nfor the local registry. Usually the only reason to change this file is when updating\nthe Kubernetes version.<\/p>\n\n<div class=\"language-yaml highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"na\">kind<\/span><span class=\"pi\">:<\/span> <span class=\"s\">Cluster<\/span>\n<span class=\"na\">apiVersion<\/span><span class=\"pi\">:<\/span> <span class=\"s\">kind.x-k8s.io\/v1alpha4<\/span>\n<span class=\"na\">name<\/span><span class=\"pi\">:<\/span> <span class=\"s\">mgmt<\/span>\n<span class=\"na\">networking<\/span><span class=\"pi\">:<\/span>\n  <span class=\"na\">apiServerAddress<\/span><span class=\"pi\">:<\/span> <span class=\"s2\">\"<\/span><span class=\"s\">127.0.0.1\"<\/span>\n<span class=\"na\">nodes<\/span><span class=\"pi\">:<\/span>\n<span class=\"pi\">-<\/span> <span class=\"na\">role<\/span><span class=\"pi\">:<\/span> <span class=\"s\">control-plane<\/span>\n  <span class=\"na\">image<\/span><span class=\"pi\">:<\/span> <span class=\"s\">docker.io\/kindest\/node:v1.25.3<\/span>\n  <span class=\"na\">extraMounts<\/span><span class=\"pi\">:<\/span>\n    <span class=\"pi\">-<\/span> <span class=\"na\">hostPath<\/span><span class=\"pi\">:<\/span> <span class=\"s\">\/var\/run\/docker.sock<\/span>\n      <span class=\"na\">containerPath<\/span><span class=\"pi\">:<\/span> <span class=\"s\">\/var\/run\/docker.sock<\/span>\n<span class=\"na\">containerdConfigPatches<\/span><span class=\"pi\">:<\/span>\n<span class=\"pi\">-<\/span> <span class=\"pi\">|-<\/span>\n  <span class=\"s\">[plugins.\"io.containerd.grpc.v1.cri\".registry.mirrors.\"localhost:5000\"]<\/span>\n    <span class=\"s\">endpoint = [\"http:\/\/kind-registry:5000\"]<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p><strong><code class=\"language-plaintext highlighter-rouge\">01-start-mgmt-cluster.sh<\/code><\/strong><\/p>\n\n<p>A wrapper to start the Kind cluster used by Cluster API as the management cluster.\nIt will be named <code class=\"language-plaintext highlighter-rouge\">mgmt<\/code> in Kind. Running this command should look like:<\/p>\n\n<p><img src=\"\/img\/capi-hacks-1.png\" class=\"img-responsive center-block\" alt=\"start the mgmt cluster\" \/><\/p>\n\n<p><strong><code class=\"language-plaintext highlighter-rouge\">02-apply-localregistryhosting-configmap.sh<\/code><\/strong><\/p>\n\n<p>Add the local registry to the management cluster. This could probably be rolled\ninto the previous script, but just in case you don\u2019t want the local registry it\nis separate. Running this command is relatively uninteresting, but should look\nlike this:<\/p>\n\n<p><img src=\"\/img\/capi-hacks-2.png\" class=\"img-responsive center-block\" alt=\"setup the registry\" \/><\/p>\n\n<p><strong><code class=\"language-plaintext highlighter-rouge\">03-clusterctl-init.sh<\/code><\/strong><\/p>\n\n<p>Install Cluster API into the management cluster using the <code class=\"language-plaintext highlighter-rouge\">clusterctl<\/code> command line\ntool. This file also contains the version information for the local Cluster API and\nKubemark provider information. If it runs successfully it should look similar to\nthis:<\/p>\n\n<p><img src=\"\/img\/capi-hacks-3.png\" class=\"img-responsive center-block\" alt=\"installing capi\" \/><\/p>\n\n<p>I usually confirm that things are working by checking all the pods on the management\ncluster.<\/p>\n\n<p><img src=\"\/img\/capi-hacks-4.png\" class=\"img-responsive center-block\" alt=\"checking pods\" \/><\/p>\n\n<p>If things don\u2019t go well, you might see an error like this:<\/p>\n\n<p><img src=\"\/img\/capi-hacks-5.png\" class=\"img-responsive center-block\" alt=\"failed capi install\" \/><\/p>\n\n<p>This error shows us that we have a version mismatch between the expected and the found\nversions for Cluster API. In cases like this either the version should be changed\nin the script, or the configuration in the <code class=\"language-plaintext highlighter-rouge\">$HOME\/.cluster-api<\/code> directory should\nbe checked.<\/p>\n\n<h2 id=\"launching-kubemark-clusters\">Launching Kubemark Clusters<\/h2>\n\n<p>Finally, the moment has arrived. We are ready to start deploying clusters. The\n<code class=\"language-plaintext highlighter-rouge\">kubemark<\/code> directory of the <a href=\"https:\/\/github.com\/elmiko\/capi-hacks\">CAPI Hacks<\/a> contains some pre-formatted\nmanifests for deploying clusters. I start by creating the objects in the\n<code class=\"language-plaintext highlighter-rouge\">kubemark-workload-control-plane.yaml<\/code> manifest file, this will create a new cluster\nwith a single Docker Machine to host the control plane. I am using a Docker Machine\nhere so that the control plane pods will actually run. After running\n<code class=\"language-plaintext highlighter-rouge\">kubectl create -f kubemark-workload-control-plane.yaml<\/code>, I watch the Machine objects\nuntil I see the control plane is <code class=\"language-plaintext highlighter-rouge\">Running<\/code>.<\/p>\n\n<p><img src=\"\/img\/capi-hacks-6.png\" class=\"img-responsive center-block\" alt=\"watching control plane machines\" \/><\/p>\n\n<p>Next I apply a Container Network Interface (CNI) provider to the new workload cluster\nto ensure that the nodes of the cluster can become fully ready. I use the <code class=\"language-plaintext highlighter-rouge\">deploy-cni.sh<\/code>\nscript to add Calico as the CNI provider (there is also a script to deploy OVN Kubernetes).\nI also use the <code class=\"language-plaintext highlighter-rouge\">get-kubeconfig.sh<\/code> script to make managing the kubeconfig files a little\neasier. When successful it looks something like this:<\/p>\n\n<p><img src=\"\/img\/capi-hacks-7.png\" class=\"img-responsive center-block\" alt=\"deploying CNI\" \/><\/p>\n\n<p>Lastly, I create the workload cluster compute nodes by running <code class=\"language-plaintext highlighter-rouge\">kubectl create -f kubemark-workload-md0.yaml<\/code>.\nThis manifest contains the Cluster API objects for the MachineDeployment and related\ninfrastructure resources to add Kubemark Machines to our workload cluster.\nKubemark is so fast to load that within a 5-10 seconds I have all the new machines\nin a running state:<\/p>\n\n<p><img src=\"\/img\/capi-hacks-8.png\" class=\"img-responsive center-block\" alt=\"examine kubemark\" \/><\/p>\n\n<p>Inspecting the pods on the workload cluster you might note that the Calico containers\nassigned to Kubemark nodes are stuck at the <code class=\"language-plaintext highlighter-rouge\">Init:0\/3<\/code> status. I\u2019m not quite sure\nwhy this happens but I suspect it is an artifact of Kubemark, I\u2019d like to investigate\nfurther but for the time being it does not seem to cause problems with testing.<\/p>\n\n<p><img src=\"\/img\/capi-hacks-9.png\" class=\"img-responsive center-block\" alt=\"examine pods\" \/><\/p>\n\n<h2 id=\"to-the-moon\">To the Moon!<\/h2>\n\n<p>The workflow is now nearly completely automated, or at least reduced to a much simpler\nseries of commands. I have also found some allies along the way as people have shared\nsuggestions, bug fixes, and improvements with me through my\n<a href=\"https:\/\/github.com\/elmiko\/cluster-api-kubemark-ansible\">Cluster API Kubemark Ansible playbook<\/a>\nand <a href=\"https:\/\/github.com\/elmiko\/capi-hacks\">CAPI Hacks<\/a> repositories.<\/p>\n\n<p>There are more scripts and helpers inside the <a href=\"https:\/\/github.com\/elmiko\/capi-hacks\">CAPI Hacks<\/a>, after setting up\nclusters I tend to use the Cluster Autoscaler scripts to test the scaling mechanisms\nof that code. I am also learning about others using a similar workflow to test the\ninner working of Cluster API as well.<\/p>\n\n<p>If you\u2019ve made it this far, thank you, I hope you\u2019ve learned a little more about how\nto setup virtualized testing environments for Kubernetes, and maybe even tried it out\nfor yourself. If you have ideas or suggestions, or just want to chat about how to\nget Cluster API and Kubemark working better, open an issue on one of those repositories\nor come find me on the <a href=\"https:\/\/kubernetes.slack.com\">Kubernetes Slack<\/a> instance as <code class=\"language-plaintext highlighter-rouge\">@elmiko<\/code>, and until next time happy hacking =)<\/p>\n\n","pubDate":"Sat, 21 Jan 2023 00:00:00 +0000","link":"https:\/\/notes.elmiko.dev\/2023\/01\/21\/automating-my-hollow-kubernetes-test-rig.html","guid":"https:\/\/notes.elmiko.dev\/2023\/01\/21\/automating-my-hollow-kubernetes-test-rig.html"},{"title":"Why I Keep Python In the Tool Box","description":"<p>I started learning the <a href=\"https:\/\/python.org\">Python language<\/a> back in the late\n2000s while I was working at a company writing global positioning software for\nin-car navigation. It was a fun job and a great team to work with, and I had\nrejoined them after a year hiatus to lead an effort of converting the software\nstack from Windows CE to Linux. We were also building a new hardware platform,\nbut I was always on the software side and left the physical bits to my capable\ncolleagues. I was able to use the language to great effect when creating a\nD-Bus interface library that our application developers could use to communicate\nbetween host apps, and again when creating a cross-compiling harness to build\nentire ARM root filesystems (this was in the days when Yocto and Buildroot were\nyoung).<\/p>\n\n<p>Over the years my love of Python continued and deepened as I learned to use it\nfor writing all sorts of applications, eventually working on the OpenStack project\n(which was pretty much all server side Python).  I\u2019d venture that next\nto nearly two decades of C experience, Python is the second best language that\nI know. Which brings me to today\u2019s example.<\/p>\n\n<h2 id=\"making-a-molehill-out-of-a-mountain\">Making a molehill out of a mountain<\/h2>\n\n<p><img src=\"\/img\/molehill.png\" class=\"img-responsive center-block\" alt=\"molehill\" \/>\n<span class=\"pull-right\"><em><a href=\"https:\/\/www.google.com\/profiles\/dieder.plu\">Dieder Plu<\/a>, <a href=\"https:\/\/creativecommons.org\/licenses\/by-sa\/3.0\/deed.en\">CC-BY-SA 3.0<\/a><\/em><\/span><br \/><\/p>\n\n<p>One of the areas of Kubernetes that the team I\u2019m on at Red Hat maintains for\nOpenShift are the <a href=\"https:\/\/kubernetes.io\/docs\/concepts\/architecture\/cloud-controller\/\">Cloud Controller Managers<\/a>.\nThese are a set of Kubernetes controllers that run in-cluster to help make\nintegrating with the underlying infrastructure smoother. As you might imagine\neach one of these controllers is written specifically for a single infrastructure\nprovider. In the past this code had all been integrated into the main Kubernetes\ncode repository, but as maintaining these bits in a common place does not scale\nwell with the addition of ever more providers, there has been an effort to\n<a href=\"https:\/\/github.com\/kubernetes\/enhancements\/tree\/master\/keps\/sig-cloud-provider\/2395-removing-in-tree-cloud-providers\">remove them from the main source repository<\/a>.<\/p>\n\n<p>As part of my work at Red Hat, and with the Kubernetes community, I have been\ninvestigating ways that we can grow the testing coverage for these new external\ncloud controller managers. One of the things I would like to do, if possible, is\ncreate a way for each provider to write their own interface implementation which\nwould allow utilizing a central set of tests for all providers, current and future.\nTo that end I have been browsing the upstream end-to-end tests which\n<a href=\"https:\/\/github.com\/kubernetes\/kubernetes\/tree\/master\/test\/e2e\">exist in the Kubernetes repository<\/a>.<\/p>\n\n<p>One of the core pieces of functionality for cloud controller managers is watching\n<a href=\"https:\/\/kubernetes.io\/docs\/concepts\/services-networking\/service\/\">Kubernetes Services<\/a>\nand ensuring that they are backed by a load balancer (where applicable). These\ntests are scattered throughout the Kubernetes end-to-end tests and I wanted to\nfind a convenient way to find them all. A quick suggestion from <a href=\"https:\/\/github.com\/andrewsykim\">Andrew<\/a>\n, one of the SIG Cloud Provider chairs, was to use the <a href=\"https:\/\/github.com\/onsi\/ginkgo\">Ginkgo<\/a>\nbinary tool with regular expression to find the tests. Which turned out to be a\ngreat suggestion because with a quick command line I was able to parse all the\ndescriptions into a formatted JSON file. The command looked like this:<\/p>\n\n<div class=\"language-shell highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"nv\">$ <\/span>ginkgo <span class=\"nt\">-r<\/span> <span class=\"nt\">--dry-run<\/span> <span class=\"nt\">-v<\/span> <span class=\"nt\">--focus<\/span> .<span class=\"k\">*<\/span><span class=\"o\">[<\/span>sS]ervice.<span class=\"k\">*<\/span> <span class=\"nt\">--json-report<\/span> .\/service-tests.json <span class=\"nt\">--keep-going<\/span> kubernetes\/test\/e2e\/...\n<\/code><\/pre><\/div><\/div>\n\n<p>I ran this command from the parent directory of the Kubernetes repository on\nmy local host. What it does is to do a \u201cdry run\u201d of all the tests, recursing\nthrough directories, focusing only on tests with the regular expression\n<code class=\"language-plaintext highlighter-rouge\">.*[sS]ervice.*<\/code> in their hierarchy text (the stuff in the <code class=\"language-plaintext highlighter-rouge\">It(\"does stuff\")<\/code> clauses and whatnot),\nand then writes the output to a file named <code class=\"language-plaintext highlighter-rouge\">service-tests.json<\/code>. All while\ncontinuing past any failures.<\/p>\n\n<p>After running this command, I end up with a huge JSON file:<\/p>\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>-rw-r--r--. 1 mike mike 9.1M Dec  8 14:57 service-tests.json\n<\/code><\/pre><\/div><\/div>\n<p>and looking inside it doesn\u2019t get much better:<\/p>\n<div class=\"language-json highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"p\">{<\/span><span class=\"w\">\n    <\/span><span class=\"nl\">\"ContainerHierarchyTexts\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"kc\">null<\/span><span class=\"p\">,<\/span><span class=\"w\">\n    <\/span><span class=\"nl\">\"ContainerHierarchyLocations\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"kc\">null<\/span><span class=\"p\">,<\/span><span class=\"w\">\n    <\/span><span class=\"nl\">\"ContainerHierarchyLabels\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"kc\">null<\/span><span class=\"p\">,<\/span><span class=\"w\">\n    <\/span><span class=\"nl\">\"LeafNodeType\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"SynchronizedBeforeSuite\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n    <\/span><span class=\"nl\">\"LeafNodeLocation\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">{<\/span><span class=\"w\">\n        <\/span><span class=\"nl\">\"FileName\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"\/home\/mike\/dev\/kubernetes\/test\/e2e\/e2e.go\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n        <\/span><span class=\"nl\">\"LineNumber\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"mi\">77<\/span><span class=\"w\">\n    <\/span><span class=\"p\">},<\/span><span class=\"w\">\n    <\/span><span class=\"nl\">\"LeafNodeLabels\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"kc\">null<\/span><span class=\"p\">,<\/span><span class=\"w\">\n    <\/span><span class=\"nl\">\"LeafNodeText\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n    <\/span><span class=\"nl\">\"State\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"passed\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n    <\/span><span class=\"nl\">\"StartTime\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"0001-01-01T00:00:00Z\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n    <\/span><span class=\"nl\">\"EndTime\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s2\">\"0001-01-01T00:00:00Z\"<\/span><span class=\"p\">,<\/span><span class=\"w\">\n    <\/span><span class=\"nl\">\"RunTime\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"mi\">0<\/span><span class=\"p\">,<\/span><span class=\"w\">\n    <\/span><span class=\"nl\">\"ParallelProcess\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span><span class=\"w\">\n    <\/span><span class=\"nl\">\"NumAttempts\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"mi\">0<\/span><span class=\"p\">,<\/span><span class=\"w\">\n    <\/span><span class=\"nl\">\"MaxFlakeAttempts\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"mi\">0<\/span><span class=\"p\">,<\/span><span class=\"w\">\n    <\/span><span class=\"nl\">\"MaxMustPassRepeatedly\"<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"mi\">0<\/span><span class=\"w\">\n<\/span><span class=\"p\">}<\/span><span class=\"err\">,<\/span><span class=\"w\">\n<\/span><\/code><\/pre><\/div><\/div>\n<p>it\u2019s 278000 some odd lines of those entries. This is gonna take a while\u2026<\/p>\n\n<h2 id=\"the-serpent-lurking-in-the-jungle\">The serpent lurking in the jungle<\/h2>\n\n<p>As I was staring at these entries, starting to get a little cross-eyed, I wondered\nif I might use a script or something to pull all the files and line numbers out,\nmaybe associated with their titles. Just something to pair down the raw the data\nin the file. Then inspiration struck me, I could write a small Python application\nwhich could create an HTML page with links to all the test files. I could then\nuse my browser to at least parse things in a more convenient manner.<\/p>\n\n<p>The architecture I was imagining looked something like this:<\/p>\n\n<div class=\"language-plaintext highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code>   tests.json\n       |                       +-----------------------------+\n       v                   +-&gt; | http:\/\/localhost\/index.html |\n  +--------------------+   |   +-----------------------------+\n  | Python http.server | --+\n  +--------------------+   |   +--------------------------------------+\n       ^                   +-&gt; | kubernetes.git\/test\/e2e\/framework.go |\n       |                       +--------------------------------------+\n  kubernetes.git\/\n<\/code><\/pre><\/div><\/div>\n\n<p>where the Python HTTP serving is running from my local Kubernetes directory,\ntaking a Ginkgo output JSON file as input, and serving up an index page and\nsource files. I knew Python had all the necessary building blocks in the standard\nlibrary; JSON processors, HTTP servers, plenty of string formatting options.<\/p>\n\n<p>I didn\u2019t want to get too complicated as I realized two things; I didn\u2019t want to\nspend more than a couple hours putting it together, and I didn\u2019t want extra\ndependencies, only the Python standard libraries. My reasoning for the first was\nthat any extra time spent hacking on this tool added to the total time for the\ninvestigation and I was very sensitive about not getting lost in a tool sharpening\nexercise. The second reason was that I didn\u2019t want to contend with any sort of\nvirtual environments or other packaging tools. I knew that all the building\nblocks I needed were in the standard library, if I was shrewd I could do this\nwithout installing extra helpers (no matter how nice they are!).<\/p>\n\n<p>What I ended up with is something I call <a href=\"https:\/\/gitlab.com\/elmiko\/biloba.py\">biloba.py<\/a>\n, named after the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Ginkgo_biloba\">humble tree<\/a>, which\nat 170-ish lines of Python is one of the more compact but useful applications\nI\u2019ve written, and I\u2019m also quite proud of myself for accomplishing all the goals.\nIt uses only the standard library, and serves a web page built from the entries\nin the source JSON. The main index has links which open into separate tabs\nthat take you directly to the source for the test. It\u2019s fairly minimal, but allowed\nme to take that list of tests and look through all of them within the span of\nabout a week. The output looks like this:<\/p>\n\n<p><img src=\"\/img\/biloba-py-index.png\" class=\"img-responsive center-block\" alt=\"index page from biloba.py\" \/><\/p>\n\n<p>and the links to the code open into new tabs that are fairly plain:<\/p>\n\n<p><img src=\"\/img\/biloba-py-code.png\" class=\"img-responsive center-block\" alt=\"code page from biloba.py\" \/><\/p>\n\n<p>Simple, but effective.<\/p>\n\n<h2 id=\"what-does-it-do\">What does it do?<\/h2>\n\n<p>Since it\u2019s so small, let\u2019s take a look at some of the choices I made and perhaps\nI can give some of my reasoning. Before we get started though, I\u2019d like to\nacknowledge at the outset that Python is a dynamically typed language (although it does\nhave options for static typing), and as such I tend to use it as a way to <em>sketch out<\/em>\napplications quickly. I like it\u2019s pseudo-code style and the dynamic typing allows\nme to run quickly with scissors, this might not be to every person\u2019s liking and I\nacknowledge that bias at the outset.<\/p>\n\n<h3 id=\"html\">HTML<\/h3>\n\n<div class=\"language-python highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"n\">html_template<\/span> <span class=\"o\">=<\/span> <span class=\"s\">'''\n&lt;html&gt;\n&lt;head&gt;\n&lt;title&gt;Biloba&lt;\/title&gt;\n&lt;style&gt;\nspan.highlight {\n    background: #bababa;\n    display: block;\n}\n&lt;\/style&gt;\n&lt;\/head&gt;\n&lt;body&gt;\n{body}\n&lt;\/body&gt;\n&lt;\/html&gt;\n'''<\/span>\n\n<span class=\"n\">index<\/span> <span class=\"o\">=<\/span> <span class=\"n\">html_template<\/span><span class=\"p\">.<\/span><span class=\"nb\">format<\/span><span class=\"p\">(<\/span><span class=\"n\">body<\/span><span class=\"o\">=<\/span><span class=\"s\">'Not generated'<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>The first part here establishes a template that I will use to create the wrapper\npage that holds all the other pages. I can reuse this for the index and for the\ncode pages. It is also marked up for Python\u2019s\n<a href=\"https:\/\/docs.python.org\/3\/library\/string.html#formatstrings\">format string syntax<\/a>,\nwhich makes it convenient for that reuse. I also declare a global variable for\nthe index page so that I can have a value to know if things did not load properly,\nand to reuse with the handler and main functions.<\/p>\n\n<h3 id=\"data-helpers\">Data Helpers<\/h3>\n\n<div class=\"language-python highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"k\">class<\/span> <span class=\"nc\">Suite<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">__init__<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">suite<\/span><span class=\"p\">):<\/span>\n        <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">reports<\/span> <span class=\"o\">=<\/span> <span class=\"p\">[]<\/span>\n        <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">description<\/span> <span class=\"o\">=<\/span> <span class=\"n\">suite<\/span><span class=\"p\">.<\/span><span class=\"n\">get<\/span><span class=\"p\">(<\/span><span class=\"s\">'SuiteDescription'<\/span><span class=\"p\">)<\/span>\n        <span class=\"k\">if<\/span> <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">description<\/span> <span class=\"ow\">is<\/span> <span class=\"bp\">None<\/span> <span class=\"ow\">or<\/span> <span class=\"nb\">len<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">description<\/span><span class=\"p\">)<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">0<\/span><span class=\"p\">:<\/span>\n            <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">description<\/span> <span class=\"o\">=<\/span> <span class=\"s\">'No suite description set'<\/span>\n\n        <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">path<\/span> <span class=\"o\">=<\/span> <span class=\"n\">suite<\/span><span class=\"p\">.<\/span><span class=\"n\">get<\/span><span class=\"p\">(<\/span><span class=\"s\">'SuitePath'<\/span><span class=\"p\">)<\/span>\n        <span class=\"k\">if<\/span> <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">path<\/span> <span class=\"ow\">is<\/span> <span class=\"bp\">None<\/span> <span class=\"ow\">or<\/span> <span class=\"nb\">len<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">path<\/span><span class=\"p\">)<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">0<\/span><span class=\"p\">:<\/span>\n            <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">path<\/span> <span class=\"o\">=<\/span> <span class=\"s\">'No suite path set'<\/span>\n\n        <span class=\"k\">for<\/span> <span class=\"n\">i<\/span><span class=\"p\">,<\/span> <span class=\"n\">report<\/span> <span class=\"ow\">in<\/span> <span class=\"nb\">enumerate<\/span><span class=\"p\">(<\/span><span class=\"n\">suite<\/span><span class=\"p\">.<\/span><span class=\"n\">get<\/span><span class=\"p\">(<\/span><span class=\"s\">'SpecReports'<\/span><span class=\"p\">,<\/span> <span class=\"p\">[])):<\/span>\n            <span class=\"k\">if<\/span> <span class=\"n\">report<\/span><span class=\"p\">.<\/span><span class=\"n\">get<\/span><span class=\"p\">(<\/span><span class=\"s\">'State'<\/span><span class=\"p\">)<\/span> <span class=\"o\">!=<\/span> <span class=\"s\">'passed'<\/span><span class=\"p\">:<\/span>\n                <span class=\"k\">continue<\/span>\n\n            <span class=\"k\">try<\/span><span class=\"p\">:<\/span>\n                <span class=\"n\">newreport<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Report<\/span><span class=\"p\">(<\/span><span class=\"n\">report<\/span><span class=\"p\">)<\/span>\n                <span class=\"n\">logging<\/span><span class=\"p\">.<\/span><span class=\"n\">info<\/span><span class=\"p\">(<\/span><span class=\"sa\">f<\/span><span class=\"s\">'processed report for <\/span><span class=\"si\">{<\/span><span class=\"n\">newreport<\/span><span class=\"p\">.<\/span><span class=\"n\">filename<\/span><span class=\"si\">}<\/span><span class=\"s\">@<\/span><span class=\"si\">{<\/span><span class=\"n\">newreport<\/span><span class=\"p\">.<\/span><span class=\"n\">linenumber<\/span><span class=\"si\">}<\/span><span class=\"s\">'<\/span><span class=\"p\">)<\/span>\n                <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">append_report<\/span><span class=\"p\">(<\/span><span class=\"n\">newreport<\/span><span class=\"p\">)<\/span>\n            <span class=\"k\">except<\/span> <span class=\"nb\">Exception<\/span> <span class=\"k\">as<\/span> <span class=\"n\">ex<\/span><span class=\"p\">:<\/span>\n                <span class=\"n\">nl<\/span> <span class=\"o\">=<\/span> <span class=\"n\">report<\/span><span class=\"p\">.<\/span><span class=\"n\">get<\/span><span class=\"p\">(<\/span><span class=\"s\">'LeafNodeLocation'<\/span><span class=\"p\">,<\/span> <span class=\"p\">{})<\/span>\n                <span class=\"n\">fn<\/span> <span class=\"o\">=<\/span> <span class=\"n\">nl<\/span><span class=\"p\">.<\/span><span class=\"n\">get<\/span><span class=\"p\">(<\/span><span class=\"s\">'FileName'<\/span><span class=\"p\">)<\/span>\n                <span class=\"n\">ln<\/span> <span class=\"o\">=<\/span> <span class=\"n\">nl<\/span><span class=\"p\">.<\/span><span class=\"n\">get<\/span><span class=\"p\">(<\/span><span class=\"s\">'LineNumber'<\/span><span class=\"p\">)<\/span>\n                <span class=\"n\">logging<\/span><span class=\"p\">.<\/span><span class=\"n\">error<\/span><span class=\"p\">(<\/span><span class=\"sa\">f<\/span><span class=\"s\">'error processing report for <\/span><span class=\"si\">{<\/span><span class=\"n\">fn<\/span><span class=\"si\">}<\/span><span class=\"s\">@<\/span><span class=\"si\">{<\/span><span class=\"n\">ln<\/span><span class=\"si\">}<\/span><span class=\"s\">'<\/span><span class=\"p\">)<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">append_report<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">report<\/span><span class=\"p\">):<\/span>\n        <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">reports<\/span><span class=\"p\">.<\/span><span class=\"n\">append<\/span><span class=\"p\">(<\/span><span class=\"n\">report<\/span><span class=\"p\">)<\/span>\n        <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">reports<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">sorted<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">reports<\/span><span class=\"p\">,<\/span> <span class=\"n\">key<\/span><span class=\"o\">=<\/span><span class=\"k\">lambda<\/span> <span class=\"n\">r<\/span><span class=\"p\">:<\/span> <span class=\"n\">r<\/span><span class=\"p\">.<\/span><span class=\"n\">hierarchy<\/span><span class=\"p\">)<\/span>\n\n\n<span class=\"k\">class<\/span> <span class=\"nc\">Report<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">__init__<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">report<\/span><span class=\"p\">):<\/span>\n        <span class=\"n\">hierarchy<\/span> <span class=\"o\">=<\/span> <span class=\"n\">report<\/span><span class=\"p\">.<\/span><span class=\"n\">get<\/span><span class=\"p\">(<\/span><span class=\"s\">'ContainerHierarchyTexts'<\/span><span class=\"p\">)<\/span>\n        <span class=\"k\">if<\/span> <span class=\"n\">hierarchy<\/span> <span class=\"ow\">is<\/span> <span class=\"bp\">None<\/span> <span class=\"ow\">or<\/span> <span class=\"nb\">len<\/span><span class=\"p\">(<\/span><span class=\"n\">hierarchy<\/span><span class=\"p\">)<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">0<\/span><span class=\"p\">:<\/span>\n            <span class=\"n\">hierarchy<\/span> <span class=\"o\">=<\/span> <span class=\"p\">[<\/span><span class=\"s\">'No hierarchy defined'<\/span><span class=\"p\">]<\/span>\n        <span class=\"n\">hierarchy<\/span> <span class=\"o\">=<\/span> <span class=\"s\">' \/ '<\/span><span class=\"p\">.<\/span><span class=\"n\">join<\/span><span class=\"p\">(<\/span><span class=\"n\">hierarchy<\/span><span class=\"p\">)<\/span>\n        <span class=\"k\">if<\/span> <span class=\"nb\">len<\/span><span class=\"p\">(<\/span><span class=\"n\">hierarchy<\/span><span class=\"p\">)<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">0<\/span><span class=\"p\">:<\/span>\n            <span class=\"n\">hierarchy<\/span> <span class=\"o\">=<\/span> <span class=\"s\">'No hierarchy text set'<\/span>\n        <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">hierarchy<\/span> <span class=\"o\">=<\/span> <span class=\"n\">hierarchy<\/span>\n        <span class=\"n\">logging<\/span><span class=\"p\">.<\/span><span class=\"n\">debug<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">hierarchy<\/span><span class=\"p\">)<\/span>\n\n        <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">text<\/span> <span class=\"o\">=<\/span> <span class=\"n\">report<\/span><span class=\"p\">.<\/span><span class=\"n\">get<\/span><span class=\"p\">(<\/span><span class=\"s\">'LeafNodeText'<\/span><span class=\"p\">)<\/span>\n        <span class=\"k\">if<\/span> <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">text<\/span> <span class=\"ow\">is<\/span> <span class=\"bp\">None<\/span><span class=\"p\">:<\/span>\n            <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">text<\/span> <span class=\"o\">=<\/span> <span class=\"s\">'No leaf node text set'<\/span>\n\n        <span class=\"n\">leafnodeloc<\/span> <span class=\"o\">=<\/span> <span class=\"n\">report<\/span><span class=\"p\">.<\/span><span class=\"n\">get<\/span><span class=\"p\">(<\/span><span class=\"s\">'LeafNodeLocation'<\/span><span class=\"p\">,<\/span> <span class=\"p\">{})<\/span>\n        <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">filename<\/span> <span class=\"o\">=<\/span> <span class=\"n\">leafnodeloc<\/span><span class=\"p\">.<\/span><span class=\"n\">get<\/span><span class=\"p\">(<\/span><span class=\"s\">'FileName'<\/span><span class=\"p\">)<\/span>\n        <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">linenumber<\/span> <span class=\"o\">=<\/span> <span class=\"n\">leafnodeloc<\/span><span class=\"p\">.<\/span><span class=\"n\">get<\/span><span class=\"p\">(<\/span><span class=\"s\">'LineNumber'<\/span><span class=\"p\">)<\/span>\n\n        <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">nodetype<\/span> <span class=\"o\">=<\/span> <span class=\"n\">report<\/span><span class=\"p\">.<\/span><span class=\"n\">get<\/span><span class=\"p\">(<\/span><span class=\"s\">'LeafNodeType'<\/span><span class=\"p\">,<\/span> <span class=\"s\">''<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>These next two classes, <code class=\"language-plaintext highlighter-rouge\">Suite<\/code> and <code class=\"language-plaintext highlighter-rouge\">Report<\/code>, are convenience wrappers that\nallow me to transform from the JSON format to an API that I can use when\ngenerating HTML pages. Where possible I try to have it fail gracefully with\ndefault messages that are easy to spot in the generated HTML content. I also\ncombine the hierarchy text into a more readable format and save the paths to\nthe individual source files with line numbers.<\/p>\n\n<h3 id=\"http\">HTTP<\/h3>\n\n<div class=\"language-python highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"k\">class<\/span> <span class=\"nc\">BilobaHttpRequestHandler<\/span><span class=\"p\">(<\/span><span class=\"n\">http<\/span><span class=\"p\">.<\/span><span class=\"n\">server<\/span><span class=\"p\">.<\/span><span class=\"n\">SimpleHTTPRequestHandler<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">do_GET<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">):<\/span>\n        <span class=\"n\">logging<\/span><span class=\"p\">.<\/span><span class=\"n\">info<\/span><span class=\"p\">(<\/span><span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">path<\/span><span class=\"p\">)<\/span>\n        <span class=\"k\">if<\/span> <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">path<\/span> <span class=\"o\">==<\/span> <span class=\"s\">'\/'<\/span><span class=\"p\">:<\/span>\n            <span class=\"n\">content<\/span> <span class=\"o\">=<\/span> <span class=\"n\">index<\/span>\n            <span class=\"n\">body<\/span> <span class=\"o\">=<\/span> <span class=\"n\">content<\/span><span class=\"p\">.<\/span><span class=\"n\">encode<\/span><span class=\"p\">(<\/span><span class=\"s\">'UTF-8'<\/span><span class=\"p\">,<\/span> <span class=\"s\">'replace'<\/span><span class=\"p\">)<\/span>\n            <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">send_response<\/span><span class=\"p\">(<\/span><span class=\"n\">HTTPStatus<\/span><span class=\"p\">.<\/span><span class=\"n\">OK<\/span><span class=\"p\">)<\/span>\n            <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">send_header<\/span><span class=\"p\">(<\/span><span class=\"s\">'Content-Type'<\/span><span class=\"p\">,<\/span> <span class=\"s\">'text\/html'<\/span><span class=\"p\">)<\/span>\n            <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">send_header<\/span><span class=\"p\">(<\/span><span class=\"s\">'Content-Length'<\/span><span class=\"p\">,<\/span> <span class=\"nb\">str<\/span><span class=\"p\">(<\/span><span class=\"nb\">len<\/span><span class=\"p\">(<\/span><span class=\"n\">body<\/span><span class=\"p\">)))<\/span>\n            <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">end_headers<\/span><span class=\"p\">()<\/span>\n            <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">wfile<\/span><span class=\"p\">.<\/span><span class=\"n\">write<\/span><span class=\"p\">(<\/span><span class=\"n\">body<\/span><span class=\"p\">)<\/span>\n        <span class=\"k\">elif<\/span> <span class=\"s\">'favicon'<\/span> <span class=\"ow\">in<\/span> <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">path<\/span><span class=\"p\">:<\/span>\n            <span class=\"nb\">super<\/span><span class=\"p\">().<\/span><span class=\"n\">do_GET<\/span><span class=\"p\">()<\/span>\n        <span class=\"k\">else<\/span><span class=\"p\">:<\/span>\n            <span class=\"c1\"># if not the index, then try to load the file and inject in an html wrapper\n<\/span>            <span class=\"k\">try<\/span><span class=\"p\">:<\/span>\n                <span class=\"n\">path<\/span><span class=\"p\">,<\/span> <span class=\"n\">param<\/span> <span class=\"o\">=<\/span> <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">path<\/span><span class=\"p\">.<\/span><span class=\"n\">split<\/span><span class=\"p\">(<\/span><span class=\"s\">'?'<\/span><span class=\"p\">,<\/span> <span class=\"n\">maxsplit<\/span><span class=\"o\">=<\/span><span class=\"mi\">1<\/span><span class=\"p\">)<\/span>\n                <span class=\"n\">linenumber<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">int<\/span><span class=\"p\">(<\/span><span class=\"n\">param<\/span><span class=\"p\">.<\/span><span class=\"n\">split<\/span><span class=\"p\">(<\/span><span class=\"s\">'='<\/span><span class=\"p\">,<\/span> <span class=\"n\">maxsplit<\/span><span class=\"o\">=<\/span><span class=\"mi\">1<\/span><span class=\"p\">)[<\/span><span class=\"mi\">1<\/span><span class=\"p\">])<\/span>\n                <span class=\"n\">logging<\/span><span class=\"p\">.<\/span><span class=\"n\">debug<\/span><span class=\"p\">(<\/span><span class=\"sa\">f<\/span><span class=\"s\">'attempting to load <\/span><span class=\"si\">{<\/span><span class=\"n\">path<\/span><span class=\"si\">}<\/span><span class=\"s\">, highlighting linenumber <\/span><span class=\"si\">{<\/span><span class=\"n\">linenumber<\/span><span class=\"si\">}<\/span><span class=\"s\">'<\/span><span class=\"p\">)<\/span>\n                <span class=\"n\">content<\/span> <span class=\"o\">=<\/span> <span class=\"s\">'&lt;pre&gt;<\/span><span class=\"se\">\\n<\/span><span class=\"s\">'<\/span>\n                <span class=\"k\">with<\/span> <span class=\"nb\">open<\/span><span class=\"p\">(<\/span><span class=\"n\">path<\/span><span class=\"p\">)<\/span> <span class=\"k\">as<\/span> <span class=\"n\">fp<\/span><span class=\"p\">:<\/span>\n                    <span class=\"n\">lines<\/span> <span class=\"o\">=<\/span> <span class=\"n\">fp<\/span><span class=\"p\">.<\/span><span class=\"n\">read<\/span><span class=\"p\">().<\/span><span class=\"n\">splitlines<\/span><span class=\"p\">()<\/span>\n                    <span class=\"k\">for<\/span> <span class=\"n\">i<\/span><span class=\"p\">,<\/span> <span class=\"n\">line<\/span> <span class=\"ow\">in<\/span> <span class=\"nb\">enumerate<\/span><span class=\"p\">(<\/span><span class=\"n\">lines<\/span><span class=\"p\">):<\/span>\n                        <span class=\"k\">if<\/span> <span class=\"n\">i<\/span><span class=\"o\">+<\/span><span class=\"mi\">1<\/span> <span class=\"o\">==<\/span> <span class=\"n\">linenumber<\/span><span class=\"p\">:<\/span>\n                            <span class=\"n\">content<\/span> <span class=\"o\">+=<\/span> <span class=\"sa\">f<\/span><span class=\"s\">'&lt;span id=\"highlighted-test\" class=\"highlight\"&gt;<\/span><span class=\"si\">{<\/span><span class=\"n\">line<\/span><span class=\"si\">}<\/span><span class=\"s\">&lt;\/span&gt;'<\/span>\n                        <span class=\"k\">else<\/span><span class=\"p\">:<\/span>\n                            <span class=\"n\">content<\/span> <span class=\"o\">+=<\/span> <span class=\"n\">line<\/span>\n                        <span class=\"n\">content<\/span> <span class=\"o\">+=<\/span> <span class=\"s\">'<\/span><span class=\"se\">\\n<\/span><span class=\"s\">'<\/span>\n                <span class=\"n\">content<\/span> <span class=\"o\">+=<\/span> <span class=\"s\">'&lt;\/pre&gt;'<\/span>\n                <span class=\"n\">content<\/span> <span class=\"o\">=<\/span> <span class=\"n\">html_template<\/span><span class=\"p\">.<\/span><span class=\"nb\">format<\/span><span class=\"p\">(<\/span><span class=\"n\">body<\/span><span class=\"o\">=<\/span><span class=\"n\">content<\/span><span class=\"p\">)<\/span>\n                <span class=\"n\">body<\/span> <span class=\"o\">=<\/span> <span class=\"n\">content<\/span><span class=\"p\">.<\/span><span class=\"n\">encode<\/span><span class=\"p\">(<\/span><span class=\"s\">'UTF-8'<\/span><span class=\"p\">,<\/span> <span class=\"s\">'replace'<\/span><span class=\"p\">)<\/span>\n                <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">send_response<\/span><span class=\"p\">(<\/span><span class=\"n\">HTTPStatus<\/span><span class=\"p\">.<\/span><span class=\"n\">OK<\/span><span class=\"p\">)<\/span>\n                <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">send_header<\/span><span class=\"p\">(<\/span><span class=\"s\">'Content-Type'<\/span><span class=\"p\">,<\/span> <span class=\"s\">'text\/html'<\/span><span class=\"p\">)<\/span>\n                <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">send_header<\/span><span class=\"p\">(<\/span><span class=\"s\">'Content-Length'<\/span><span class=\"p\">,<\/span> <span class=\"nb\">str<\/span><span class=\"p\">(<\/span><span class=\"nb\">len<\/span><span class=\"p\">(<\/span><span class=\"n\">body<\/span><span class=\"p\">)))<\/span>\n                <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">end_headers<\/span><span class=\"p\">()<\/span>\n                <span class=\"bp\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">wfile<\/span><span class=\"p\">.<\/span><span class=\"n\">write<\/span><span class=\"p\">(<\/span><span class=\"n\">body<\/span><span class=\"p\">)<\/span>\n            <span class=\"k\">except<\/span> <span class=\"nb\">Exception<\/span> <span class=\"k\">as<\/span> <span class=\"n\">ex<\/span><span class=\"p\">:<\/span>\n                <span class=\"n\">logging<\/span><span class=\"p\">.<\/span><span class=\"n\">debug<\/span><span class=\"p\">(<\/span><span class=\"n\">ex<\/span><span class=\"p\">)<\/span>\n                <span class=\"nb\">super<\/span><span class=\"p\">().<\/span><span class=\"n\">do_GET<\/span><span class=\"p\">()<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>This next part is where things get a little tricky. This class builds upon Python\u2019s\nstandard library implementation for the <a href=\"https:\/\/docs.python.org\/3\/library\/http.server.html#http.server.SimpleHTTPRequestHandler\">http.server.SimpleHTTPRequestHandler<\/a> to\ncreate a richer interface. By default, the <code class=\"language-plaintext highlighter-rouge\">SimpleHTTPRequestHandler<\/code> will create\nthe directory browser view that is familiar to anyone who has tried running\n<code class=\"language-plaintext highlighter-rouge\">python -m http.server<\/code> in their terminal (go try now if you haven\u2019t XD). But, in\nbiloba.py I\u2019d like to override that action when I see a request for <code class=\"language-plaintext highlighter-rouge\">\/<\/code>\nor any other URL that looks like a directory. I\u2019m overriding the <code class=\"language-plaintext highlighter-rouge\">do_GET<\/code> function\nof the base class so that I can inspect every HTTP GET request that is received.<\/p>\n\n<p>The <code class=\"language-plaintext highlighter-rouge\">if \/ elif \/ else<\/code> clause is where we choose to either send back the index page\nif the request is for the root, or ignore if a request for a favicon, and lastly\ntry to open the URL as a file. There is also some logic to pull out the line\nnumber parameter, if it exists, and then add the highlighted line to the code\nfile rendered template. If all else fails, or there is an exception, this function\nhands control over to the parent\u2019s implementation because it has much better support\nfor errors and erroneous input.<\/p>\n\n<h3 id=\"main\">Main<\/h3>\n\n<div class=\"language-python highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"k\">def<\/span> <span class=\"nf\">main<\/span><span class=\"p\">(<\/span><span class=\"n\">filename<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">fp<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">open<\/span><span class=\"p\">(<\/span><span class=\"n\">filename<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">report<\/span> <span class=\"o\">=<\/span> <span class=\"n\">json<\/span><span class=\"p\">.<\/span><span class=\"n\">load<\/span><span class=\"p\">(<\/span><span class=\"n\">fp<\/span><span class=\"p\">)<\/span>\n\n    <span class=\"n\">suites<\/span> <span class=\"o\">=<\/span> <span class=\"p\">[]<\/span>\n    <span class=\"k\">for<\/span> <span class=\"n\">i<\/span><span class=\"p\">,<\/span> <span class=\"n\">suite<\/span> <span class=\"ow\">in<\/span> <span class=\"nb\">enumerate<\/span><span class=\"p\">(<\/span><span class=\"n\">report<\/span><span class=\"p\">):<\/span>\n        <span class=\"k\">try<\/span><span class=\"p\">:<\/span>\n            <span class=\"n\">newsuite<\/span> <span class=\"o\">=<\/span> <span class=\"n\">Suite<\/span><span class=\"p\">(<\/span><span class=\"n\">suite<\/span><span class=\"p\">)<\/span>\n            <span class=\"n\">logging<\/span><span class=\"p\">.<\/span><span class=\"n\">info<\/span><span class=\"p\">(<\/span><span class=\"sa\">f<\/span><span class=\"s\">'created suite for <\/span><span class=\"si\">{<\/span><span class=\"n\">newsuite<\/span><span class=\"p\">.<\/span><span class=\"n\">path<\/span><span class=\"si\">}<\/span><span class=\"s\">'<\/span><span class=\"p\">)<\/span>\n            <span class=\"n\">suites<\/span><span class=\"p\">.<\/span><span class=\"n\">append<\/span><span class=\"p\">(<\/span><span class=\"n\">newsuite<\/span><span class=\"p\">)<\/span>\n        <span class=\"k\">except<\/span> <span class=\"nb\">Exception<\/span><span class=\"p\">:<\/span>\n            <span class=\"n\">logging<\/span><span class=\"p\">.<\/span><span class=\"n\">error<\/span><span class=\"p\">(<\/span><span class=\"sa\">f<\/span><span class=\"s\">'error processing suite at index <\/span><span class=\"si\">{<\/span><span class=\"n\">i<\/span><span class=\"si\">}<\/span><span class=\"s\">'<\/span><span class=\"p\">)<\/span>\n            <span class=\"k\">continue<\/span>\n\n    <span class=\"n\">body<\/span> <span class=\"o\">=<\/span> <span class=\"s\">''<\/span>\n    <span class=\"n\">suites<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">sorted<\/span><span class=\"p\">(<\/span><span class=\"n\">suites<\/span><span class=\"p\">,<\/span> <span class=\"n\">key<\/span><span class=\"o\">=<\/span><span class=\"k\">lambda<\/span> <span class=\"n\">s<\/span><span class=\"p\">:<\/span> <span class=\"n\">s<\/span><span class=\"p\">.<\/span><span class=\"n\">description<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">for<\/span> <span class=\"n\">suite<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">suites<\/span><span class=\"p\">:<\/span>\n        <span class=\"k\">if<\/span> <span class=\"nb\">len<\/span><span class=\"p\">(<\/span><span class=\"n\">suite<\/span><span class=\"p\">.<\/span><span class=\"n\">reports<\/span><span class=\"p\">)<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">0<\/span><span class=\"p\">:<\/span>\n            <span class=\"k\">continue<\/span>\n\n        <span class=\"n\">body<\/span> <span class=\"o\">+=<\/span> <span class=\"sa\">f<\/span><span class=\"s\">'&lt;h1&gt;<\/span><span class=\"si\">{<\/span><span class=\"n\">suite<\/span><span class=\"p\">.<\/span><span class=\"n\">description<\/span><span class=\"si\">}<\/span><span class=\"s\">&lt;\/h1&gt;<\/span><span class=\"se\">\\n<\/span><span class=\"s\">'<\/span>\n\n        <span class=\"n\">body<\/span> <span class=\"o\">+=<\/span> <span class=\"s\">'&lt;ul&gt;'<\/span>\n        <span class=\"k\">for<\/span> <span class=\"n\">report<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">suite<\/span><span class=\"p\">.<\/span><span class=\"n\">reports<\/span><span class=\"p\">:<\/span>\n            <span class=\"n\">body<\/span> <span class=\"o\">+=<\/span> <span class=\"sa\">f<\/span><span class=\"s\">'&lt;li&gt;<\/span><span class=\"si\">{<\/span><span class=\"n\">report<\/span><span class=\"p\">.<\/span><span class=\"n\">hierarchy<\/span><span class=\"si\">}<\/span><span class=\"s\">&lt;ul&gt;'<\/span>\n            <span class=\"k\">if<\/span> <span class=\"nb\">len<\/span><span class=\"p\">(<\/span><span class=\"n\">report<\/span><span class=\"p\">.<\/span><span class=\"n\">nodetype<\/span><span class=\"p\">)<\/span> <span class=\"o\">&gt;<\/span> <span class=\"mi\">0<\/span> <span class=\"p\">:<\/span>\n                <span class=\"n\">body<\/span> <span class=\"o\">+=<\/span> <span class=\"sa\">f<\/span><span class=\"s\">'&lt;li&gt;<\/span><span class=\"si\">{<\/span><span class=\"n\">report<\/span><span class=\"p\">.<\/span><span class=\"n\">nodetype<\/span><span class=\"si\">}<\/span><span class=\"s\"> <\/span><span class=\"si\">{<\/span><span class=\"n\">report<\/span><span class=\"p\">.<\/span><span class=\"n\">text<\/span><span class=\"si\">}<\/span><span class=\"s\">&lt;\/li&gt;'<\/span>\n            <span class=\"k\">else<\/span><span class=\"p\">:<\/span>\n                <span class=\"n\">body<\/span> <span class=\"o\">+=<\/span> <span class=\"sa\">f<\/span><span class=\"s\">'&lt;li&gt;<\/span><span class=\"si\">{<\/span><span class=\"n\">report<\/span><span class=\"p\">.<\/span><span class=\"n\">text<\/span><span class=\"si\">}<\/span><span class=\"s\">&lt;\/li&gt;'<\/span>\n            <span class=\"n\">body<\/span> <span class=\"o\">+=<\/span> <span class=\"sa\">f<\/span><span class=\"s\">'&lt;li&gt;&lt;a href=\"<\/span><span class=\"si\">{<\/span><span class=\"n\">report<\/span><span class=\"p\">.<\/span><span class=\"n\">filename<\/span><span class=\"si\">}<\/span><span class=\"s\">?linenumber=<\/span><span class=\"si\">{<\/span><span class=\"n\">report<\/span><span class=\"p\">.<\/span><span class=\"n\">linenumber<\/span><span class=\"si\">}<\/span><span class=\"s\">#highlighted-test\" target=\"_blank\"&gt;<\/span><span class=\"si\">{<\/span><span class=\"n\">report<\/span><span class=\"p\">.<\/span><span class=\"n\">filename<\/span><span class=\"si\">}<\/span><span class=\"s\">@<\/span><span class=\"si\">{<\/span><span class=\"n\">report<\/span><span class=\"p\">.<\/span><span class=\"n\">linenumber<\/span><span class=\"si\">}<\/span><span class=\"s\">&lt;\/a&gt;&lt;\/li&gt;&lt;\/ul&gt;&lt;\/li&gt;'<\/span>\n        <span class=\"n\">body<\/span> <span class=\"o\">+=<\/span> <span class=\"s\">'&lt;\/ul&gt;'<\/span>\n\n    <span class=\"k\">global<\/span> <span class=\"n\">index<\/span>\n    <span class=\"n\">index<\/span> <span class=\"o\">=<\/span> <span class=\"n\">html_template<\/span><span class=\"p\">.<\/span><span class=\"nb\">format<\/span><span class=\"p\">(<\/span><span class=\"n\">body<\/span><span class=\"o\">=<\/span><span class=\"n\">body<\/span><span class=\"p\">)<\/span>\n\n\n    <span class=\"n\">server_address<\/span> <span class=\"o\">=<\/span> <span class=\"p\">(<\/span><span class=\"s\">'127.0.0.1'<\/span><span class=\"p\">,<\/span> <span class=\"mi\">8080<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">httpd<\/span> <span class=\"o\">=<\/span> <span class=\"n\">http<\/span><span class=\"p\">.<\/span><span class=\"n\">server<\/span><span class=\"p\">.<\/span><span class=\"n\">HTTPServer<\/span><span class=\"p\">(<\/span><span class=\"n\">server_address<\/span><span class=\"p\">,<\/span> <span class=\"n\">BilobaHttpRequestHandler<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">try<\/span><span class=\"p\">:<\/span>\n        <span class=\"k\">print<\/span><span class=\"p\">(<\/span><span class=\"s\">'serving at http:\/\/127.0.0.1:8080\/'<\/span><span class=\"p\">)<\/span>\n        <span class=\"n\">httpd<\/span><span class=\"p\">.<\/span><span class=\"n\">serve_forever<\/span><span class=\"p\">()<\/span>\n    <span class=\"k\">except<\/span> <span class=\"nb\">KeyboardInterrupt<\/span><span class=\"p\">:<\/span>\n        <span class=\"n\">logging<\/span><span class=\"p\">.<\/span><span class=\"n\">warning<\/span><span class=\"p\">(<\/span><span class=\"s\">'<\/span><span class=\"se\">\\n<\/span><span class=\"s\">Keyboard interrupt received, exiting...'<\/span><span class=\"p\">)<\/span>\n        <span class=\"n\">sys<\/span><span class=\"p\">.<\/span><span class=\"nb\">exit<\/span><span class=\"p\">(<\/span><span class=\"mi\">0<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>I like to build my Python applications with a <code class=\"language-plaintext highlighter-rouge\">main<\/code> function, mainly to help\nme remember where the whole thing starts. In this case, the main does a couple\nthings. First it tries to process the file name it is given as a JSON file and\ncreate code versions of the suites and reports that exist within it, also sorting\nthese by name.<\/p>\n\n<p>Next, it creates the HTML for the index page, with its list of test names and\nlinks to the individual code files.<\/p>\n\n<p>Lastly, it starts Python\u2019s standard HTTP server using the custom handler class\nas the default handler. This server is not meant for production use cases, but\nrunning it locally in development is perfectly fine for my use case. I also\nwrap this command with an exception handler to make the eventual \u201ccontrol-c\u201d quit\na little more tidy.<\/p>\n\n<h3 id=\"boilerplate\">Boilerplate<\/h3>\n\n<div class=\"language-python highlighter-rouge\"><div class=\"highlight\"><pre class=\"highlight\"><code><span class=\"k\">if<\/span> <span class=\"n\">__name__<\/span> <span class=\"o\">==<\/span> <span class=\"s\">'__main__'<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">parser<\/span> <span class=\"o\">=<\/span> <span class=\"n\">argparse<\/span><span class=\"p\">.<\/span><span class=\"n\">ArgumentParser<\/span><span class=\"p\">(<\/span><span class=\"n\">description<\/span><span class=\"o\">=<\/span><span class=\"s\">\"Reformat info from a Ginkgo test report JSON file\"<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">parser<\/span><span class=\"p\">.<\/span><span class=\"n\">add_argument<\/span><span class=\"p\">(<\/span><span class=\"s\">'filename'<\/span><span class=\"p\">,<\/span> <span class=\"n\">help<\/span><span class=\"o\">=<\/span><span class=\"s\">'the json file to process'<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">parser<\/span><span class=\"p\">.<\/span><span class=\"n\">add_argument<\/span><span class=\"p\">(<\/span><span class=\"s\">'--debug'<\/span><span class=\"p\">,<\/span> <span class=\"n\">action<\/span><span class=\"o\">=<\/span><span class=\"s\">'store_true'<\/span><span class=\"p\">,<\/span> <span class=\"n\">help<\/span><span class=\"o\">=<\/span><span class=\"s\">'turn on debug logging'<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">args<\/span> <span class=\"o\">=<\/span> <span class=\"n\">parser<\/span><span class=\"p\">.<\/span><span class=\"n\">parse_args<\/span><span class=\"p\">()<\/span>\n    <span class=\"k\">if<\/span> <span class=\"n\">args<\/span><span class=\"p\">.<\/span><span class=\"n\">debug<\/span><span class=\"p\">:<\/span>\n        <span class=\"n\">logging<\/span><span class=\"p\">.<\/span><span class=\"n\">basicConfig<\/span><span class=\"p\">(<\/span><span class=\"n\">level<\/span><span class=\"o\">=<\/span><span class=\"n\">logging<\/span><span class=\"p\">.<\/span><span class=\"n\">DEBUG<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">main<\/span><span class=\"p\">(<\/span><span class=\"n\">args<\/span><span class=\"p\">.<\/span><span class=\"n\">filename<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre><\/div><\/div>\n\n<p>Finally comes the Python boilerplate we are all familiar with when writing\napplication start code. I also tend to add my argument configuration stuff in\nthese wrappers, and then pass in the arguments I need to the main function.<\/p>\n\n<h2 id=\"whats-next-for-the-tiny-bilobapy\">What\u2019s next for the tiny biloba.py<\/h2>\n\n<p>I fixed one minor thing along the way, which turned out to be a 2 line change,\nbut otherwise I\u2019m actively resisting putting more time in on it. I\u2019m not sure\nI will need it again, although if I do then I will probably add a little\nstyling to the code pages in the form of line numbers and maybe a little background color.<\/p>\n\n<p>Another thought I had was to add the ability for biloba.py to run the ginkgo\ncommand and harvest the output to a temporary file. I\u2019m not quite sure if that\nwould be useful, but I think if I start to do more of these \u201cgrep\u201d style runs\nthen I might add that.<\/p>\n\n<p>I was super stoked with building this little application, it really turned what\nlooked like several mountains of work into something that was very manageable.\nThe power of modern tools like Python, and many other languages which I could have\nused, has really amazed me over the years. I encourage every out there to grow\ntheir tool box, whether with Python, another language, or even an entirely different\npiece of software all together. If you <strong>are<\/strong> looking for some place to learn Python, check out\n<a href=\"https:\/\/ocw.mit.edu\/courses\/6-0001-introduction-to-computer-science-and-programming-in-python-fall-2016\/\">MIT\u2019s Introduction to Computer Science and Programming in Python<\/a>.\nIt\u2019s a free course with videos and assignment material, it does use an older\nversion of Python but all the core principles are still useful. I hope you get\nout there and have some fun building your tool box out, and as always\nhappy hacking =)<\/p>\n","pubDate":"Sun, 18 Dec 2022 00:00:00 +0000","link":"https:\/\/notes.elmiko.dev\/2022\/12\/18\/why-i-keep-python-in-the-tool-box.html","guid":"https:\/\/notes.elmiko.dev\/2022\/12\/18\/why-i-keep-python-in-the-tool-box.html"}]}}