{"@attributes":{"version":"2.0"},"channel":{"title":"DEV Community: Prayson Wilfred Daniel","description":"The latest articles on DEV Community by Prayson Wilfred Daniel (@proteusiq).","link":"https:\/\/dev.to\/proteusiq","image":{"url":"https:\/\/media2.dev.to\/dynamic\/image\/width=90,height=90,fit=cover,gravity=auto,format=auto\/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F123003%2F3f713759-b0fb-4baa-928c-1f2351eaf701.JPG","title":"DEV Community: Prayson Wilfred Daniel","link":"https:\/\/dev.to\/proteusiq"},"language":"en","item":[{"title":"TDD = Communicate + Connect + Craft","pubDate":"Sat, 15 Mar 2025 16:10:25 +0000","link":"https:\/\/dev.to\/proteusiq\/tdd-communicate-connect-craft-3ndn","guid":"https:\/\/dev.to\/proteusiq\/tdd-communicate-connect-craft-3ndn","description":"<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fyzcz3b44bvib0wty0wid.jpeg\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fyzcz3b44bvib0wty0wid.jpeg\" alt=\"meh\" width=\"800\" height=\"533\"><\/a><\/p>\n\n<p>\ud83e\udd8a Let us dispel a common misconception: the essence of Test-Driven Development(TDD) is not about the act of writing tests. <\/p>\n\n<p>True, when we emerge from the TDD process, we hold in our hands a suite of tests\u2014yet these are not the destination, but rather a byproduct of a far grander pursuit. <\/p>\n\n<p>The true aim of TDD is to serve as a compass, guiding us through the intricate journey of creation. It is the architectural blueprint guiding our path.<\/p>\n\n<p>TDD is a language of connection\u2014an invitation extended to fellow developers and stakeholders to step into the theater of our minds, to see the world as we envision it. <\/p>\n\n<p>It embodies a programming paradigm where the very design of our solutions emerges organically from the use cases we seek to address. <\/p>\n\n<p>Through this disciplined dance of code and intent, we craft not just software, but a shared understanding\u2014a harmony of purpose and execution.<\/p>\n\n<p>TDD is not a rule. It is a value. A value to help us find direction as a community. After all, coding is a community endeavour.<\/p>\n\n","category":"programming"},{"title":"python-dotenv will bite you. Watch out","pubDate":"Wed, 05 Mar 2025 21:22:48 +0000","link":"https:\/\/dev.to\/proteusiq\/python-dotenv-will-bite-you-watch-out-19od","guid":"https:\/\/dev.to\/proteusiq\/python-dotenv-will-bite-you-watch-out-19od","description":"<div class=\"ltag__link\">\n  <a href=\"\/proteusiq\" class=\"ltag__link__link\">\n    <div class=\"ltag__link__pic\">\n      <img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F123003%2F3f713759-b0fb-4baa-928c-1f2351eaf701.JPG\" alt=\"proteusiq\">\n    <\/div>\n  <\/a>\n  <a href=\"https:\/\/dev.to\/proteusiq\/trending-anti-pattern-loading-environments-j55\" class=\"ltag__link__link\">\n    <div class=\"ltag__link__content\">\n      <h2>Why `load_dotenv()` Is an Anti-Pattern\u2014and How to Do Better<\/h2>\n      <h3>Prayson Wilfred Daniel \u30fb Mar 5<\/h3>\n      <div class=\"ltag__link__taglist\">\n        <span class=\"ltag__link__tag\">#python<\/span>\n        <span class=\"ltag__link__tag\">#ai<\/span>\n        <span class=\"ltag__link__tag\">#programming<\/span>\n      <\/div>\n    <\/div>\n  <\/a>\n<\/div>\n\n\n","category":["python","ai","programming"]},{"title":"Why `load_dotenv()` Is an Anti-Pattern\u2014and How to Do Better","pubDate":"Wed, 05 Mar 2025 16:43:21 +0000","link":"https:\/\/dev.to\/proteusiq\/trending-anti-pattern-loading-environments-j55","guid":"https:\/\/dev.to\/proteusiq\/trending-anti-pattern-loading-environments-j55","description":"<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fyrpkwa55im2oa9apk4d2.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fyrpkwa55im2oa9apk4d2.png\" alt=\"No way\" width=\"800\" height=\"457\"><\/a><\/p>\n\n<p>If you\u2019ve ever written <code>from dotenv import load_dotenv; load_dotenv()<\/code> at the top of a Python script, you\u2019re not alone. It\u2019s a rite of passage for developers managing environment variables\u2014until it bites you. <\/p>\n\n<p>This little convenience from <code>python-dotenv<\/code> promises to load your <code>.env<\/code> file into <code>os.environ<\/code> with minimal fuss, but beneath its simplicity lurks a mess of anti-patterns that clash with Python\u2019s core principles. <\/p>\n\n<p>It\u2019s implicit, fragile, and a maintenance nightmare. Let\u2019s tear it apart and build something better with Pydantic, <code>python-decouple<\/code>, and good old <code>os.getenv()<\/code>.<\/p>\n\n<h2>\n  \n  \n  The Sins of <code>load_dotenv()<\/code>\n<\/h2>\n\n<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F1ts9m2dgyrp11fwhzbqn.jpeg\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F1ts9m2dgyrp11fwhzbqn.jpeg\" alt=\"Noob\" width=\"800\" height=\"600\"><\/a><\/p>\n\n<h3>\n  \n  \n  It\u2019s Un-Pythonic Magic\n<\/h3>\n\n<p>Python\u2019s Zen (PEP 20) chants \u201cexplicit is better than implicit,\u201d but <code>load_dotenv()<\/code> is a sorcerer\u2019s trick. Call it, and it silently mutates <code>os.environ<\/code>\u2014no meaningful return value, no clear signal of success. <\/p>\n\n<p>Did it find <code>.env<\/code>? <br>\nDid it override existing vars? <\/p>\n\n<p>You\u2019re left guessing.<\/p>\n\n<p>With <code>load_dotenv()<\/code>, you\u2019re stuck probing <code>os.environ<\/code> to see what happened. That\u2019s not how Python rolls.<\/p>\n<h3>\n  \n  \n  No meaningful \"Return\", Just Side Effects\n<\/h3>\n\n<p>A good function gives you something back. <code>load_dotenv()<\/code>? Meh! It returns <code>boolean<\/code>. Is that useful? Nope. It alters global state like a rogue script kiddie. You call it, and your environment changes behind a curtain. <\/p>\n\n<p>Want to know what it loaded? Tough luck\u2014go fish in <code>os.environ<\/code>. <\/p>\n\n<p>A Pythonic version might return a dict of loaded values, letting <em>you<\/em> decide what to do next. Instead, it\u2019s a void that forces trust over transparency.<\/p>\n<h3>\n  \n  \n  It Breaks Single Responsibility\n<\/h3>\n\n<p>The Single Responsibility Principle says one function, one job. <code>load_dotenv()<\/code> laughs at that. It locates <code>.env<\/code>, parses it, decides override rules, and shoves everything into <code>os.environ<\/code>\u2014all in one opaque blob. <\/p>\n\n<p>That\u2019s four responsibilities mashed together, making it brittle and hard to debug. A saner design would split those steps, but no, it\u2019s a monolith that does too much and hides too much.<\/p>\n<h3>\n  \n  \n  Fragile in Production\n<\/h3>\n\n<p>In dev, <code>.env<\/code> files are cute. In production? They\u2019re a liability. Platforms like Docker or Heroku inject environment variables directly\u2014no file needed. <code>load_dotenv()<\/code> ties you to a file-based workflow that falls apart outside your laptop. <\/p>\n\n<p>Worse, its order-dependence (call it too late, and imported modules miss the vars) turns your codebase into a timing puzzle. It\u2019s a dev crutch that doesn\u2019t scale.<\/p>\n<h2>\n  \n  \n  The Better Way: Three Pythonic Alternatives\n<\/h2>\n\n<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fvvt7cbvrl687pzn7m3e2.jpeg\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fvvt7cbvrl687pzn7m3e2.jpeg\" alt=\"Here is a black and white illustration featuring a Mandalorian helmet with the phrase \" width=\"800\" height=\"600\"><\/a><\/p>\n\n<p>Ditch the magic. Here are three solutions that respect Python\u2019s ethos, keep your code clean, and work from dev to prod.<\/p>\n<h3>\n  \n  \n  1. Pydantic Settings: The Modern Powerhouse\n<\/h3>\n\n<p>Pydantic\u2019s <code>BaseSettings<\/code> (or <code>BaseModel<\/code> with settings config in 2.x) brings type safety and clarity to your config game.<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"n\">pydantic_settings<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">BaseSettings<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">Settings<\/span><span class=\"p\">(<\/span><span class=\"n\">BaseSettings<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">database_url<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span>\n    <span class=\"n\">api_key<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span>\n\n    <span class=\"k\">class<\/span> <span class=\"nc\">Config<\/span><span class=\"p\">:<\/span>\n        <span class=\"n\">env_file<\/span> <span class=\"o\">=<\/span> <span class=\"sh\">\"<\/span><span class=\"s\">.env<\/span><span class=\"sh\">\"<\/span>  <span class=\"c1\"># Optional\n<\/span>\n<span class=\"n\">settings<\/span> <span class=\"o\">=<\/span> <span class=\"nc\">Settings<\/span><span class=\"p\">()<\/span>\n<span class=\"nf\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">settings<\/span><span class=\"p\">.<\/span><span class=\"n\">database_url<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p><strong>Why It Rocks<\/strong>:<\/p>\n\n<ul>\n<li>Explicitly defines what you need, with types.<\/li>\n<li>Pulls from env vars or <code>.env<\/code> without mutating globals.<\/li>\n<li>Raises loud errors if something\u2019s missing (no silent fails).<\/li>\n<li>Scales seamlessly to production\u2014no file dependency required.<\/li>\n<\/ul>\n\n<p><strong>Trade-Off<\/strong>: Adds <code>pydantic<\/code> as a dependency, but the robustness is worth it.<\/p>\n\n<h3>\n  \n  \n  2. Python-Decouple: Lightweight and Pragmatic\n<\/h3>\n\n<p><code>python-decouple<\/code> keeps it simple, fetching values without the bloat.<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"n\">decouple<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">config<\/span>\n\n<span class=\"n\">database_url<\/span> <span class=\"o\">=<\/span> <span class=\"nf\">config<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">DATABASE_URL<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n<span class=\"n\">api_key<\/span> <span class=\"o\">=<\/span> <span class=\"nf\">config<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">API_KEY<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">default<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">fallback<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">cast<\/span><span class=\"o\">=<\/span><span class=\"nb\">str<\/span><span class=\"p\">)<\/span>\n<span class=\"nf\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">database_url<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p><strong>Why It Rocks<\/strong>:<\/p>\n\n<ul>\n<li>Explicit key-by-key access\u2014no blanket loads.<\/li>\n<li>Returns values, not side effects.<\/li>\n<li>Supports defaults for resilience.<\/li>\n<li>Leaner than Pydantic, still avoids <code>load_dotenv()<\/code>\u2019s traps.<\/li>\n<\/ul>\n\n<p><strong>Trade-Off<\/strong>: Another dependency, less validation than Pydantic.<\/p>\n\n<h3>\n  \n  \n  3. Export + <code>os.getenv()<\/code>: The No-Dependency Classic\n<\/h3>\n\n<p>Skip libraries. Set vars with <code>export<\/code> (or your platform\u2019s equivalent) and use Python\u2019s built-in <code>os.getenv()<\/code>.<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"n\">os<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">getenv<\/span>\n\n<span class=\"n\">database_url<\/span> <span class=\"o\">=<\/span> <span class=\"nf\">getenv<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">DATABASE_URL<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n<span class=\"n\">api_key<\/span> <span class=\"o\">=<\/span> <span class=\"nf\">getenv<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">API_KEY<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span> <span class=\"sh\">\"<\/span><span class=\"s\">default_value<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n<span class=\"nf\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">database_url<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p><strong>Why It Rocks<\/strong>:<\/p>\n\n<ul>\n<li>Zero dependencies\u2014pure Python.<\/li>\n<li>Explicit reads, no hidden writes.<\/li>\n<li>Works anywhere env vars are set (shell, Docker, CI).<\/li>\n<li>Total control, no file assumptions.<\/li>\n<\/ul>\n\n<p><strong>Trade-Off<\/strong>: Manual setup in dev (e.g., <code>export MY_VAR=value<\/code>), no <code>.env<\/code> parsing out of the box.<\/p>\n\n<h2>\n  \n  \n  Making It Work for You\n<\/h2>\n\n<ul>\n<li>\n<strong>Big Project?<\/strong> Pydantic\u2019s your friend\u2014structured, typed, future-proof.<\/li>\n<li>\n<strong>Quick and Dirty?<\/strong> <code>python-decouple<\/code> gets you there without fuss.<\/li>\n<li>\n<strong>Minimalist?<\/strong> <code>os.getenv()<\/code> with a shell script (e.g., <code>source .env<\/code>) keeps it raw and real.<\/li>\n<\/ul>\n\n<p>For dev convenience, Pydantic and <code>python-decouple<\/code> can still use <code>.env<\/code> files\u2014but they don\u2019t force it. With <code>os.getenv()<\/code>, pair it with a script like:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight shell\"><code><span class=\"c\">#!\/bin\/bash<\/span>\n<span class=\"nb\">set<\/span> <span class=\"nt\">-a<\/span>\n<span class=\"nb\">source<\/span> .env\n<span class=\"nb\">set<\/span> +a\npython app.py\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<h2>\n  \n  \n  The Verdict\n<\/h2>\n\n<p><code>load_dotenv()<\/code> isn\u2019t evil\u2014it\u2019s just lazy. It trades short-term ease for long-term pain, breaking Python\u2019s clarity and control. Swap it for Pydantic\u2019s polish, <code>python-decouple<\/code>\u2019s simplicity, or <code>os.getenv()<\/code>\u2019s purity. Your code will thank you, your team will thank you, and you\u2019ll never debug a mysterious env var again.<\/p>\n\n<p>What\u2019s your pick? Drop a comment\u2014I\u2019m betting on Pydantic for the win.<\/p>\n\n\n\n\n","category":["python","ai","programming"]},{"title":"Organic Coding: Safeguarding the Future of AI Code-Assistance Tools","pubDate":"Sat, 11 May 2024 10:55:34 +0000","link":"https:\/\/dev.to\/proteusiq\/organic-coding-safeguarding-the-future-of-ai-code-assistance-tools-cc6","guid":"https:\/\/dev.to\/proteusiq\/organic-coding-safeguarding-the-future-of-ai-code-assistance-tools-cc6","description":"<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Felnld9dh20r7p7mjp80o.jpeg\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Felnld9dh20r7p7mjp80o.jpeg\" alt=\"woman dale-3\" width=\"800\" height=\"457\"><\/a><br>\nObserving the steady rise of AI code-assistance tools such as OpenAI\u2019s ChatGPT and Codex, GitHub\u2019s Co-pilot, Replit\u2019s Ghostwriter, in the developer community, I am experiencing a mix of excitement and concern. <\/p>\n\n<p>It's encouraging to see how swiftly these tools provide answers and facilitate software development. The speed and quality are unparalleled, painting a promising future for code generation and software development. <\/p>\n\n<p>On the other hand, I am concerned. My Stack Overflow\u2019s  reputation is at the top 2%. This is alarming as I have hardly contributed this year. This could be a sign that more and more developers are turning to AI code-assistance tools rather than community code-assistance forums.<\/p>\n\n<p>My main concern lies in the source of knowledge used to train these increasingly proficient models. If the vibrant communities that generated and updates these knowledge base\/data used to train AI code assistance tools were to dwindle, we risk stunting our creative problem-solving methods. This concern comes to the fore as we stride towards a singularity, where there's no apparent further improvement in coding.<\/p>\n\n<p>Herein lies the importance of \"Organic Coding,\" a concept I coined to address this very concern. It call for a group of rebels that are willing to revert or stay to manual coding, not to reject AI, but rather as a means to provide high-quality, AI-free generated code for training AI systems. The goal is to ensure the evolution, growth, and improvement of our AI tools, without the risk of them becoming self-replicating error machines.<\/p>\n\n<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fyix4m56bnyevtp7o1cqg.jpeg\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fyix4m56bnyevtp7o1cqg.jpeg\" alt=\"dalle-3\" width=\"800\" height=\"457\"><\/a><\/p>\n\n<p>The Organic Coding approach preserves and utilizes the ingenuity, creativity, and critical thinking abilities of a community of human coders. This approach ensures our AI systems continue to learn from diverse and innovative coding solutions, thus avoiding the pitfall of learning from their self-generated code, which could limit innovative problem-solving approaches.<\/p>\n\n<p>What we need, moving forward, is a dedicated group of \"green-coders\" or \"organic coders,\" who code deliberately without AI assistance. Their contributions will form the training backbone for these AI tools, safeguarding the continuous improvement of the technological solutions we depend on.<\/p>\n\n<p>While the future of AI code-assist tools is undeniably bright, it's essential that we nurture this progression with careful consideration. Through Organic Coding, we can ensure that the vibrant communities of developers continue to play a crucial role in shaping the AI systems of the future.<\/p>\n\n<p><code>until then<\/code>, keep on coding, organically \ud83e\udee3<\/p>\n\n"},{"title":"In Rust for Python: A Match from Heaven","pubDate":"Wed, 03 Jan 2024 12:28:14 +0000","link":"https:\/\/dev.to\/proteusiq\/in-rust-for-python-a-match-from-heaven-51fn","guid":"https:\/\/dev.to\/proteusiq\/in-rust-for-python-a-match-from-heaven-51fn","description":"<p>Python, much like the cheerful Flounder, is known for its friendliness and accessibility. It darts through development waters with ease and grace, beloved by many for its simplicity. But every now and then, Python finds the currents of performance and efficiency challenging to navigate.<\/p>\n\n<p>Enter Rust, the Sebastian of the coding sea. Wise and performance-oriented, Rust offers unparalleled safety and speed. However, Rust's strictness can sometimes feel confining, contrasting with Python's free-flowing nature. <\/p>\n\n<p>A match made in heaven, or perhaps more fittingly, under the ocean, occurs when Rust guides Python through the trickier, more performance-intensive waters<\/p>\n\n<p>This story unfolds as a captivating journey where the agile Flounder, representing the Python programming language, navigates the vast seas of coding under the wise guidance of Sebastian, symbolizing Rust. Central to their adventure are three powerful tridents: cargo, <a href=\"https:\/\/github.com\/PyO3\/pyo3\" rel=\"noopener noreferrer\">PyO3<\/a>, and <a href=\"https:\/\/github.com\/PyO3\/maturin\" rel=\"noopener noreferrer\">maturin<\/a>. <\/p>\n\n\n\n\n<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fj0qe4ic6o166rg6g08hm.jpeg\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fj0qe4ic6o166rg6g08hm.jpeg\" alt=\"crab snake\" width=\"800\" height=\"457\"><\/a><\/p>\n\n<h3>\n  \n  \n  Chapter 1: The Call for Optimization\n<\/h3>\n\n<p>In our tale, Python seeks to enhance its capabilities, longing for the speed and efficiency needed to process a complex task: implementing a k-Nearest Neighbors (kNN) algorithm. The kNN, simple yet effective, is the perfect test for combining Python's simplicity with Rust's performance.<\/p>\n\n<p>\n  for complete codes, project structure  and requirements\n  <ul>\n<li><p><a href=\"https:\/\/github.com\/Proteusiq\/hadithi\/tree\/main\/dev.io\/knn\" rel=\"noopener noreferrer\">complete code and structure code<\/a> <\/p><\/li>\n<li><p>installation steps<br>\n<\/p><\/li>\n<\/ul>\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight shell\"><code><span class=\"c\"># rust, pixi and python<\/span>\ncurl <span class=\"nt\">--proto<\/span> <span class=\"s1\">'=https'<\/span> <span class=\"nt\">--tlsv1<\/span>.2 <span class=\"nt\">-sSf<\/span> https:\/\/sh.rustup.rs | sh\ncurl <span class=\"nt\">-fsSL<\/span> https:\/\/pixi.sh\/install.sh | bash\n\n<span class=\"c\"># download the repository knn and install<\/span>\npixi <span class=\"nb\">install \n<\/span>pixi run maturin build <span class=\"o\">&amp;&amp;<\/span> pixi run python <span class=\"nt\">-m<\/span> use.ml\n<\/code><\/pre>\n\n<\/div>\n\n\n\n\n<\/p>\n\n<p>Rust kNN<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight rust\"><code><span class=\"c1\">\/\/ src\/crab.rc<\/span>\n<span class=\"k\">use<\/span> <span class=\"nn\">ndarray<\/span><span class=\"p\">::<\/span><span class=\"n\">Array1<\/span><span class=\"p\">;<\/span>\n<span class=\"k\">use<\/span> <span class=\"nn\">std<\/span><span class=\"p\">::<\/span><span class=\"nn\">collections<\/span><span class=\"p\">::<\/span><span class=\"n\">HashMap<\/span><span class=\"p\">;<\/span>\n\n<span class=\"c1\">\/\/ calculates the Euclidean distance between two points<\/span>\n<span class=\"k\">pub<\/span> <span class=\"k\">fn<\/span> <span class=\"nf\">euclidean_distance<\/span><span class=\"p\">(<\/span><span class=\"n\">x1<\/span><span class=\"p\">:<\/span> <span class=\"o\">&amp;<\/span><span class=\"n\">Array1<\/span><span class=\"o\">&lt;<\/span><span class=\"nb\">f64<\/span><span class=\"o\">&gt;<\/span><span class=\"p\">,<\/span> <span class=\"n\">x2<\/span><span class=\"p\">:<\/span> <span class=\"o\">&amp;<\/span><span class=\"n\">Array1<\/span><span class=\"o\">&lt;<\/span><span class=\"nb\">f64<\/span><span class=\"o\">&gt;<\/span><span class=\"p\">)<\/span> <span class=\"k\">-&gt;<\/span> <span class=\"nb\">f64<\/span> <span class=\"p\">{<\/span>\n    <span class=\"p\">(<\/span><span class=\"n\">x1<\/span> <span class=\"o\">-<\/span> <span class=\"n\">x2<\/span><span class=\"p\">)<\/span><span class=\"nf\">.mapv<\/span><span class=\"p\">(|<\/span><span class=\"n\">a<\/span><span class=\"p\">|<\/span> <span class=\"n\">a<\/span><span class=\"nf\">.powi<\/span><span class=\"p\">(<\/span><span class=\"mi\">2<\/span><span class=\"p\">))<\/span><span class=\"nf\">.sum<\/span><span class=\"p\">()<\/span><span class=\"nf\">.sqrt<\/span><span class=\"p\">()<\/span>\n<span class=\"p\">}<\/span>\n\n<span class=\"k\">pub<\/span> <span class=\"k\">struct<\/span> <span class=\"n\">KNN<\/span> <span class=\"p\">{<\/span>\n    <span class=\"n\">k<\/span><span class=\"p\">:<\/span> <span class=\"nb\">usize<\/span><span class=\"p\">,<\/span> <span class=\"c1\">\/\/ number of nearest neighbors<\/span>\n    <span class=\"n\">x_train<\/span><span class=\"p\">:<\/span> <span class=\"nb\">Option<\/span><span class=\"o\">&lt;<\/span><span class=\"nb\">Vec<\/span><span class=\"o\">&lt;<\/span><span class=\"n\">Array1<\/span><span class=\"o\">&lt;<\/span><span class=\"nb\">f64<\/span><span class=\"o\">&gt;&gt;&gt;<\/span><span class=\"p\">,<\/span>\n    <span class=\"n\">y_train<\/span><span class=\"p\">:<\/span> <span class=\"nb\">Option<\/span><span class=\"o\">&lt;<\/span><span class=\"nb\">Vec<\/span><span class=\"o\">&lt;<\/span><span class=\"nb\">i32<\/span><span class=\"o\">&gt;&gt;<\/span><span class=\"p\">,<\/span>\n<span class=\"p\">}<\/span>\n\n<span class=\"k\">impl<\/span> <span class=\"n\">KNN<\/span> <span class=\"p\">{<\/span>\n    <span class=\"k\">pub<\/span> <span class=\"k\">fn<\/span> <span class=\"nf\">new<\/span><span class=\"p\">(<\/span><span class=\"n\">k<\/span><span class=\"p\">:<\/span> <span class=\"nb\">usize<\/span><span class=\"p\">)<\/span> <span class=\"k\">-&gt;<\/span> <span class=\"n\">KNN<\/span> <span class=\"p\">{<\/span>\n        <span class=\"n\">KNN<\/span> <span class=\"p\">{<\/span>\n            <span class=\"n\">k<\/span><span class=\"p\">,<\/span>\n            <span class=\"n\">x_train<\/span><span class=\"p\">:<\/span> <span class=\"nb\">None<\/span><span class=\"p\">,<\/span>\n            <span class=\"n\">y_train<\/span><span class=\"p\">:<\/span> <span class=\"nb\">None<\/span><span class=\"p\">,<\/span>\n        <span class=\"p\">}<\/span>\n    <span class=\"p\">}<\/span>\n\n    <span class=\"k\">pub<\/span> <span class=\"k\">fn<\/span> <span class=\"nf\">fit<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"k\">mut<\/span> <span class=\"k\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">x<\/span><span class=\"p\">:<\/span> <span class=\"nb\">Vec<\/span><span class=\"o\">&lt;<\/span><span class=\"n\">Array1<\/span><span class=\"o\">&lt;<\/span><span class=\"nb\">f64<\/span><span class=\"o\">&gt;&gt;<\/span><span class=\"p\">,<\/span> <span class=\"n\">y<\/span><span class=\"p\">:<\/span> <span class=\"nb\">Vec<\/span><span class=\"o\">&lt;<\/span><span class=\"nb\">i32<\/span><span class=\"o\">&gt;<\/span><span class=\"p\">)<\/span> <span class=\"p\">{<\/span>\n        <span class=\"k\">self<\/span><span class=\"py\">.x_train<\/span> <span class=\"o\">=<\/span> <span class=\"nf\">Some<\/span><span class=\"p\">(<\/span><span class=\"n\">x<\/span><span class=\"p\">);<\/span>\n        <span class=\"k\">self<\/span><span class=\"py\">.y_train<\/span> <span class=\"o\">=<\/span> <span class=\"nf\">Some<\/span><span class=\"p\">(<\/span><span class=\"n\">y<\/span><span class=\"p\">);<\/span>\n    <span class=\"p\">}<\/span>\n\n    <span class=\"k\">pub<\/span> <span class=\"k\">fn<\/span> <span class=\"nf\">predict<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"k\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">x<\/span><span class=\"p\">:<\/span> <span class=\"nb\">Vec<\/span><span class=\"o\">&lt;<\/span><span class=\"n\">Array1<\/span><span class=\"o\">&lt;<\/span><span class=\"nb\">f64<\/span><span class=\"o\">&gt;&gt;<\/span><span class=\"p\">)<\/span> <span class=\"k\">-&gt;<\/span> <span class=\"nb\">Vec<\/span><span class=\"o\">&lt;<\/span><span class=\"nb\">i32<\/span><span class=\"o\">&gt;<\/span> <span class=\"p\">{<\/span>\n        <span class=\"n\">x<\/span><span class=\"nf\">.iter<\/span><span class=\"p\">()<\/span><span class=\"nf\">.map<\/span><span class=\"p\">(|<\/span><span class=\"n\">xi<\/span><span class=\"p\">|<\/span> <span class=\"k\">self<\/span><span class=\"nf\">.predict_one<\/span><span class=\"p\">(<\/span><span class=\"n\">xi<\/span><span class=\"p\">))<\/span><span class=\"nf\">.collect<\/span><span class=\"p\">()<\/span>\n    <span class=\"p\">}<\/span>\n\n    <span class=\"k\">fn<\/span> <span class=\"nf\">predict_one<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"k\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">x<\/span><span class=\"p\">:<\/span> <span class=\"o\">&amp;<\/span><span class=\"n\">Array1<\/span><span class=\"o\">&lt;<\/span><span class=\"nb\">f64<\/span><span class=\"o\">&gt;<\/span><span class=\"p\">)<\/span> <span class=\"k\">-&gt;<\/span> <span class=\"nb\">i32<\/span> <span class=\"p\">{<\/span>\n        <span class=\"k\">let<\/span> <span class=\"n\">x_train<\/span> <span class=\"o\">=<\/span> <span class=\"k\">self<\/span><span class=\"py\">.x_train<\/span><span class=\"nf\">.as_ref<\/span><span class=\"p\">()<\/span><span class=\"nf\">.expect<\/span><span class=\"p\">(<\/span><span class=\"s\">\"Model not fitted\"<\/span><span class=\"p\">);<\/span>\n        <span class=\"k\">let<\/span> <span class=\"n\">y_train<\/span> <span class=\"o\">=<\/span> <span class=\"k\">self<\/span><span class=\"py\">.y_train<\/span><span class=\"nf\">.as_ref<\/span><span class=\"p\">()<\/span><span class=\"nf\">.expect<\/span><span class=\"p\">(<\/span><span class=\"s\">\"Model not fitted\"<\/span><span class=\"p\">);<\/span>\n\n        <span class=\"c1\">\/\/ compute the distance of the input point from all training points<\/span>\n        <span class=\"k\">let<\/span> <span class=\"n\">distances<\/span> <span class=\"o\">=<\/span> <span class=\"n\">x_train<\/span>\n            <span class=\"nf\">.iter<\/span><span class=\"p\">()<\/span>\n            <span class=\"nf\">.map<\/span><span class=\"p\">(|<\/span><span class=\"n\">x_train<\/span><span class=\"p\">|<\/span> <span class=\"nf\">euclidean_distance<\/span><span class=\"p\">(<\/span><span class=\"n\">x<\/span><span class=\"p\">,<\/span> <span class=\"n\">x_train<\/span><span class=\"p\">))<\/span>\n            <span class=\"py\">.collect<\/span><span class=\"p\">::<\/span><span class=\"o\">&lt;<\/span><span class=\"nb\">Vec<\/span><span class=\"o\">&lt;<\/span><span class=\"n\">_<\/span><span class=\"o\">&gt;&gt;<\/span><span class=\"p\">();<\/span>\n\n        <span class=\"c1\">\/\/ sort the indices of training points by their distance to the input point<\/span>\n        <span class=\"k\">let<\/span> <span class=\"k\">mut<\/span> <span class=\"n\">indices<\/span><span class=\"p\">:<\/span> <span class=\"nb\">Vec<\/span><span class=\"o\">&lt;<\/span><span class=\"nb\">usize<\/span><span class=\"o\">&gt;<\/span> <span class=\"o\">=<\/span> <span class=\"p\">(<\/span><span class=\"mi\">0<\/span><span class=\"o\">..<\/span><span class=\"n\">distances<\/span><span class=\"nf\">.len<\/span><span class=\"p\">())<\/span><span class=\"nf\">.collect<\/span><span class=\"p\">();<\/span>\n        <span class=\"n\">indices<\/span><span class=\"nf\">.sort_by<\/span><span class=\"p\">(|<\/span><span class=\"o\">&amp;<\/span><span class=\"n\">i<\/span><span class=\"p\">,<\/span> <span class=\"o\">&amp;<\/span><span class=\"n\">j<\/span><span class=\"p\">|<\/span> <span class=\"n\">distances<\/span><span class=\"p\">[<\/span><span class=\"n\">i<\/span><span class=\"p\">]<\/span><span class=\"nf\">.partial_cmp<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"n\">distances<\/span><span class=\"p\">[<\/span><span class=\"n\">j<\/span><span class=\"p\">])<\/span><span class=\"nf\">.unwrap<\/span><span class=\"p\">());<\/span>\n\n        <span class=\"c1\">\/\/ count the labels of the k nearest neighbors<\/span>\n        <span class=\"k\">let<\/span> <span class=\"k\">mut<\/span> <span class=\"n\">counter<\/span> <span class=\"o\">=<\/span> <span class=\"nn\">HashMap<\/span><span class=\"p\">::<\/span><span class=\"nf\">new<\/span><span class=\"p\">();<\/span>\n        <span class=\"k\">for<\/span> <span class=\"o\">&amp;<\/span><span class=\"n\">i<\/span> <span class=\"k\">in<\/span> <span class=\"o\">&amp;<\/span><span class=\"n\">indices<\/span><span class=\"p\">[<\/span><span class=\"o\">..<\/span><span class=\"k\">self<\/span><span class=\"py\">.k<\/span><span class=\"p\">]<\/span> <span class=\"p\">{<\/span>\n            <span class=\"o\">*<\/span><span class=\"n\">counter<\/span><span class=\"nf\">.entry<\/span><span class=\"p\">(<\/span><span class=\"n\">y_train<\/span><span class=\"p\">[<\/span><span class=\"n\">i<\/span><span class=\"p\">])<\/span><span class=\"nf\">.or_insert<\/span><span class=\"p\">(<\/span><span class=\"mi\">0<\/span><span class=\"p\">)<\/span> <span class=\"o\">+=<\/span> <span class=\"mi\">1<\/span><span class=\"p\">;<\/span>\n        <span class=\"p\">}<\/span>\n\n        <span class=\"c1\">\/\/ return the most common label among the k nearest neighbors<\/span>\n        <span class=\"n\">counter<\/span>\n            <span class=\"nf\">.into_iter<\/span><span class=\"p\">()<\/span>\n            <span class=\"nf\">.max_by_key<\/span><span class=\"p\">(|<\/span><span class=\"o\">&amp;<\/span><span class=\"p\">(<\/span><span class=\"n\">_<\/span><span class=\"p\">,<\/span> <span class=\"n\">count<\/span><span class=\"p\">)|<\/span> <span class=\"n\">count<\/span><span class=\"p\">)<\/span>\n            <span class=\"nf\">.map<\/span><span class=\"p\">(|(<\/span><span class=\"n\">label<\/span><span class=\"p\">,<\/span> <span class=\"n\">_<\/span><span class=\"p\">)|<\/span> <span class=\"n\">label<\/span><span class=\"p\">)<\/span>\n            <span class=\"nf\">.unwrap<\/span><span class=\"p\">()<\/span>\n    <span class=\"p\">}<\/span>\n<span class=\"p\">}<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>\n  crab.rs tests\n  <br>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight rust\"><code><span class=\"c1\">\/\/ above code ...<\/span>\n<span class=\"nd\">#[cfg(test)]<\/span>\n<span class=\"k\">mod<\/span> <span class=\"n\">tests<\/span> <span class=\"p\">{<\/span>\n    <span class=\"k\">use<\/span> <span class=\"k\">super<\/span><span class=\"p\">::<\/span><span class=\"o\">*<\/span><span class=\"p\">;<\/span>\n\n    <span class=\"nd\">#[test]<\/span>\n    <span class=\"k\">fn<\/span> <span class=\"nf\">test_euclidean_distance<\/span><span class=\"p\">()<\/span> <span class=\"p\">{<\/span>\n        <span class=\"k\">let<\/span> <span class=\"n\">x1<\/span> <span class=\"o\">=<\/span> <span class=\"nn\">Array1<\/span><span class=\"p\">::<\/span><span class=\"nf\">from<\/span><span class=\"p\">(<\/span><span class=\"nd\">vec!<\/span><span class=\"p\">[<\/span><span class=\"mf\">1.0<\/span><span class=\"p\">,<\/span> <span class=\"mf\">2.0<\/span><span class=\"p\">]);<\/span>\n        <span class=\"k\">let<\/span> <span class=\"n\">x2<\/span> <span class=\"o\">=<\/span> <span class=\"nn\">Array1<\/span><span class=\"p\">::<\/span><span class=\"nf\">from<\/span><span class=\"p\">(<\/span><span class=\"nd\">vec!<\/span><span class=\"p\">[<\/span><span class=\"mf\">4.0<\/span><span class=\"p\">,<\/span> <span class=\"mf\">6.0<\/span><span class=\"p\">]);<\/span>\n        <span class=\"k\">let<\/span> <span class=\"n\">distance<\/span> <span class=\"o\">=<\/span> <span class=\"nf\">euclidean_distance<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"n\">x1<\/span><span class=\"p\">,<\/span> <span class=\"o\">&amp;<\/span><span class=\"n\">x2<\/span><span class=\"p\">);<\/span>\n        <span class=\"k\">let<\/span> <span class=\"n\">expected_distance<\/span> <span class=\"o\">=<\/span> <span class=\"mf\">5.0<\/span><span class=\"p\">;<\/span>\n        <span class=\"nd\">assert!<\/span><span class=\"p\">((<\/span><span class=\"n\">distance<\/span> <span class=\"o\">-<\/span> <span class=\"n\">expected_distance<\/span><span class=\"p\">)<\/span><span class=\"nf\">.abs<\/span><span class=\"p\">()<\/span> <span class=\"o\">&lt;<\/span> <span class=\"mf\">1e-5<\/span><span class=\"p\">);<\/span>\n    <span class=\"p\">}<\/span>\n\n    <span class=\"nd\">#[test]<\/span>\n    <span class=\"k\">fn<\/span> <span class=\"nf\">test_knn_predict<\/span><span class=\"p\">()<\/span> <span class=\"p\">{<\/span>\n        <span class=\"k\">let<\/span> <span class=\"k\">mut<\/span> <span class=\"n\">knn<\/span> <span class=\"o\">=<\/span> <span class=\"nn\">KNN<\/span><span class=\"p\">::<\/span><span class=\"nf\">new<\/span><span class=\"p\">(<\/span><span class=\"mi\">1<\/span><span class=\"p\">);<\/span>\n\n        <span class=\"c1\">\/\/ Train the model with iris first and last data<\/span>\n        <span class=\"k\">let<\/span> <span class=\"n\">x_train<\/span> <span class=\"o\">=<\/span> <span class=\"nd\">vec!<\/span><span class=\"p\">[<\/span>\n            <span class=\"nn\">Array1<\/span><span class=\"p\">::<\/span><span class=\"nf\">from<\/span><span class=\"p\">(<\/span><span class=\"nd\">vec!<\/span><span class=\"p\">[<\/span><span class=\"mf\">5.1<\/span><span class=\"p\">,<\/span> <span class=\"mf\">3.5<\/span><span class=\"p\">,<\/span> <span class=\"mf\">1.4<\/span><span class=\"p\">,<\/span> <span class=\"mf\">0.2<\/span><span class=\"p\">]),<\/span>\n            <span class=\"nn\">Array1<\/span><span class=\"p\">::<\/span><span class=\"nf\">from<\/span><span class=\"p\">(<\/span><span class=\"nd\">vec!<\/span><span class=\"p\">[<\/span><span class=\"mf\">5.9<\/span><span class=\"p\">,<\/span> <span class=\"mf\">3.<\/span><span class=\"p\">,<\/span> <span class=\"mf\">5.1<\/span><span class=\"p\">,<\/span> <span class=\"mf\">1.8<\/span><span class=\"p\">]),<\/span>\n        <span class=\"p\">];<\/span>\n        <span class=\"k\">let<\/span> <span class=\"n\">y_train<\/span> <span class=\"o\">=<\/span> <span class=\"nd\">vec!<\/span><span class=\"p\">[<\/span><span class=\"mi\">0<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">];<\/span>\n        <span class=\"n\">knn<\/span><span class=\"nf\">.fit<\/span><span class=\"p\">(<\/span><span class=\"n\">x_train<\/span><span class=\"p\">,<\/span> <span class=\"n\">y_train<\/span><span class=\"p\">);<\/span>\n\n        <span class=\"c1\">\/\/ Predict a known value iris second point<\/span>\n        <span class=\"k\">let<\/span> <span class=\"n\">x_test<\/span> <span class=\"o\">=<\/span> <span class=\"nd\">vec!<\/span><span class=\"p\">[<\/span><span class=\"nn\">Array1<\/span><span class=\"p\">::<\/span><span class=\"nf\">from<\/span><span class=\"p\">(<\/span><span class=\"nd\">vec!<\/span><span class=\"p\">[<\/span><span class=\"mf\">4.7<\/span><span class=\"p\">,<\/span> <span class=\"mf\">3.2<\/span><span class=\"p\">,<\/span> <span class=\"mf\">1.3<\/span><span class=\"p\">,<\/span> <span class=\"mf\">0.2<\/span><span class=\"p\">])];<\/span>\n        <span class=\"k\">let<\/span> <span class=\"n\">predictions<\/span> <span class=\"o\">=<\/span> <span class=\"n\">knn<\/span><span class=\"nf\">.predict<\/span><span class=\"p\">(<\/span><span class=\"n\">x_test<\/span><span class=\"p\">);<\/span>\n\n        <span class=\"nd\">assert_eq!<\/span><span class=\"p\">(<\/span><span class=\"n\">predictions<\/span><span class=\"p\">,<\/span> <span class=\"nd\">vec!<\/span><span class=\"p\">[<\/span><span class=\"mi\">0<\/span><span class=\"p\">]);<\/span>\n    <span class=\"p\">}<\/span>\n<span class=\"p\">}<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n<h2>\n  \n  \n  <\/h2>\n\n\n\n<\/p>\n<h3>\n  \n  \n  Chapter 2: Enchanted Tools - maturin and PyO3\n<\/h3>\n\n<p>To bridge their worlds, Python and Rust turn to two magical tools: <code>maturin<\/code> and <code>PyO3<\/code>. Like Triton's trident, these tools hold the power to unite different realms. <code>maturin<\/code>, a build system that compiles Rust libraries into Python packages, works seamlessly alongside PyO3, which allows Rust functions to be called from Python. <\/p>\n\n<p>These tools promise to simplify the journey, making the integration process feel like part of an enchanting undersea adventure.<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight rust\"><code><span class=\"c1\">\/\/ src\/snake.rs<\/span>\n<span class=\"k\">use<\/span> <span class=\"k\">crate<\/span><span class=\"p\">::<\/span><span class=\"nn\">crab<\/span><span class=\"p\">::<\/span><span class=\"n\">KNN<\/span> <span class=\"k\">as<\/span> <span class=\"n\">rKNN<\/span><span class=\"p\">;<\/span>\n<span class=\"k\">use<\/span> <span class=\"nn\">ndarray<\/span><span class=\"p\">::<\/span><span class=\"n\">Array1<\/span><span class=\"p\">;<\/span>\n<span class=\"k\">use<\/span> <span class=\"nn\">pyo3<\/span><span class=\"p\">::<\/span><span class=\"nn\">prelude<\/span><span class=\"p\">::<\/span><span class=\"o\">*<\/span><span class=\"p\">;<\/span>\n\n<span class=\"nd\">#[pyclass]<\/span>\n<span class=\"k\">struct<\/span> <span class=\"n\">KNN<\/span> <span class=\"p\">{<\/span>\n    <span class=\"n\">knn<\/span><span class=\"p\">:<\/span> <span class=\"n\">rKNN<\/span><span class=\"p\">,<\/span>\n<span class=\"p\">}<\/span>\n\n<span class=\"nd\">#[pymethods]<\/span>\n<span class=\"k\">impl<\/span> <span class=\"n\">KNN<\/span> <span class=\"p\">{<\/span>\n    <span class=\"nd\">#[new]<\/span>\n    <span class=\"k\">fn<\/span> <span class=\"nf\">new<\/span><span class=\"p\">(<\/span><span class=\"n\">k<\/span><span class=\"p\">:<\/span> <span class=\"nb\">usize<\/span><span class=\"p\">)<\/span> <span class=\"k\">-&gt;<\/span> <span class=\"k\">Self<\/span> <span class=\"p\">{<\/span>\n        <span class=\"n\">KNN<\/span> <span class=\"p\">{<\/span> <span class=\"n\">knn<\/span><span class=\"p\">:<\/span> <span class=\"nn\">rKNN<\/span><span class=\"p\">::<\/span><span class=\"nf\">new<\/span><span class=\"p\">(<\/span><span class=\"n\">k<\/span><span class=\"p\">)<\/span> <span class=\"p\">}<\/span>\n    <span class=\"p\">}<\/span>\n\n    <span class=\"k\">fn<\/span> <span class=\"nf\">fit<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"k\">mut<\/span> <span class=\"k\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">x_train<\/span><span class=\"p\">:<\/span> <span class=\"nb\">Vec<\/span><span class=\"o\">&lt;<\/span><span class=\"nb\">Vec<\/span><span class=\"o\">&lt;<\/span><span class=\"nb\">f64<\/span><span class=\"o\">&gt;&gt;<\/span><span class=\"p\">,<\/span> <span class=\"n\">y_train<\/span><span class=\"p\">:<\/span> <span class=\"nb\">Vec<\/span><span class=\"o\">&lt;<\/span><span class=\"nb\">i32<\/span><span class=\"o\">&gt;<\/span><span class=\"p\">)<\/span> <span class=\"p\">{<\/span>\n        <span class=\"k\">let<\/span> <span class=\"n\">x_train<\/span><span class=\"p\">:<\/span> <span class=\"nb\">Vec<\/span><span class=\"o\">&lt;<\/span><span class=\"n\">_<\/span><span class=\"o\">&gt;<\/span> <span class=\"o\">=<\/span> <span class=\"n\">x_train<\/span><span class=\"nf\">.into_iter<\/span><span class=\"p\">()<\/span><span class=\"nf\">.map<\/span><span class=\"p\">(<\/span><span class=\"nn\">Array1<\/span><span class=\"p\">::<\/span><span class=\"n\">from<\/span><span class=\"p\">)<\/span><span class=\"nf\">.collect<\/span><span class=\"p\">();<\/span>\n        <span class=\"k\">self<\/span><span class=\"py\">.knn<\/span><span class=\"nf\">.fit<\/span><span class=\"p\">(<\/span><span class=\"n\">x_train<\/span><span class=\"p\">,<\/span> <span class=\"n\">y_train<\/span><span class=\"p\">);<\/span>\n    <span class=\"p\">}<\/span>\n\n    <span class=\"k\">fn<\/span> <span class=\"nf\">predict<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"k\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">x_test<\/span><span class=\"p\">:<\/span> <span class=\"nb\">Vec<\/span><span class=\"o\">&lt;<\/span><span class=\"nb\">Vec<\/span><span class=\"o\">&lt;<\/span><span class=\"nb\">f64<\/span><span class=\"o\">&gt;&gt;<\/span><span class=\"p\">)<\/span> <span class=\"k\">-&gt;<\/span> <span class=\"nb\">Vec<\/span><span class=\"o\">&lt;<\/span><span class=\"nb\">i32<\/span><span class=\"o\">&gt;<\/span> <span class=\"p\">{<\/span>\n        <span class=\"k\">let<\/span> <span class=\"n\">x_test<\/span><span class=\"p\">:<\/span> <span class=\"nb\">Vec<\/span><span class=\"o\">&lt;<\/span><span class=\"n\">_<\/span><span class=\"o\">&gt;<\/span> <span class=\"o\">=<\/span> <span class=\"n\">x_test<\/span><span class=\"nf\">.into_iter<\/span><span class=\"p\">()<\/span><span class=\"nf\">.map<\/span><span class=\"p\">(<\/span><span class=\"nn\">Array1<\/span><span class=\"p\">::<\/span><span class=\"n\">from<\/span><span class=\"p\">)<\/span><span class=\"nf\">.collect<\/span><span class=\"p\">();<\/span>\n        <span class=\"k\">self<\/span><span class=\"py\">.knn<\/span><span class=\"nf\">.predict<\/span><span class=\"p\">(<\/span><span class=\"n\">x_test<\/span><span class=\"p\">)<\/span>\n    <span class=\"p\">}<\/span>\n<span class=\"p\">}<\/span>\n\n<span class=\"nd\">#[pymodule]<\/span>\n<span class=\"k\">fn<\/span> <span class=\"nf\">knn<\/span><span class=\"p\">(<\/span><span class=\"n\">_py<\/span><span class=\"p\">:<\/span> <span class=\"n\">Python<\/span><span class=\"p\">,<\/span> <span class=\"n\">m<\/span><span class=\"p\">:<\/span> <span class=\"o\">&amp;<\/span><span class=\"n\">PyModule<\/span><span class=\"p\">)<\/span> <span class=\"k\">-&gt;<\/span> <span class=\"n\">PyResult<\/span><span class=\"o\">&lt;<\/span><span class=\"p\">()<\/span><span class=\"o\">&gt;<\/span> <span class=\"p\">{<\/span>\n    <span class=\"n\">m<\/span><span class=\"py\">.add_class<\/span><span class=\"p\">::<\/span><span class=\"o\">&lt;<\/span><span class=\"n\">KNN<\/span><span class=\"o\">&gt;<\/span><span class=\"p\">()<\/span><span class=\"o\">?<\/span><span class=\"p\">;<\/span>\n    <span class=\"nf\">Ok<\/span><span class=\"p\">(())<\/span>\n<span class=\"p\">}<\/span>\n\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>\n  src\/lib.rs\n  <br>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight rust\"><code><span class=\"k\">mod<\/span> <span class=\"n\">snake<\/span><span class=\"p\">;<\/span>\n<span class=\"k\">pub<\/span> <span class=\"k\">mod<\/span> <span class=\"n\">crab<\/span><span class=\"p\">;<\/span> \n<\/code><\/pre>\n\n<\/div>\n\n\n\n\n<\/p>\n\n\n\n\n<h3>\n  \n  \n  Chapter 3: Overcoming the Ursula of Complexity\n<\/h3>\n\n<p>But no adventure is without its challenges. Complexity and integration issues loom like Ursula, threatening to hinder the progress of our heroes. Rust's strict type system and ownership rules can be daunting, and Python's dynamic nature poses its own set of integration challenges.<\/p>\n\n<p>Yet, with the power of maturin and PyO3, these obstacles are skillfully navigated. The tools weave Rust's functions into Python's script like a spell, allowing Python to call upon the kNN algorithm as if it were its own.<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight shell\"><code><span class=\"c\"># develop | build depending on the stage knn package<\/span>\npixi run maturin build\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>As the final step, our heroes test their creation. Python, with its extensive libraries and tools, puts the Rust-implemented kNN to the test, feeding it data as it would in any Pythonic environment. The algorithm performs flawlessly, classifying data points with the speed and accuracy that only Rust can provide, all within the comfortable and familiar embrace of Python.<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"c1\"># use\/ml.py\n<\/span><span class=\"kn\">from<\/span> <span class=\"n\">numpy.typing<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">NDArray<\/span>\n<span class=\"kn\">from<\/span> <span class=\"n\">sklearn.datasets<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">load_iris<\/span>\n<span class=\"kn\">from<\/span> <span class=\"n\">sklearn.model_selection<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">train_test_split<\/span>\n<span class=\"kn\">from<\/span> <span class=\"n\">sklearn.metrics<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">classification_report<\/span>\n\n<span class=\"kn\">from<\/span> <span class=\"n\">knn<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">KNN<\/span>\n\n\n<span class=\"k\">def<\/span> <span class=\"nf\">get_iris<\/span><span class=\"p\">(<\/span>\n    <span class=\"n\">train_size<\/span><span class=\"p\">:<\/span> <span class=\"nb\">float<\/span> <span class=\"o\">=<\/span> <span class=\"mf\">0.8<\/span><span class=\"p\">,<\/span> <span class=\"n\">random_state<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">42<\/span>\n<span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">tuple<\/span><span class=\"p\">[<\/span><span class=\"n\">NDArray<\/span><span class=\"p\">,<\/span> <span class=\"n\">NDArray<\/span><span class=\"p\">,<\/span> <span class=\"n\">NDArray<\/span><span class=\"p\">,<\/span> <span class=\"n\">NDArray<\/span><span class=\"p\">]:<\/span>\n    <span class=\"n\">iris<\/span> <span class=\"o\">=<\/span> <span class=\"nf\">load_iris<\/span><span class=\"p\">()<\/span>\n    <span class=\"n\">X<\/span><span class=\"p\">,<\/span> <span class=\"n\">y<\/span> <span class=\"o\">=<\/span> <span class=\"n\">iris<\/span><span class=\"p\">.<\/span><span class=\"n\">data<\/span><span class=\"p\">,<\/span> <span class=\"n\">iris<\/span><span class=\"p\">.<\/span><span class=\"n\">target<\/span>\n\n    <span class=\"k\">return<\/span> <span class=\"nf\">train_test_split<\/span><span class=\"p\">(<\/span>\n        <span class=\"n\">X<\/span><span class=\"p\">,<\/span> <span class=\"n\">y<\/span><span class=\"p\">,<\/span> <span class=\"n\">train_size<\/span><span class=\"o\">=<\/span><span class=\"n\">train_size<\/span><span class=\"p\">,<\/span> <span class=\"n\">stratify<\/span><span class=\"o\">=<\/span><span class=\"n\">y<\/span><span class=\"p\">,<\/span> <span class=\"n\">random_state<\/span><span class=\"o\">=<\/span><span class=\"n\">random_state<\/span>\n    <span class=\"p\">)<\/span>\n\n\n<span class=\"k\">if<\/span> <span class=\"n\">__name__<\/span> <span class=\"o\">==<\/span> <span class=\"sh\">\"<\/span><span class=\"s\">__main__<\/span><span class=\"sh\">\"<\/span><span class=\"p\">:<\/span>\n    <span class=\"c1\"># get data\n<\/span>    <span class=\"n\">X_train<\/span><span class=\"p\">,<\/span> <span class=\"n\">X_test<\/span><span class=\"p\">,<\/span> <span class=\"n\">y_train<\/span><span class=\"p\">,<\/span> <span class=\"n\">y_test<\/span> <span class=\"o\">=<\/span> <span class=\"nf\">get_iris<\/span><span class=\"p\">(<\/span><span class=\"n\">train_size<\/span><span class=\"o\">=<\/span><span class=\"mf\">0.8<\/span><span class=\"p\">)<\/span>\n\n    <span class=\"c1\"># get number of classes\n<\/span>    <span class=\"n\">k<\/span> <span class=\"o\">=<\/span> <span class=\"nf\">len<\/span><span class=\"p\">(<\/span><span class=\"nf\">set<\/span><span class=\"p\">(<\/span><span class=\"n\">y_train<\/span><span class=\"p\">))<\/span>\n\n    <span class=\"c1\"># create a kNN predictor and train\n<\/span>    <span class=\"n\">predictor<\/span> <span class=\"o\">=<\/span> <span class=\"nc\">KNN<\/span><span class=\"p\">(<\/span><span class=\"n\">k<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">predictor<\/span><span class=\"p\">.<\/span><span class=\"nf\">fit<\/span><span class=\"p\">(<\/span><span class=\"n\">X_train<\/span><span class=\"p\">,<\/span> <span class=\"n\">y_train<\/span><span class=\"p\">)<\/span>\n\n    <span class=\"c1\"># predict on test set\n<\/span>    <span class=\"n\">y_pred<\/span> <span class=\"o\">=<\/span> <span class=\"n\">predictor<\/span><span class=\"p\">.<\/span><span class=\"nf\">predict<\/span><span class=\"p\">(<\/span><span class=\"n\">X_test<\/span><span class=\"p\">)<\/span>\n\n    <span class=\"c1\"># evaluate performance\n<\/span>    <span class=\"nf\">print<\/span><span class=\"p\">(<\/span><span class=\"nf\">classification_report<\/span><span class=\"p\">(<\/span><span class=\"n\">y_true<\/span><span class=\"o\">=<\/span><span class=\"n\">y_test<\/span><span class=\"p\">,<\/span> <span class=\"n\">y_pred<\/span><span class=\"o\">=<\/span><span class=\"n\">y_pred<\/span><span class=\"p\">))<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n\n\n\n<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fq1iof8eodar071jfsz74.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fq1iof8eodar071jfsz74.png\" alt=\"final\" width=\"768\" height=\"439\"><\/a><\/p>\n\n<h3>\n  \n  \n  The Final Showdown: A New Dawn in the Programming Kingdom\n<\/h3>\n\n<p>As our story concludes, Python and Rust, along with their magical allies <code>maturin<\/code> and <code>PyO3<\/code>, have shown that together, they are more than the sum of their parts. They prove that combining the strengths of different languages can lead to powerful and efficient solutions, pushing the boundaries of what's possible in the programming world.<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight shell\"><code>pixi run python <span class=\"nt\">-m<\/span> use.ml\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>In the end, the sea of code is vast and deep, filled with opportunities for those brave enough to explore its depths. Python and Rust, once separate, now swim together in harmony, ready for the next great adventure.<\/p>\n\n\n\n\n<p>\u26a0\ufe0f  Rust code is not, and was not meant, to be the effient way of performing kNN. If you do have a more effient way, do share on the comments.<\/p>\n\n\n\n\n<p>\"Now, [<code>until then<\/code>] let's eat before this crab wanders off my plate.\" -Grimsby<\/p>\n\n","category":["python","rust","machinelearning"]},{"title":"To Code and Beyond: A Neverland Adventure in Bash, Lua, Python, and Rust","pubDate":"Sat, 23 Dec 2023 14:06:16 +0000","link":"https:\/\/dev.to\/proteusiq\/to-code-and-beyond-a-neverland-adventure-in-bash-lua-python-and-rust-1jon","guid":"https:\/\/dev.to\/proteusiq\/to-code-and-beyond-a-neverland-adventure-in-bash-lua-python-and-rust-1jon","description":"<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fepnd8ccp1dbf02z63h9f.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fepnd8ccp1dbf02z63h9f.png\" alt=\"ship\" width=\"768\" height=\"439\"><\/a><\/p>\n\n<h3>\n  \n  \n  Prologue: Departure to Neverland\n<\/h3>\n\n<p>Once upon a time, in the mystical world of terminals, we find ourselves tagging along on a Peter Pan-themed odyssey, soaring across the skies of Neverland. Our quest? To discover the magic similarities and differences of the unique spirits of <code>Bash<\/code>, <code>Lua<\/code>, <code>Python<\/code>, and <code>Rust<\/code>.<\/p>\n\n\n\n\n<h3>\n  \n  \n  Chapter 1: Aboard the Jolly Roger with Bash and Lua\n<\/h3>\n\n<p>Under the moonlit sky, we join Captain Hook on the Jolly Roger, navigating the traditional seas of Bash and Lua. Like seasoned sailors chanting an age-old shanty, these languages use for loops as their trusted compass.<\/p>\n\n<p><strong>Bash - The Captain's Command:<\/strong><br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight shell\"><code><span class=\"c\">#!\/bin\/bash<\/span>\n\nsum_odd_int_array<span class=\"o\">()<\/span> <span class=\"o\">{<\/span>\n    <span class=\"nb\">local sum<\/span><span class=\"o\">=<\/span>0\n    <span class=\"k\">for <\/span>x <span class=\"k\">in<\/span> <span class=\"s2\">\"<\/span><span class=\"nv\">$@<\/span><span class=\"s2\">\"<\/span><span class=\"p\">;<\/span> <span class=\"k\">do\n        if<\/span> <span class=\"o\">((<\/span> x % 2 <span class=\"o\">!=<\/span> 0 <span class=\"o\">))<\/span><span class=\"p\">;<\/span> <span class=\"k\">then<\/span>\n           <span class=\"o\">((<\/span> <span class=\"nb\">sum<\/span>+<span class=\"o\">=<\/span>x <span class=\"o\">))<\/span>\n        <span class=\"k\">fi\n    done\n    <\/span><span class=\"nb\">echo<\/span> <span class=\"nv\">$sum<\/span>\n<span class=\"o\">}<\/span>\n\n<span class=\"nv\">array<\/span><span class=\"o\">=(<\/span>1 2 3 4 5<span class=\"o\">)<\/span>\nsum_odd_int_array <span class=\"k\">${<\/span><span class=\"nv\">array<\/span><span class=\"p\">[@]<\/span><span class=\"k\">}<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p><strong>Lua - The First Mate's Chant:<\/strong><br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight lua\"><code><span class=\"k\">function<\/span> <span class=\"nf\">sum_odd_int_array<\/span><span class=\"p\">(<\/span><span class=\"n\">array<\/span><span class=\"p\">)<\/span>\n    <span class=\"kd\">local<\/span> <span class=\"n\">sum<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">0<\/span>\n    <span class=\"k\">for<\/span> <span class=\"n\">_<\/span><span class=\"p\">,<\/span> <span class=\"n\">x<\/span> <span class=\"k\">in<\/span> <span class=\"nb\">ipairs<\/span><span class=\"p\">(<\/span><span class=\"n\">array<\/span><span class=\"p\">)<\/span> <span class=\"k\">do<\/span>\n        <span class=\"k\">if<\/span> <span class=\"n\">x<\/span> <span class=\"o\">%<\/span> <span class=\"mi\">2<\/span> <span class=\"o\">~=<\/span> <span class=\"mi\">0<\/span> <span class=\"k\">then<\/span>\n            <span class=\"n\">sum<\/span> <span class=\"o\">=<\/span> <span class=\"n\">sum<\/span> <span class=\"o\">+<\/span> <span class=\"n\">x<\/span>\n        <span class=\"k\">end<\/span>\n    <span class=\"k\">end<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">sum<\/span>\n<span class=\"k\">end<\/span>\n\n<span class=\"kd\">local<\/span> <span class=\"n\">array<\/span> <span class=\"o\">=<\/span> <span class=\"p\">{<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"mi\">3<\/span><span class=\"p\">,<\/span> <span class=\"mi\">4<\/span><span class=\"p\">,<\/span> <span class=\"mi\">5<\/span><span class=\"p\">}<\/span>\n<span class=\"nb\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">sum_odd_int_array<\/span><span class=\"p\">(<\/span><span class=\"n\">array<\/span><span class=\"p\">))<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n\n\n\n<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F944bnwt2e2bxylxbtjgo.jpeg\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F944bnwt2e2bxylxbtjgo.jpeg\" alt=\"peter\" width=\"768\" height=\"416\"><\/a><\/p>\n\n<h3>\n  \n  \n  Chapter 2: Tinker Bell's Python Whispers\n<\/h3>\n\n<p>In the heart of Neverland, Tinker Bell whisks us away, revealing the wonders of Python. She shows us two paths: one trodden by many, and the other, a secret trail lit by her magical glow.<\/p>\n\n<p><strong>The Beaten Path - Traditional For Loop:<\/strong><br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">def<\/span> <span class=\"nf\">sum_odd_int_array_for_loop<\/span><span class=\"p\">(<\/span><span class=\"n\">array<\/span><span class=\"p\">:<\/span> <span class=\"nb\">list<\/span><span class=\"p\">[<\/span><span class=\"nb\">int<\/span><span class=\"p\">])<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">int<\/span><span class=\"p\">:<\/span>\n    <span class=\"nb\">sum<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">0<\/span>\n    <span class=\"k\">for<\/span> <span class=\"n\">x<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">array<\/span><span class=\"p\">:<\/span>\n        <span class=\"k\">if<\/span> <span class=\"n\">x<\/span> <span class=\"o\">%<\/span> <span class=\"mi\">2<\/span> <span class=\"o\">!=<\/span> <span class=\"mi\">0<\/span><span class=\"p\">:<\/span>\n            <span class=\"nb\">sum<\/span> <span class=\"o\">+=<\/span> <span class=\"n\">x<\/span>\n    <span class=\"k\">return<\/span> <span class=\"nb\">sum<\/span>\n\n<span class=\"n\">array<\/span> <span class=\"o\">=<\/span> <span class=\"p\">[<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"mi\">3<\/span><span class=\"p\">,<\/span> <span class=\"mi\">4<\/span><span class=\"p\">,<\/span> <span class=\"mi\">5<\/span><span class=\"p\">]<\/span>\n<span class=\"nf\">print<\/span><span class=\"p\">(<\/span><span class=\"nf\">sum_odd_int_array_for_loop<\/span><span class=\"p\">(<\/span><span class=\"n\">array<\/span><span class=\"p\">))<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p><strong>Tinker Bell's Enchanted Trail - Comprehension:<\/strong><br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">def<\/span> <span class=\"nf\">sum_odd_int_array<\/span><span class=\"p\">(<\/span><span class=\"n\">array<\/span><span class=\"p\">:<\/span> <span class=\"nb\">list<\/span><span class=\"p\">[<\/span><span class=\"nb\">int<\/span><span class=\"p\">])<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">int<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"nf\">sum<\/span><span class=\"p\">(<\/span><span class=\"n\">x<\/span> <span class=\"k\">for<\/span> <span class=\"n\">x<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">array<\/span> <span class=\"k\">if<\/span> <span class=\"n\">x<\/span> <span class=\"o\">%<\/span> <span class=\"mi\">2<\/span> <span class=\"o\">!=<\/span> <span class=\"mi\">0<\/span><span class=\"p\">)<\/span>\n\n<span class=\"n\">array<\/span> <span class=\"o\">=<\/span> <span class=\"p\">[<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"mi\">3<\/span><span class=\"p\">,<\/span> <span class=\"mi\">4<\/span><span class=\"p\">,<\/span> <span class=\"mi\">5<\/span><span class=\"p\">]<\/span>\n<span class=\"nf\">print<\/span><span class=\"p\">(<\/span><span class=\"nf\">sum_odd_int_array<\/span><span class=\"p\">(<\/span><span class=\"n\">array<\/span><span class=\"p\">))<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n\n\n\n<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fsdhcahcmq4qfkh3uiu7z.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fsdhcahcmq4qfkh3uiu7z.png\" alt=\"rust\" width=\"768\" height=\"439\"><\/a><\/p>\n\n<h3>\n  \n  \n  Chapter 3: The Lost Boys' Rusty Innovations\n<\/h3>\n\n<p>Deep in Neverland's forests, the Lost Boys unveil their secret: a marvel of Rust. First, a familiar structure, echoing the old ways. Then, a creation so ingenious, it seemed woven from the threads of the future.<\/p>\n\n<p><strong>The Olden Design - Traditional For Loop:<\/strong><br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight rust\"><code><span class=\"k\">fn<\/span> <span class=\"nf\">sum_odd_int_array_for_loop<\/span><span class=\"p\">(<\/span><span class=\"n\">array<\/span><span class=\"p\">:<\/span> <span class=\"o\">&amp;<\/span><span class=\"p\">[<\/span><span class=\"nb\">i32<\/span><span class=\"p\">])<\/span> <span class=\"k\">-&gt;<\/span> <span class=\"nb\">i32<\/span> <span class=\"p\">{<\/span>\n    <span class=\"k\">let<\/span> <span class=\"k\">mut<\/span> <span class=\"n\">sum<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">0<\/span><span class=\"p\">;<\/span>\n    <span class=\"k\">for<\/span> <span class=\"o\">&amp;<\/span><span class=\"n\">x<\/span> <span class=\"k\">in<\/span> <span class=\"n\">array<\/span> <span class=\"p\">{<\/span>\n        <span class=\"k\">if<\/span> <span class=\"n\">x<\/span> <span class=\"o\">%<\/span> <span class=\"mi\">2<\/span> <span class=\"o\">!=<\/span> <span class=\"mi\">0<\/span> <span class=\"p\">{<\/span>\n            <span class=\"n\">sum<\/span> <span class=\"o\">+=<\/span> <span class=\"n\">x<\/span><span class=\"p\">;<\/span>\n        <span class=\"p\">}<\/span>\n    <span class=\"p\">}<\/span>\n    <span class=\"n\">sum<\/span>\n<span class=\"p\">}<\/span>\n\n<span class=\"k\">fn<\/span> <span class=\"nf\">main<\/span><span class=\"p\">()<\/span> <span class=\"p\">{<\/span>\n    <span class=\"k\">let<\/span> <span class=\"n\">array<\/span> <span class=\"o\">=<\/span> <span class=\"p\">[<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"mi\">3<\/span><span class=\"p\">,<\/span> <span class=\"mi\">4<\/span><span class=\"p\">,<\/span> <span class=\"mi\">5<\/span><span class=\"p\">];<\/span>\n    <span class=\"nd\">println!<\/span><span class=\"p\">(<\/span><span class=\"s\">\"{}\"<\/span><span class=\"p\">,<\/span> <span class=\"nf\">sum_odd_int_array_for_loop<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"n\">array<\/span><span class=\"p\">));<\/span>\n<span class=\"p\">}<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p><strong>The Future Woven - Iterator Method:<\/strong><br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight rust\"><code><span class=\"k\">fn<\/span> <span class=\"nf\">sum_odd_int_array<\/span><span class=\"p\">(<\/span><span class=\"n\">array<\/span><span class=\"p\">:<\/span> <span class=\"o\">&amp;<\/span><span class=\"p\">[<\/span><span class=\"nb\">i32<\/span><span class=\"p\">])<\/span> <span class=\"k\">-&gt;<\/span> <span class=\"nb\">i32<\/span> <span class=\"p\">{<\/span>\n    <span class=\"n\">array<\/span><span class=\"nf\">.iter<\/span><span class=\"p\">()<\/span><span class=\"nf\">.filter<\/span><span class=\"p\">(|<\/span><span class=\"o\">&amp;&amp;<\/span><span class=\"n\">x<\/span><span class=\"p\">|<\/span> <span class=\"n\">x<\/span> <span class=\"o\">%<\/span> <span class=\"mi\">2<\/span> <span class=\"o\">!=<\/span> <span class=\"mi\">0<\/span><span class=\"p\">)<\/span><span class=\"nf\">.sum<\/span><span class=\"p\">()<\/span>\n<span class=\"p\">}<\/span>\n\n<span class=\"k\">fn<\/span> <span class=\"nf\">main<\/span><span class=\"p\">()<\/span> <span class=\"p\">{<\/span>\n    <span class=\"k\">let<\/span> <span class=\"n\">array<\/span> <span class=\"o\">=<\/span> <span class=\"p\">[<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"mi\">3<\/span><span class=\"p\">,<\/span> <span class=\"mi\">4<\/span><span class=\"p\">,<\/span> <span class=\"mi\">5<\/span><span class=\"p\">];<\/span>\n    <span class=\"nd\">println!<\/span><span class=\"p\">(<\/span><span class=\"s\">\"{}\"<\/span><span class=\"p\">,<\/span> <span class=\"nf\">sum_odd_int_array<\/span><span class=\"p\">(<\/span><span class=\"o\">&amp;<\/span><span class=\"n\">array<\/span><span class=\"p\">));<\/span>\n<span class=\"p\">}<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n\n\n\n<h3>\n  \n  \n  Epilogue: Magic in the Code\n<\/h3>\n\n<p>From the steady chants of Bash and Lua to the whimsical whispers of Python and the ingenious creations of Rust, each language brings its own spellbinding qualities. We're reminded of the magic and wonder that each language holds.<\/p>\n\n<p>As ageless programmers on a Neverland odyssey, we discover the art of transcending traditional loops, delving into the allure of modern programming languages and their captivating syntactic sugar.<\/p>\n\n<p>In this Neverland of code, the adventure never ends, and with each line written, we continue to weave our own magical tales.<\/p>\n\n<p><code>Until then<\/code>, keep on coding with \ud83e\ude84 and\ud83e\ude9ds.<\/p>\n\n","category":["rust","python","lua","bash"]},{"title":"The One Plot to Rule Them All","pubDate":"Thu, 07 Dec 2023 13:41:24 +0000","link":"https:\/\/dev.to\/proteusiq\/the-one-plot-to-rule-them-all-52e1","guid":"https:\/\/dev.to\/proteusiq\/the-one-plot-to-rule-them-all-52e1","description":"<h4>\n  \n  \n  Prologue: In a Data Shire Far Away in Copenhagen\n<\/h4>\n\n<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F13nahf3tk76qm4t350er.gif\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F13nahf3tk76qm4t350er.gif\" alt=\"copenhagen\" width=\"800\" height=\"400\"><\/a><\/p>\n\n<p>In the quiet Shire of Data Science, where rows and columns lived in harmony, there existed many visualization tools, akin to the peaceful hobbits of this land. But amidst these, a tool emerged - HiPlot, the One Ring to rule them all, that did not just display data but unveiled layers of hidden insights. <\/p>\n\n<p><em>\u2139\ufe0f One Ring used in a <code>!=<\/code> \ud83d\udc7f way. <a href=\"https:\/\/github.com\/Proteusiq\/hadithi\/tree\/main\/dev.io\/notebooks\" rel=\"noopener noreferrer\">Code and data<\/a> behind my <a href=\"https:\/\/www.linkedin.com\/feed\/update\/urn:li:activity:7136839244513259520?updateEntityUrn=urn%3Ali%3Afs_feedUpdate%3A%28V2%2Curn%3Ali%3Aactivity%3A7136839244513259520%29\" rel=\"noopener noreferrer\">PyData Copenhagen Talk<\/a> that inspired this article<\/em> <\/p>\n\n<h3>\n  \n  \n  Chapter 1: The Fellowship of the Tools\n<\/h3>\n\n<p>Like Frodo Baggins, a humble hobbit thrust into an unexpected journey, I ventured into the vast downland woods of data matching toward forging models that could predict apartment sale price with the fellowship of mat plotting tools. <\/p>\n\n<p>Mat plotting tools at my disposal were akin to the trusted, yet simple, tools of a Hobbit's life. <code>Seaborn<\/code>, for instance, was a loyal companion. Its ability to generate statistical data visualisation to lit correlation matrices, distribution and outliers with violin or box plots were invaluable.<\/p>\n\n<p><strong>Spies of the Dark Lord<\/strong><br>\n<a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fmd4siar8ptkvjke1x7un.gif\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fmd4siar8ptkvjke1x7un.gif\" alt=\"Middle Age\" width=\"800\" height=\"400\"><\/a><\/p>\n\n<p><strong>Shelob the Model Diagnostics<\/strong><br>\n<a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fqz705sgxl0grg1wgmmer.gif\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fqz705sgxl0grg1wgmmer.gif\" alt=\"useless\" width=\"1848\" height=\"465\"><\/a><\/p>\n\n<p>While useful, their process was akin to walking the long, winding paths of Middle Earth on foot. Generating correlations or identifying outliers was informative, but not swift. <\/p>\n\n<p>In the fast-paced world of the Dark Lord of Mordor, feature selection and model results diagnostics, where decisions and iterations need to happen at the speed of thought, these tools, like a slow-moving ent, lacked the urgency required.<\/p>\n\n<h3>\n  \n  \n  Chapter 2: The Eye of Sauron - Spotting Outliers\n<\/h3>\n\n<p>Then, as Gandalf presented the One Ring to Frodo, HiPlot came into my grasp. Like the swift steeds of Rohan - fast, powerful, and majestic. HiPlot transcended the traditional methods, offering a visualization experience that was not just informative but also incredibly rapid. Its interactive and intuitive interface allowed for a faster, almost instantaneous exploration of data, aligning perfectly with the needs of quick-thinking ML development.<\/p>\n\n<p>Armed with HiPlot, I confronted the Eye of Sauron - outliers in my datasets. Like Frodo using the ring to see the unseen, HiPlot illuminated these anomalies in stark contrast against the norm.<\/p>\n\n<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fklhrlvsyik5yhcqnp06j.gif\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fklhrlvsyik5yhcqnp06j.gif\" alt=\"features\" width=\"800\" height=\"400\"><\/a><\/p>\n\n<p>Each outlier stood out, clear as the Eye atop Barad-d\u00fbr, no longer a hidden threat to my model\u2019s integrity.<\/p>\n\n<h3>\n  \n  \n  Chapter 3: The Paths of the Dead - Unearthing Features\n<\/h3>\n\n<p>In the Paths of the Dead, Frodo had to trust his instincts to navigate. Similarly, HiPlot guided me through the spectral world of feature selection. It revealed which variables were mere echoes and which were vital to my quest of predicting apartment prices. <br>\n<a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fiyv0zhkr5rpvpxyrqdn4.gif\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fiyv0zhkr5rpvpxyrqdn4.gif\" alt=\"correction\" width=\"800\" height=\"400\"><\/a> <br>\nLike the Army of the Dead aiding Aragorn, these features became my allies.<\/p>\n\n<p>The final leg of my journey was akin to Frodo's ascent of Mount Doom. Understanding the inner workings of my models was a treacherous path, fraught with complexity and confusion. Yet, HiPlot, like Samwise Gamgee, was my steadfast companion, shedding light on the strengths and weaknesses within my algorithms. <\/p>\n\n<p><strong>Palant\u00edr of R2 and RMSE in One Future<\/strong><\/p>\n\n<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F7ml5otn1bdrcjxgqfcbu.gif\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F7ml5otn1bdrcjxgqfcbu.gif\" alt=\"r2\" width=\"800\" height=\"400\"><\/a><\/p>\n\n<p>Together, we unraveled the mysteries, bringing clarity to chaos. We connected dots of the proportion of the variance for a dependent variable that is explained by independent variables, <code>R2<\/code> by selecting the range of <code>RMSE<\/code>.<\/p>\n\n<h3>\n  \n  \n  Epilogue: HiPlot - The Ring of Power\n<\/h3>\n\n<p>As Frodo's journey transformed him from a simple hobbit to the savior of Middle Earth, HiPlot transformed my approach to data. It wasn\u2019t just a visualization tool; it was a source of truth and insight, a guide through the perilous realm of data science. In the land of models and metrics, HiPlot stands as the One Plot to rule them all, a beacon of clarity in an often-convoluted world.<\/p>\n\n<p><code>Until<\/code> then, keep on coding with the Horn of Rohan<\/p>\n\n","category":["tutorial","machinelearning","beginners","python"]},{"title":"Down the Data Rabbit Hole: Alice's Adventure from Pandas to DuckDB Wonderland","pubDate":"Thu, 16 Nov 2023 20:40:15 +0000","link":"https:\/\/dev.to\/proteusiq\/down-the-data-rabbit-hole-alices-adventure-from-pandas-to-duckdb-wonderland-25h0","guid":"https:\/\/dev.to\/proteusiq\/down-the-data-rabbit-hole-alices-adventure-from-pandas-to-duckdb-wonderland-25h0","description":"<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fd0yam0ahzhmu0gnlikiz.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fd0yam0ahzhmu0gnlikiz.png\" alt=\"Alice falling\" width=\"800\" height=\"599\"><\/a><\/p>\n\n<h3>\n  \n  \n  Prologue: Falling Down the Data Rabbit Hole\n<\/h3>\n\n<p>In a universe where data unfolds like a never-ending story, meet Alice, a data analyst with an adventurous spirit. One day, a flicker of curiosity leads her to an unexpected rabbit hole. With a heart full of excitement, she dives in. This is no ordinary journey: it\u2019s Alice\u2019s vibrant plunge from the familiar world of Pandas into the magical realm of DuckDB. Join her as she discovers Wonderland on this captivating adventure, where every discovery is a thrill and every insight a delight.<\/p>\n\n\n\n\n<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F04q5vpcyjal6fsqo0j63.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F04q5vpcyjal6fsqo0j63.png\" alt=\"tea party\" width=\"800\" height=\"599\"><\/a><\/p>\n\n<h3>\n  \n  \n  <strong>Chapter 1: The Pandas Tea Party<\/strong>\n<\/h3>\n\n<p>Alice found herself at a peculiar tea party, hosted by none other than the Mad Hatter, known to the data world as Pandas. Surrounded by stacks of CSV files, she began her usual routine:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">import<\/span> <span class=\"n\">pandas<\/span> <span class=\"k\">as<\/span> <span class=\"n\">pd<\/span>\n<span class=\"kn\">from<\/span> <span class=\"n\">pathlib<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Path<\/span>\n\n<span class=\"c1\"># At the Pandas tea party, surrounded by piles of CSV files\n<\/span><span class=\"n\">file_list<\/span> <span class=\"o\">=<\/span> <span class=\"nc\">Path<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">\/teaparty<\/span><span class=\"sh\">\"<\/span><span class=\"p\">).<\/span><span class=\"nf\">glob<\/span><span class=\"p\">(<\/span><span class=\"sh\">'<\/span><span class=\"s\">cards*.csv<\/span><span class=\"sh\">'<\/span><span class=\"p\">)<\/span>\n<span class=\"n\">concatenated_df<\/span> <span class=\"o\">=<\/span> <span class=\"n\">pd<\/span><span class=\"p\">.<\/span><span class=\"nf\">concat<\/span><span class=\"p\">((<\/span><span class=\"n\">pd<\/span><span class=\"p\">.<\/span><span class=\"nf\">read_csv<\/span><span class=\"p\">(<\/span><span class=\"nb\">file<\/span><span class=\"p\">)<\/span> \n<span class=\"k\">for<\/span> <span class=\"nb\">file<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">file_list<\/span><span class=\"p\">),<\/span> \n<span class=\"n\">ignore_index<\/span><span class=\"o\">=<\/span><span class=\"bp\">True<\/span><span class=\"p\">)<\/span>\n\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>As she worked, concatenating file after file, Alice couldn't help but feel overwhelmed. The process was familiar but cumbersome, much like the chaotic chatter around the tea party table.<\/p>\n\n\n\n\n<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fpxors4i6aozbalnp2o1j.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fpxors4i6aozbalnp2o1j.png\" alt=\"chasing\" width=\"800\" height=\"599\"><\/a><\/p>\n\n<h2>\n  \n  \n  <strong>Chapter 2: Chasing the DuckDB White Rabbit<\/strong>\n<\/h2>\n\n<p>Just then, Alice spotted a White Rabbit, darting past the tea party. This rabbit, known as DuckDB, beckoned her with the promise of a more efficient way to handle her data. Intrigued, Alice followed and discovered a world where data was not a burden but a delight:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">import<\/span> <span class=\"n\">duckdb<\/span>\n\n<span class=\"c1\"># Chasing the White Rabbit into a world of simpler data handling\n<\/span><span class=\"n\">concatenated_df<\/span> <span class=\"o\">=<\/span> <span class=\"n\">duckdb<\/span><span class=\"p\">.<\/span><span class=\"nf\">query<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"\"\"<\/span><span class=\"s\">\n      SELECT \n         * \n      FROM <\/span><span class=\"sh\">'<\/span><span class=\"s\">.\/teaparty\/cards*.csv<\/span><span class=\"sh\">'<\/span><span class=\"s\">\n<\/span><span class=\"sh\">\"\"\"<\/span><span class=\"p\">).<\/span><span class=\"nf\">df<\/span><span class=\"p\">()<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>In awe, Alice watched as multiple files blended into one with a simple query, no longer needing to juggle them manually. The White Rabbit had shown her a more elegant and less memory-intensive path.<\/p>\n\n\n\n\n<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fgjej5k6kzu3l0890iz3c.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fgjej5k6kzu3l0890iz3c.png\" alt=\"queen\" width=\"800\" height=\"599\"><\/a><\/p>\n\n<h2>\n  \n  \n  <strong>Chapter 3: The DuckDB Wonderland of Transformations<\/strong>\n<\/h2>\n\n<p>As Alice ventured deeper into this wonderland, she encountered the Queen of Hearts, who demanded swift and complex data transformations. In the realm of DuckDB, this was not only possible but surprisingly straightforward:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"n\">transformed_df<\/span> <span class=\"o\">=<\/span> <span class=\"n\">duckdb<\/span><span class=\"p\">.<\/span><span class=\"nf\">query<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"\"\"<\/span><span class=\"s\">\n    SELECT\n        queen,\n        SUM(soldier) as total_soldiers,\n        AVG(hearts) as avg_hearts\n    FROM <\/span><span class=\"sh\">'<\/span><span class=\"s\">.\/teaparty\/cards*.csv<\/span><span class=\"sh\">'<\/span><span class=\"s\">\n    GROUP BY queen\n<\/span><span class=\"sh\">\"\"\"<\/span><span class=\"p\">).<\/span><span class=\"nf\">df<\/span><span class=\"p\">()<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>With DuckDB, Alice easily satisfied the Queen's demands, performing tasks that once seemed as daunting as painting the roses red.<\/p>\n\n<h2>\n  \n  \n  <strong>Epilogue: Awakening in a New World of Data Processing<\/strong>\n<\/h2>\n\n<p>As Alice awoke from her journey, she found herself back in her world, but with a new perspective on data processing. The lessons from DuckDB Wonderland had transformed her approach, making her tasks more efficient and her analyses more powerful.<\/p>\n\n<p>Join Alice in her adventure and explore the wonders of DuckDB. Let it transform your data processing tasks from a mundane chore into a magical experience, just like a day spent in Wonderland.<\/p>\n\n\n\n\n<p>Until then, keep on coding like a Cheshire Cat. <\/p>\n\n","category":["python","pandas","duckdb","beginners"]},{"title":"Web Scraping\u2019s Evolution: From Vintage Vibes to Futuristic Feels","pubDate":"Thu, 19 Oct 2023 17:28:51 +0000","link":"https:\/\/dev.to\/proteusiq\/web-scrapings-evolution-from-vintage-vibes-to-futuristic-feels-3nma","guid":"https:\/\/dev.to\/proteusiq\/web-scrapings-evolution-from-vintage-vibes-to-futuristic-feels-3nma","description":"<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fwyzzi02rl8q9nxmjjyws.jpeg\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fwyzzi02rl8q9nxmjjyws.jpeg\" alt=\"cats\" width=\"800\" height=\"800\"><\/a><br>\nHold onto your keyboards and brew that extra cup of coffee, because we're about to dive headfirst into the rollercoaster ride that is the evolution of web scraping in Python. From the primitive age of <code>sockets<\/code> to the futuristic promise of <code>httpx<\/code>, it's a tale of code, grit, and snake bytes!<\/p>\n<h3>\n  \n  \n  The Stone Age: Direct Sockets\n<\/h3>\n\n<p>Imagine a time when there were no libraries, just you, raw bytes, and a connection to the server.<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">import<\/span> <span class=\"n\">socket<\/span>\n\n<span class=\"n\">URI<\/span> <span class=\"o\">=<\/span> <span class=\"sh\">\"<\/span><span class=\"s\">data.pr4e.org<\/span><span class=\"sh\">\"<\/span>\n<span class=\"k\">with<\/span> <span class=\"n\">socket<\/span><span class=\"p\">.<\/span><span class=\"nf\">socket<\/span><span class=\"p\">(<\/span><span class=\"n\">socket<\/span><span class=\"p\">.<\/span><span class=\"n\">AF_INET<\/span><span class=\"p\">,<\/span> <span class=\"n\">socket<\/span><span class=\"p\">.<\/span><span class=\"n\">SOCK_STREAM<\/span><span class=\"p\">)<\/span> <span class=\"k\">as<\/span> <span class=\"n\">s<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">s<\/span><span class=\"p\">.<\/span><span class=\"nf\">connect<\/span><span class=\"p\">((<\/span><span class=\"n\">URI<\/span><span class=\"p\">,<\/span> <span class=\"mi\">80<\/span><span class=\"p\">))<\/span>\n    <span class=\"n\">cmd<\/span> <span class=\"o\">=<\/span> <span class=\"sa\">f<\/span><span class=\"sh\">\"<\/span><span class=\"s\">GET \/romeo.txt HTTP\/1.0<\/span><span class=\"se\">\\r\\n<\/span><span class=\"s\">Host: <\/span><span class=\"si\">{<\/span><span class=\"n\">URI<\/span><span class=\"si\">}<\/span><span class=\"se\">\\r\\n\\r\\n<\/span><span class=\"sh\">\"<\/span><span class=\"p\">.<\/span><span class=\"nf\">encode<\/span><span class=\"p\">()<\/span>\n    <span class=\"n\">s<\/span><span class=\"p\">.<\/span><span class=\"nf\">send<\/span><span class=\"p\">(<\/span><span class=\"n\">cmd<\/span><span class=\"p\">)<\/span>\n\n    <span class=\"n\">data<\/span> <span class=\"o\">=<\/span> <span class=\"sa\">b<\/span><span class=\"sh\">''<\/span>\n    <span class=\"k\">while<\/span> <span class=\"bp\">True<\/span><span class=\"p\">:<\/span>\n        <span class=\"n\">response<\/span> <span class=\"o\">=<\/span> <span class=\"n\">s<\/span><span class=\"p\">.<\/span><span class=\"nf\">recv<\/span><span class=\"p\">(<\/span><span class=\"mi\">512<\/span><span class=\"p\">)<\/span>\n        <span class=\"k\">if<\/span> <span class=\"nf\">len<\/span><span class=\"p\">(<\/span><span class=\"n\">response<\/span><span class=\"p\">)<\/span> <span class=\"o\">&lt;<\/span> <span class=\"mi\">1<\/span><span class=\"p\">:<\/span>\n            <span class=\"k\">break<\/span>\n\n        <span class=\"n\">data<\/span> <span class=\"o\">+=<\/span> <span class=\"n\">response<\/span>\n\n<span class=\"nf\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">data<\/span><span class=\"p\">.<\/span><span class=\"nf\">decode<\/span><span class=\"p\">())<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>With <code>sockets<\/code>, you'd ride the raw frontier, establishing a connection straight from the saloon, sending messages to the server in Morse code (or what felt like it). It was raw, it was real, but boy, was it cumbersome.<\/p>\n\n<h3>\n  \n  \n  The Wild West: <code>urllib<\/code>\n<\/h3>\n\n<p>Fast forward, and we reach the Wild West of Python web scraping: the <code>urllib<\/code> era. Now, artists and coders alike didn't need to wrangle with sockets directly. A new dawn, a new way to fetch, and the world of web data was at our fingertips!<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"n\">urllib<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">request<\/span>\n\n<span class=\"n\">URL<\/span> <span class=\"o\">=<\/span> <span class=\"sh\">'<\/span><span class=\"s\">http:\/\/data.pr4e.org\/romeo.txt<\/span><span class=\"sh\">'<\/span>\n<span class=\"n\">response<\/span> <span class=\"o\">=<\/span> <span class=\"n\">request<\/span><span class=\"p\">.<\/span><span class=\"nf\">urlopen<\/span><span class=\"p\">(<\/span><span class=\"n\">URL<\/span><span class=\"p\">)<\/span>\n<span class=\"n\">data<\/span> <span class=\"o\">=<\/span> <span class=\"sa\">b<\/span><span class=\"sh\">''<\/span>\n<span class=\"k\">for<\/span> <span class=\"n\">line<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">response<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">data<\/span> <span class=\"o\">+=<\/span> <span class=\"n\">line<\/span>\n\n<span class=\"nf\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">response<\/span><span class=\"p\">.<\/span><span class=\"n\">headers<\/span><span class=\"p\">.<\/span><span class=\"nf\">as_string<\/span><span class=\"p\">())<\/span>\n<span class=\"nf\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">data<\/span><span class=\"p\">.<\/span><span class=\"nf\">decode<\/span><span class=\"p\">())<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>No more Morse code \u2013 just the pure, distilled essence of web content, ready for digital Wild West saloon consumption. But wait, the saga continues...<\/p>\n\n<p>Before we spotlight the upcoming third-party divas, let's craft our backstage hero, the <code>as_string<\/code> function. It'll elegantly choreograph those <code>headers<\/code> to ensure each starlet delivers a consistent, classic encore.<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">def<\/span> <span class=\"nf\">as_string<\/span><span class=\"p\">(<\/span><span class=\"n\">headers<\/span><span class=\"p\">:<\/span><span class=\"nb\">dict<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">str<\/span><span class=\"p\">:<\/span>\n    <span class=\"sh\">\"\"\"<\/span><span class=\"s\">\n    Pretty Prints Dictionary  \n    <\/span><span class=\"sh\">\"\"\"<\/span>\n    <span class=\"n\">text<\/span> <span class=\"o\">=<\/span> <span class=\"p\">(<\/span><span class=\"sh\">'<\/span><span class=\"s\"> <\/span><span class=\"se\">\\n<\/span><span class=\"sh\">'<\/span><span class=\"p\">.<\/span><span class=\"nf\">join<\/span><span class=\"p\">(<\/span><span class=\"sa\">f<\/span><span class=\"sh\">'<\/span><span class=\"si\">{<\/span><span class=\"n\">key<\/span><span class=\"si\">}<\/span><span class=\"s\">: <\/span><span class=\"si\">{<\/span><span class=\"n\">value<\/span><span class=\"si\">}<\/span><span class=\"sh\">'<\/span> \n                 <span class=\"k\">for<\/span> <span class=\"n\">key<\/span><span class=\"p\">,<\/span> <span class=\"n\">value<\/span> \n                 <span class=\"ow\">in<\/span> <span class=\"n\">headers<\/span><span class=\"p\">.<\/span><span class=\"nf\">items<\/span><span class=\"p\">()))<\/span>\n\n    <span class=\"k\">return<\/span> <span class=\"sa\">f<\/span><span class=\"sh\">\"<\/span><span class=\"si\">{<\/span><span class=\"n\">text<\/span><span class=\"si\">}<\/span><span class=\"s\"> <\/span><span class=\"se\">\\n\\n<\/span><span class=\"sh\">\"<\/span>\n\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<h3>\n  \n  \n  The Rockstar: <code>requests<\/code>\n<\/h3>\n\n<p>Enter the rockstar of web scraping: <code>requests<\/code>. With its leather jacket and cool demeanour, <code>requests<\/code> was like the guitarist who just knew how to shred:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">import<\/span> <span class=\"n\">requests<\/span>\n\n<span class=\"n\">URL<\/span> <span class=\"o\">=<\/span> <span class=\"sh\">'<\/span><span class=\"s\">http:\/\/data.pr4e.org\/romeo.txt<\/span><span class=\"sh\">'<\/span>\n<span class=\"n\">response<\/span> <span class=\"o\">=<\/span> <span class=\"n\">requests<\/span><span class=\"p\">.<\/span><span class=\"nf\">get<\/span><span class=\"p\">(<\/span><span class=\"n\">URL<\/span><span class=\"p\">)<\/span>\n\n<span class=\"nf\">print<\/span><span class=\"p\">(<\/span><span class=\"nf\">as_string<\/span><span class=\"p\">(<\/span><span class=\"n\">response<\/span><span class=\"p\">.<\/span><span class=\"n\">headers<\/span><span class=\"p\">))<\/span>\n<span class=\"nf\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">response<\/span><span class=\"p\">.<\/span><span class=\"n\">text<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>No more manual labor. Everything from session management to cookie handling was as effortless as playing an air guitar. Life was good, but the stage was set for the next sensation.<\/p>\n\n<h3>\n  \n  \n  The Futurist: <code>httpx<\/code>\n<\/h3>\n\n<p>In the neon glow of the future, where the world buzzes with asynchronous operations, <code>httpx<\/code> made its entrance, floating on the cloud of efficiency:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">import<\/span> <span class=\"n\">httpx<\/span>\n\n<span class=\"c1\"># response = httpx.get(URL) # synchronous \n<\/span>\n<span class=\"c1\"># asynchronous magic\n<\/span><span class=\"k\">async<\/span> <span class=\"k\">with<\/span> <span class=\"n\">httpx<\/span><span class=\"p\">.<\/span><span class=\"nc\">AsyncClient<\/span><span class=\"p\">()<\/span> <span class=\"k\">as<\/span> <span class=\"n\">client<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">response<\/span> <span class=\"o\">=<\/span> <span class=\"k\">await<\/span> <span class=\"n\">client<\/span><span class=\"p\">.<\/span><span class=\"nf\">get<\/span><span class=\"p\">(<\/span><span class=\"n\">URL<\/span><span class=\"p\">)<\/span>\n\n<span class=\"nf\">print<\/span><span class=\"p\">(<\/span><span class=\"nf\">as_string<\/span><span class=\"p\">(<\/span><span class=\"n\">response<\/span><span class=\"p\">.<\/span><span class=\"n\">headers<\/span><span class=\"p\">))<\/span>\n<span class=\"nf\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">response<\/span><span class=\"p\">.<\/span><span class=\"n\">text<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>With both synchronous rhythms and asynchronous beats, <code>httpx<\/code> danced its way into the heart of developers, promising a brighter, faster web scraping future.<\/p>\n\n<h3>\n  \n  \n  Epilogue of Made-up Story\n<\/h3>\n\n<p>So, there you have it: a tale of trials, triumphs, and transformations. From raw connections to refined libraries, web scraping in Python has been a journey worth every byte. As we look to the future, one thing's for sure: the adventure is far from over. <\/p>\n\n<h3>\n  \n  \n  Output: <code>stdout.flush()<\/code>\n<\/h3>\n\n<p>The print outputs of all code above are like:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight shell\"><code>\nHTTP\/1.1 200 OK\nDate: Fri, 20 Oct 2023 08:20:33 GMT\nServer: Apache\/2.4.18 <span class=\"o\">(<\/span>Ubuntu<span class=\"o\">)<\/span>\nLast-Modified: Sat, 13 May 2017 11:22:22 GMT\nETag: <span class=\"s2\">\"a7-54f6609245537\"<\/span>\nAccept-Ranges: bytes\nContent-Length: 167\nCache-Control: max-age<span class=\"o\">=<\/span>0, no-cache, no-store, must-revalidate\nPragma: no-cache\nExpires: Wed, 11 Jan 1984 05:00:00 GMT\nConnection: close\nContent-Type: text\/plain\n\nBut soft what light through yonder window breaks\nIt is the east and Juliet is the sun\nArise fair sun and <span class=\"nb\">kill <\/span>the envious moon\nWho is already sick and pale with grief\n<\/code><\/pre>\n\n<\/div>\n\n\n\n\n\n\n<p>Until then, keep on scraping and may your scrapes always return <code>200 OK<\/code>!<\/p>\n\n","category":["tutorial","beginners","python","webdev"]},{"title":"A One-Stop Guide to Data Science Resources","pubDate":"Mon, 16 Oct 2023 11:34:33 +0000","link":"https:\/\/dev.to\/proteusiq\/the-ultimate-one-stop-guide-to-data-science-resources-4in6","guid":"https:\/\/dev.to\/proteusiq\/the-ultimate-one-stop-guide-to-data-science-resources-4in6","description":"<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F2vniiiny4v0s9hc6qgnv.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F2vniiiny4v0s9hc6qgnv.png\" alt=\"search\" width=\"800\" height=\"795\"><\/a><br>\nThe landscape of data science is vast and ever-evolving. With the myriad of resources available, finding a comprehensive list that encapsulates the essence of this field can be daunting. I have curated a guide that offers a holistic view of data science, from understanding classical machine learning algorithms to Bayesian inference and deep learning.<\/p>\n\n<p>I have hand-selected <em>free<\/em> courses, ebooks, interactive webpages and podcasts that have played a role in my data science journey. Pick one or two and dive deep.<\/p>\n\n<p>\ud83d\udd0b - requires more brain power <br>\n\ud83e\udeab - requires less brain power<br>\n\ud83d\udc8e - personal favourites <br>\n\ud83d\udc51 - unique <\/p>\n\n\n\n\n<h2>\n  \n  \n  Understanding Machine Learning Predictions\n<\/h2>\n\n<p>We start with the foundational resources that introduce classical statistical and machine-learning algorithms<\/p>\n\n<p>\ud83d\udd0b\ud83d\udd0b\ud83e\udeab\ud83e\udeab\ud83e\udeab<\/p>\n\n<ul>\n<li><p><a href=\"https:\/\/jakevdp.github.io\/PythonDataScienceHandbook\/\" rel=\"noopener noreferrer\">Python Data Science Handbook<\/a> - Jake VanderPlas<\/p><\/li>\n<li><p><a href=\"https:\/\/www.statlearning.com\/\" rel=\"noopener noreferrer\">An Introduction to Statistical Learning<\/a> - Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani - <a href=\"https:\/\/hastie.su.domains\/ISLP\/ISLP_website.pdf\" rel=\"noopener noreferrer\">\\w Python PDF<\/a><\/p><\/li>\n<li><p><a href=\"https:\/\/mml-book.github.io\/\" rel=\"noopener noreferrer\">Mathematics for Machine Learning<\/a> - PDF, video and Jupyter Notebooks for introductions to mathematics required for ML<\/p><\/li>\n<li><p><a href=\"https:\/\/algorithmsbook.com\/\" rel=\"noopener noreferrer\">Algorithms for Decision Making<\/a> - \ud83d\udc8e an extensive overview of decision-making algorithms under uncertainty, with mathematical formulations and solutions.<\/p><\/li>\n<li><p><a href=\"https:\/\/www.cs.huji.ac.il\/~shais\/UnderstandingMachineLearning\/understanding-machine-learning-theory-algorithms.pdf\" rel=\"noopener noreferrer\">Understanding Machine Learning: From Theory to Algorithms<\/a> - Cambridge PDF - Shai Shalev Shwartz and Ben David<\/p><\/li>\n<li><p><a href=\"https:\/\/pll.harvard.edu\/course\/cs50s-introduction-artificial-intelligence-python?delta=0\" rel=\"noopener noreferrer\">CS50's Introduction to Artificial Intelligence with Python<\/a> - Harvard School of Engineering and Applied Sciences <\/p><\/li>\n<li><p><a href=\"https:\/\/openlearninglibrary.mit.edu\/courses\/course-v1:MITx+6.036+1T2019\/course\/\" rel=\"noopener noreferrer\">Introduction to Machine Learning<\/a> - MIT Open Learning Library<\/p><\/li>\n<li><p><a href=\"https:\/\/see.stanford.edu\/Course\/CS229\" rel=\"noopener noreferrer\">CS229 - Machine Learning<\/a> - Andrew Ng Stanford University<\/p><\/li>\n<li><p><a href=\"https:\/\/github.com\/jonkrohn\/ML-foundations\" rel=\"noopener noreferrer\">Machine Learning Foundations<\/a> - Classical Notebooks, Udemy|Youtube course covering Maths and Code by Jon Khrohn<\/p><\/li>\n<li><p><a href=\"https:\/\/mlcourse.ai\/book\/index.html\" rel=\"noopener noreferrer\">mlcourse.ai<\/a> - Open Machine Learning Course<\/p><\/li>\n<\/ul>\n\n\n\n\n<h3>\n  \n  \n  Deep + Reinforcement Learning\n<\/h3>\n\n<p>\ud83d\udd0b\ud83d\udd0b\ud83d\udd0b\ud83e\udeab\ud83e\udeab<br>\nThe world of neural networks and deep architectures is vast. <em>MIT 6.S191<\/em> offers a comprehensive introduction, while resources like <em>Deep Learning for Coders with fastai &amp; PyTorch<\/em> bridge the gap between theory and real-world applications.<\/p>\n\n<ul>\n<li>\n<a href=\"http:\/\/introtodeeplearning.com\/\" rel=\"noopener noreferrer\">MIT 6.S191 Introduction to Deep Learning<\/a> <\/li>\n<li>\n<a href=\"https:\/\/atcold.github.io\/pytorch-Deep-Learning\/\" rel=\"noopener noreferrer\">Deep Learning<\/a> - DS-GA 1008 \u00b7 Spring 2020 \u00b7 NYU Center For Data Science<\/li>\n<li>\n<a href=\"https:\/\/course.fast.ai\/Resources\/book.html\" rel=\"noopener noreferrer\">Deep Learning\nfor Coders with\nfastai &amp; PyTorch<\/a> - Jeremy Howard &amp;\nSylvain Gugger - <a href=\"https:\/\/course.fast.ai\/\" rel=\"noopener noreferrer\">Practical Deep Learning<\/a> course<\/li>\n<li>\n<a href=\"http:\/\/rail.eecs.berkeley.edu\/deeprlcourse\/\" rel=\"noopener noreferrer\">Deep Reinforcement Learning<\/a> - CS 285 at UC Berkeley<\/li>\n<li><p><a href=\"https:\/\/huggingface.co\/learn\/deep-rl-course\/unit1\/rl-framework\" rel=\"noopener noreferrer\">\ud83e\udd17 Huggingface Reinforcement Learning<\/a><\/p><\/li>\n<li><p><a href=\"https:\/\/mlcourse.ai\/book\/index.html\" rel=\"noopener noreferrer\">mlcourse.ai<\/a> - Open Machine Learning Course<\/p><\/li>\n<li>\n<p><a href=\"https:\/\/huggingface.co\/learn\/nlp-course\/en\/chapter1\/1?fw=pt\" rel=\"noopener noreferrer\">Natural Language Processing<\/a> <\/p>\n\n\n<h3>\n  \n  \n  Seeing Machine Learning Algorithms\n<\/h3>\n\n<p>\ud83d\udd0b\ud83e\udeab\ud83e\udeab\ud83e\udeab\ud83e\udeab<br>\nFor visual learners, this category is a goldmine. From <em>Illustrated ML<\/em> to <em>Distill<\/em>, these resources visually break down complex ML concepts, making them accessible and engaging.<br><\/p>\n<\/li>\n<li><p><a href=\"https:\/\/illustrated-machine-learning.github.io\/\" rel=\"noopener noreferrer\">Illustrated ML<\/a> - \ud83d\udc51 Aims to simplify the intricate world of Machine Learning with clear illustrations.<\/p><\/li>\n<li><p><a href=\"https:\/\/seeing-theory.brown.edu\/index.html\" rel=\"noopener noreferrer\">Seeing Theory<\/a> - A visual introduction to probability and statistics \ud83d\udc8e<\/p><\/li>\n<li><p><a href=\"https:\/\/mlu-explain.github.io\/\" rel=\"noopener noreferrer\">MLU-Expl{ai}n<\/a> - Visual explanations of core machine learning concepts<\/p><\/li>\n<li><p><a href=\"https:\/\/distill.pub\/\" rel=\"noopener noreferrer\">Distill<\/a> - Machine Learning Research Should Be Clear, Dynamic and Vivid<\/p><\/li>\n<\/ul>\n\n\n\n\n<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fbdd4eigf0ib55pjk2u7l.jpeg\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fbdd4eigf0ib55pjk2u7l.jpeg\" alt=\"bayesian\" width=\"800\" height=\"800\"><\/a> <\/p>\n\n<h3>\n  \n  \n  Bayesian Modelling: The White Box Machine Learning\n<\/h3>\n\n<p>\ud83d\udd0b\ud83d\udd0b\ud83d\udd0b\ud83d\udd0b\ud83e\udeab<br>\nBayesian statistics has gained significant traction in the data science community. Resources like <em>Think Bayes<\/em> and <em>Bayesian Methods for Hackers<\/em> provide a perfect blend of theory and application. For those seeking depth, Richard McElreath's <em>Statistical Rethinking<\/em> lectures are a treasure trove.<\/p>\n\n<ul>\n<li>\n<a href=\"http:\/\/allendowney.github.io\/ThinkBayes2\" rel=\"noopener noreferrer\">Think Bayes<\/a> - Allen Downey\u2019s classic as Jupyter Book<\/li>\n<li>\n<a href=\"https:\/\/bayesiancomputationbook.com\/welcome.html\" rel=\"noopener noreferrer\">Bayesian Modeling and Computation in Python<\/a> - Martin Osvaldo A, Kumar Ravin; Lao Junpeng, 2021<\/li>\n<li>\n<a href=\"http:\/\/camdavidsonpilon.github.io\/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers\" rel=\"noopener noreferrer\">Bayesian Methods for Hackers<\/a> - Probabilistic Programming and Bayesian Inference - DevAuthors<\/li>\n<li>\n<a href=\"https:\/\/github.com\/Booleans\/statistical-rethinking\/raw\/master\/Statistical%20Rethinking%202nd%20Edition.pdf\" rel=\"noopener noreferrer\">Statistical Rethinking 2019<\/a> - \ud83d\udc51 <a href=\"https:\/\/youtube.com\/playlist?list=PLDcUM9US4XdM9_N6XUUFrhghGJ4K25bFc\" rel=\"noopener noreferrer\">Course Fall 2017<\/a> + <a href=\"https:\/\/www.youtube.com\/playlist?list=PLDcUM9US4XdMROZ57-OIRtIK0aOynbgZN\" rel=\"noopener noreferrer\">Pre-recorded Lectures 2022<\/a> - <a href=\"https:\/\/github.com\/rmcelreath\/stat_rethinking_2022\" rel=\"noopener noreferrer\">Material 2022<\/a> - <a href=\"https:\/\/xcelab.net\/rm\/\" rel=\"noopener noreferrer\">Richard McElreath<\/a>'s lectures from Leipzig University - <a href=\"https:\/\/github.com\/pymc-devs\/pymc-resources\/tree\/main\/Rethinking_2\" rel=\"noopener noreferrer\">PyMC codes<\/a>\n<\/li>\n<li>\n<a href=\"https:\/\/www.bayesrulesbook.com\/\" rel=\"noopener noreferrer\">Bayes Rules!<\/a> - An Introduction to Applied Bayesian Modeling - Alicia A. Johnson, Miles Q. Ott, and  Mine Dogucu - <a href=\"https:\/\/github.com\/pymc-devs\/pymc-resources\/tree\/main\/Bayes_Rules\" rel=\"noopener noreferrer\">PyMC codes<\/a> - Jim Albert and Jingchen Hu - GitBook<\/li>\n<li>\n<a href=\"https:\/\/bayesball.github.io\/BOOK\/probability-a-measurement-of-uncertainty.html\" rel=\"noopener noreferrer\">Probability and Bayesian Modeling<\/a> -<\/li>\n<li>\n<a href=\"https:\/\/probml.github.io\/pml-book\/\" rel=\"noopener noreferrer\">Probabilistic Machine Learning<\/a> - a book series by Kevin Murphy + <a href=\"https:\/\/github.com\/probml\" rel=\"noopener noreferrer\">GitHub materials<\/a>\n<\/li>\n<li>\n<a href=\"https:\/\/statsthinking21.github.io\/statsthinking21-core-site\/\" rel=\"noopener noreferrer\">Statistical Thinking for the 21st Century<\/a> - Russell A. Poldrack's GitBook<\/li>\n<li>\n<a href=\"https:\/\/statswithr.github.io\/book\/\" rel=\"noopener noreferrer\">Bayesian Thinking<\/a> - A Companion to the Statistics with R Course<\/li>\n<li><a href=\"https:\/\/matheusfacure.github.io\/python-causality-handbook\/landing-page.html\" rel=\"noopener noreferrer\">Causal Inference for The Brave and True<\/a><\/li>\n<li>\n<a href=\"https:\/\/distribution-explorer.github.io\/index.html#\" rel=\"noopener noreferrer\">Probability Distribution Explorer<\/a> - Probability distributions and the stories behind<\/li>\n<\/ul>\n\n\n\n\n<h3>\n  \n  \n  Extras: Convex Optimization\n<\/h3>\n\n<p>Optimization is at the core of many algorithms. Boyd and Vandenberghe's <em>Convex Optimization<\/em> is a staple, and the courses from Stanford provide a deeper understanding.<\/p>\n\n<ul>\n<li>\n<a href=\"https:\/\/web.stanford.edu\/~boyd\/cvxbook\/\" rel=\"noopener noreferrer\">Convex Optimization<\/a> \u2013 Boyd and Vandenberghe<\/li>\n<li>\n<a href=\"https:\/\/see.stanford.edu\/Course\/EE364A\" rel=\"noopener noreferrer\">EE364A - Convex Optimization I<\/a> &amp; <a href=\"https:\/\/see.stanford.edu\/Course\/EE364B\" rel=\"noopener noreferrer\">II<\/a> - Stephen P. Boyd - Stanford University <\/li>\n<\/ul>\n\n<h3>\n  \n  \n  Podcasts\n<\/h3>\n\n<p>For those on-the-go, podcasts like <em>Learning Bayesian Statistics<\/em> and <em>Linear Digressions<\/em> provide insights into data science trends, methodologies, and applications.<\/p>\n\n<ul>\n<li>\n<a href=\"https:\/\/podcasts.apple.com\/dk\/podcast\/learning-bayesian-statistics\/id1483485062\" rel=\"noopener noreferrer\">Learning Bayesian Statistics<\/a> - Up-to-date dialogue on Bayesian inference<\/li>\n<li>\n<a href=\"http:\/\/lineardigressions.com\/\" rel=\"noopener noreferrer\">Linear Digressions<\/a> -  Host: Katie Malone &amp; Ben Jaffe - Ended 2020 ML dialogue<\/li>\n<\/ul>\n\n\n\n\n<p>This curated list has been my one-stop guide collected when I was a beginner and seasoned professional in data science. Whether you're a visual learner, an avid reader, or someone who learns by doing, there's something here for everyone. Dive in and let the exploration begin!<\/p>\n\n<p>If you have a resource worth adding, fire it on the comments.<\/p>\n\n<p>Until then, keep on learning \u2026<\/p>\n\n","category":["learning","machinelearning","education"]},{"title":"Odyssey to Python Mastery: 4 Jedi Techniques","pubDate":"Mon, 02 Oct 2023 13:16:54 +0000","link":"https:\/\/dev.to\/proteusiq\/odyssey-to-python-mastery-4-jedi-techniques-46k5","guid":"https:\/\/dev.to\/proteusiq\/odyssey-to-python-mastery-4-jedi-techniques-46k5","description":"<p>Enfold the odyssey to Python brilliance, a journey illuminated with the allure of elegance and efficiency. This article, inspired by <a href=\"https:\/\/github.com\/Proteusiq\/saa\" rel=\"noopener noreferrer\">Saa<\/a>, my Python package that translates time into human-friendly spoken expressions, takes on a voyage to explore four advanced Python concepts, unearthing the true prowess of Python language. With every stride, witness your code transform, mirroring the elegance of poetry and the finesse of an art masterpiece.<\/p>\n\n<p>\u26a0\ufe0f <em>code heavy: set out in not-too-distant galax, this article explores advance concepts<\/em> <\/p>\n\n\n\n\n<h2>\n  \n  \n  1. <strong>Single Dispatcher: The Chameleon Functions<\/strong>\n<\/h2>\n\n<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F1xawg1l6szvjfj5dap6s.jpeg\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F1xawg1l6szvjfj5dap6s.jpeg\" alt=\"chameleon\" width=\"800\" height=\"800\"><\/a><\/p>\n\n<p>Observe the allure of <code>singledispatch<\/code> from <code>functools<\/code>, a decorator that bestows your functions the gift to morph based on the type of the first argument, a beacon for writing seamless generic code.<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"n\">__future__<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">annotations<\/span>\n<span class=\"kn\">from<\/span> <span class=\"n\">functools<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">singledispatch<\/span>\n<span class=\"kn\">from<\/span> <span class=\"n\">datetime<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">time<\/span><span class=\"p\">,<\/span> <span class=\"n\">datetime<\/span>\n\n\n<span class=\"n\">TimeType<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">str<\/span> <span class=\"o\">|<\/span> <span class=\"n\">time<\/span> <span class=\"o\">|<\/span> <span class=\"n\">datetime<\/span>\n\n\n<span class=\"nd\">@singledispatch<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">clock<\/span><span class=\"p\">(<\/span><span class=\"n\">_<\/span><span class=\"p\">:<\/span> <span class=\"n\">TimeType<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">time<\/span><span class=\"p\">:<\/span>\n    <span class=\"sh\">\"\"\"<\/span><span class=\"s\">Clock Parser\n\n    Accepts string, time or datetime and return time object\n\n    Args:\n        _ (TimeType): string, time or datetime object\n\n    Raises:\n        NotImplementedError: shell for dispatching\n\n    Returns:\n        time: python time object\n    <\/span><span class=\"sh\">\"\"\"<\/span>\n    <span class=\"k\">raise<\/span> <span class=\"nb\">NotImplementedError<\/span>\n\n\n<span class=\"nd\">@clock.register<\/span><span class=\"p\">(<\/span><span class=\"nb\">str<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">_<\/span><span class=\"p\">(<\/span><span class=\"n\">t<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">time<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">datetime<\/span><span class=\"p\">.<\/span><span class=\"nf\">strptime<\/span><span class=\"p\">(<\/span><span class=\"n\">t<\/span><span class=\"p\">,<\/span> <span class=\"sh\">\"<\/span><span class=\"s\">%H:%M<\/span><span class=\"sh\">\"<\/span><span class=\"p\">).<\/span><span class=\"nf\">time<\/span><span class=\"p\">()<\/span>\n\n\n<span class=\"nd\">@clock.register<\/span><span class=\"p\">(<\/span><span class=\"n\">datetime<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">_<\/span><span class=\"p\">(<\/span><span class=\"n\">t<\/span><span class=\"p\">:<\/span> <span class=\"n\">datetime<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">time<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">t<\/span><span class=\"p\">.<\/span><span class=\"nf\">time<\/span><span class=\"p\">()<\/span>\n\n\n<span class=\"nd\">@clock.register<\/span><span class=\"p\">(<\/span><span class=\"n\">time<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">_<\/span><span class=\"p\">(<\/span><span class=\"n\">t<\/span><span class=\"p\">:<\/span> <span class=\"n\">time<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">time<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">t<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Test we must, young Padawan. Reflect upon the use cases, we shall. In the vast galaxy where stars twinkle, mirror the universe\u2019s myriad possibilities, our trials will.<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"n\">datetime<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">datetime<\/span><span class=\"p\">,<\/span> <span class=\"n\">time<\/span>\n<span class=\"kn\">import<\/span> <span class=\"n\">pytest<\/span>\n<span class=\"kn\">import<\/span> <span class=\"n\">clock<\/span>  \n\n\n<span class=\"nd\">@pytest.fixture<\/span><span class=\"p\">(<\/span><span class=\"n\">params<\/span><span class=\"o\">=<\/span><span class=\"p\">[<\/span><span class=\"sh\">\"<\/span><span class=\"s\">12:34<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">datetime<\/span><span class=\"p\">.<\/span><span class=\"nf\">now<\/span><span class=\"p\">().<\/span><span class=\"nf\">replace<\/span><span class=\"p\">(<\/span><span class=\"n\">hour<\/span><span class=\"o\">=<\/span><span class=\"mi\">12<\/span><span class=\"p\">,<\/span> <span class=\"n\">minute<\/span><span class=\"o\">=<\/span><span class=\"mi\">34<\/span><span class=\"p\">),<\/span> <span class=\"nf\">time<\/span><span class=\"p\">(<\/span><span class=\"mi\">12<\/span><span class=\"p\">,<\/span> <span class=\"mi\">34<\/span><span class=\"p\">)])<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">valid_time_input<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">yield<\/span> <span class=\"n\">request<\/span><span class=\"p\">.<\/span><span class=\"n\">param<\/span>\n\n\n<span class=\"nd\">@pytest.fixture<\/span><span class=\"p\">(<\/span><span class=\"n\">params<\/span><span class=\"o\">=<\/span><span class=\"p\">[<\/span><span class=\"mi\">1234<\/span><span class=\"p\">,<\/span> <span class=\"mf\">12.34<\/span><span class=\"p\">,<\/span> <span class=\"p\">[<\/span><span class=\"mi\">12<\/span><span class=\"p\">,<\/span> <span class=\"mi\">34<\/span><span class=\"p\">],<\/span> <span class=\"p\">{<\/span><span class=\"sh\">\"<\/span><span class=\"s\">hour<\/span><span class=\"sh\">\"<\/span><span class=\"p\">:<\/span> <span class=\"mi\">12<\/span><span class=\"p\">,<\/span> <span class=\"sh\">\"<\/span><span class=\"s\">minute<\/span><span class=\"sh\">\"<\/span><span class=\"p\">:<\/span> <span class=\"mi\">34<\/span><span class=\"p\">}])<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">invalid_time_input<\/span><span class=\"p\">(<\/span><span class=\"n\">request<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">yield<\/span> <span class=\"n\">request<\/span><span class=\"p\">.<\/span><span class=\"n\">param<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">test_clock_with_valid_input<\/span><span class=\"p\">(<\/span><span class=\"n\">valid_time_input<\/span><span class=\"p\">):<\/span>\n    <span class=\"sh\">\"<\/span><span class=\"s\">Test with valid inputs<\/span><span class=\"sh\">\"<\/span>\n\n    <span class=\"n\">result<\/span> <span class=\"o\">=<\/span> <span class=\"nf\">clock<\/span><span class=\"p\">(<\/span><span class=\"n\">valid_time_input<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"nf\">isinstance<\/span><span class=\"p\">(<\/span><span class=\"n\">result<\/span><span class=\"p\">,<\/span> <span class=\"n\">time<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"n\">result<\/span><span class=\"p\">.<\/span><span class=\"n\">hour<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">12<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"n\">result<\/span><span class=\"p\">.<\/span><span class=\"n\">minute<\/span> <span class=\"o\">==<\/span> <span class=\"mi\">34<\/span>\n\n\n<span class=\"k\">def<\/span> <span class=\"nf\">test_clock_with_invalid_input<\/span><span class=\"p\">(<\/span><span class=\"n\">invalid_time_input<\/span><span class=\"p\">):<\/span>\n    <span class=\"sh\">\"<\/span><span class=\"s\">Test with invalid input, expecting NotImplementedError<\/span><span class=\"sh\">\"<\/span>\n    <span class=\"k\">with<\/span> <span class=\"n\">pytest<\/span><span class=\"p\">.<\/span><span class=\"nf\">raises<\/span><span class=\"p\">(<\/span><span class=\"nb\">NotImplementedError<\/span><span class=\"p\">):<\/span>\n        <span class=\"nf\">clock<\/span><span class=\"p\">(<\/span><span class=\"n\">invalid_time_input<\/span><span class=\"p\">)<\/span>\n\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<h3>\n  \n  \n  Reflect and Contemplate:\n<\/h3>\n\n<p>Pave this path cautiously. Ensure the base function is etched to raise a <code>NotImplementedError<\/code> and adorn each additional implementation with <code>@function_name.register(type)<\/code>.<\/p>\n\n\n\n\n<h2>\n  \n  \n  2. <strong>Setters and Getters: The Shield and Sword<\/strong>\n<\/h2>\n\n<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Foo7wrsvwj2f08lt2c7ak.jpeg\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Foo7wrsvwj2f08lt2c7ak.jpeg\" alt=\"setter\" width=\"800\" height=\"800\"><\/a><\/p>\n\n<p>Conjure the <code>@property<\/code> decorator, your shield and sword, ensuring your objects stand resilient, cloaked in encapsulation.<\/p>\n\n<p>Beginning with elemental steps, let's unveil the potential to not only encapsulate but also validate inputs at the setting.<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"n\">__future__<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">annotations<\/span>\n\n<span class=\"n\">\u03c0<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">22<\/span><span class=\"o\">\/<\/span><span class=\"mi\">7<\/span> <span class=\"c1\">#3.14...\n<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">radius_validator<\/span><span class=\"p\">(<\/span><span class=\"n\">value<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"o\">|<\/span><span class=\"nb\">float<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">int<\/span> <span class=\"o\">|<\/span> <span class=\"nb\">float<\/span><span class=\"p\">:<\/span>\n\n    <span class=\"k\">if<\/span> <span class=\"ow\">not<\/span> <span class=\"nf\">isinstance<\/span><span class=\"p\">(<\/span><span class=\"n\">value<\/span><span class=\"p\">,<\/span> <span class=\"p\">(<\/span><span class=\"nb\">int<\/span><span class=\"p\">,<\/span> <span class=\"nb\">float<\/span><span class=\"p\">)):<\/span>\n        <span class=\"k\">raise<\/span> <span class=\"nc\">TypeError<\/span><span class=\"p\">(<\/span><span class=\"sa\">f<\/span><span class=\"sh\">\"<\/span><span class=\"s\">Radius has to be a positive int or float \ud83d\ude1e<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">elif<\/span> <span class=\"n\">value<\/span> <span class=\"o\">&lt;<\/span> <span class=\"mi\">0<\/span><span class=\"p\">:<\/span>\n        <span class=\"k\">raise<\/span> <span class=\"nc\">ValueError<\/span><span class=\"p\">(<\/span><span class=\"sa\">f<\/span><span class=\"sh\">\"<\/span><span class=\"s\">Radius cannot be negative \ud83d\ude1e<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>       \n    <span class=\"k\">return<\/span> <span class=\"n\">value<\/span>\n\n\n<span class=\"k\">class<\/span> <span class=\"nc\">Circle<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">__init__<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">radius<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"o\">|<\/span><span class=\"nb\">float<\/span><span class=\"p\">):<\/span>\n        <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">radius<\/span> <span class=\"o\">=<\/span> <span class=\"n\">radius<\/span>\n\n    <span class=\"nd\">@property<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">radius<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">int<\/span><span class=\"o\">|<\/span><span class=\"nb\">float<\/span><span class=\"p\">:<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">_radius<\/span>\n\n    <span class=\"nd\">@radius.setter<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">radius<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">value<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"o\">|<\/span><span class=\"nb\">float<\/span><span class=\"p\">):<\/span>\n\n        <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">_radius<\/span> <span class=\"o\">=<\/span> <span class=\"nf\">radius_validator<\/span><span class=\"p\">(<\/span><span class=\"n\">value<\/span><span class=\"p\">)<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">area<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">float<\/span><span class=\"p\">:<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">\u03c0<\/span> <span class=\"o\">*<\/span> <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">_radius<\/span><span class=\"o\">**<\/span><span class=\"mi\">2<\/span> \n\n    <span class=\"k\">def<\/span> <span class=\"nf\">circumference<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">float<\/span><span class=\"p\">:<\/span>\n        <span class=\"k\">return<\/span> <span class=\"mi\">2<\/span> <span class=\"o\">*<\/span> <span class=\"n\">\u03c0<\/span> <span class=\"o\">*<\/span> <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">_radius<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">__repr__<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">):<\/span>\n\n        <span class=\"k\">return<\/span> <span class=\"sa\">f<\/span><span class=\"sh\">\"<\/span><span class=\"s\">area=<\/span><span class=\"si\">{<\/span><span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"nf\">area<\/span><span class=\"p\">()<\/span><span class=\"si\">:<\/span> <span class=\"p\">.<\/span><span class=\"mi\">2<\/span><span class=\"n\">f<\/span><span class=\"si\">}<\/span><span class=\"s\">;circumference=<\/span><span class=\"si\">{<\/span><span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"nf\">circumference<\/span><span class=\"p\">()<\/span><span class=\"si\">:<\/span><span class=\"p\">.<\/span><span class=\"mi\">2<\/span><span class=\"n\">f<\/span><span class=\"si\">}<\/span><span class=\"sh\">\"<\/span>\n\n<span class=\"c1\"># c = Circle(radius=42) \n# c.radius = -1   # =&gt; throws ValueError(f\"Radius cannot be negative \ud83d\ude1e\")\n# c.radius = \"42\" # =&gt; throws TypeError(f\"Radius has to be a positive int or float \ud83d\ude1e\") \n<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Up to this point, things are looking promising. The power of setters and getters extends far beyond mere value validation at assignments; they unlock Jedi-like capabilities. Consider this: in the realm of classical Machine Learning, what if we employed a setter to save a fitted transformer and later, during prediction, utilized a getter to retrieve it? Intriguing, isn't it?<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">import<\/span> <span class=\"n\">pickle<\/span>\n<span class=\"kn\">from<\/span> <span class=\"n\">pathlib<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Path<\/span>\n<span class=\"kn\">from<\/span> <span class=\"n\">typing<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Literal<\/span>\n<span class=\"kn\">import<\/span> <span class=\"n\">pandas<\/span> <span class=\"k\">as<\/span> <span class=\"n\">pd<\/span>\n<span class=\"kn\">from<\/span> <span class=\"n\">sklearn.compose<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">ColumnTransformer<\/span>\n\n\n<span class=\"k\">class<\/span> <span class=\"nc\">TransformerTask<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">__init__<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">transformer<\/span><span class=\"p\">:<\/span><span class=\"n\">ColumnTransformer<\/span><span class=\"p\">,<\/span> \n                 <span class=\"n\">stage<\/span><span class=\"p\">:<\/span> <span class=\"n\">Literal<\/span><span class=\"p\">[<\/span><span class=\"sh\">\"<\/span><span class=\"s\">train<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span> <span class=\"sh\">\"<\/span><span class=\"s\">predict<\/span><span class=\"sh\">\"<\/span><span class=\"p\">]<\/span> <span class=\"o\">=<\/span> <span class=\"sh\">\"<\/span><span class=\"s\">train<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span> \n                 <span class=\"n\">file_path<\/span><span class=\"p\">:<\/span><span class=\"nb\">str<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">models\/transformer.pkl<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,):<\/span>\n        <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">stage<\/span> <span class=\"o\">=<\/span> <span class=\"n\">stage<\/span>\n        <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">file_path<\/span> <span class=\"o\">=<\/span> <span class=\"nc\">Path<\/span><span class=\"p\">(<\/span><span class=\"n\">file_path<\/span><span class=\"p\">)<\/span>\n        <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">_transformer<\/span> <span class=\"o\">=<\/span> <span class=\"n\">transformer<\/span>\n\n    <span class=\"nd\">@property<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">transformer<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">ColumnTransformer<\/span><span class=\"p\">:<\/span>\n        <span class=\"k\">if<\/span> <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">stage<\/span> <span class=\"o\">==<\/span> <span class=\"sh\">\"<\/span><span class=\"s\">predict<\/span><span class=\"sh\">\"<\/span> <span class=\"ow\">and<\/span> <span class=\"ow\">not<\/span> <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">file_path<\/span><span class=\"p\">.<\/span><span class=\"nf\">exists<\/span><span class=\"p\">():<\/span>\n            <span class=\"k\">raise<\/span> <span class=\"nc\">FileNotFoundError<\/span><span class=\"p\">(<\/span><span class=\"sa\">f<\/span><span class=\"sh\">\"<\/span><span class=\"si\">{<\/span><span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">file_path<\/span><span class=\"si\">}<\/span><span class=\"s\"> was not found.<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n\n        <span class=\"k\">if<\/span> <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">stage<\/span> <span class=\"o\">==<\/span> <span class=\"sh\">\"<\/span><span class=\"s\">predict<\/span><span class=\"sh\">\"<\/span> <span class=\"ow\">and<\/span> <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">file_path<\/span><span class=\"p\">.<\/span><span class=\"nf\">exists<\/span><span class=\"p\">():<\/span>\n            <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">_transformer<\/span> <span class=\"o\">=<\/span> <span class=\"n\">pickle<\/span><span class=\"p\">.<\/span><span class=\"nf\">loads<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">file_path<\/span><span class=\"p\">.<\/span><span class=\"nf\">read_bytes<\/span><span class=\"p\">())<\/span>\n\n        <span class=\"k\">return<\/span> <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">_transformer<\/span>\n\n    <span class=\"nd\">@transformer.setter<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">transformer<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">transformer<\/span><span class=\"p\">:<\/span><span class=\"n\">ColumnTransformer<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"bp\">None<\/span><span class=\"p\">:<\/span> \n        <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">file_path<\/span><span class=\"p\">.<\/span><span class=\"nf\">write_bytes<\/span><span class=\"p\">(<\/span><span class=\"n\">pickle<\/span><span class=\"p\">.<\/span><span class=\"nf\">dumps<\/span><span class=\"p\">(<\/span><span class=\"n\">transformer<\/span><span class=\"p\">))<\/span>\n\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">run<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">data<\/span><span class=\"p\">:<\/span> <span class=\"n\">pd<\/span><span class=\"p\">.<\/span><span class=\"n\">DataFrame<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">pd<\/span><span class=\"p\">.<\/span><span class=\"n\">DataFrame<\/span><span class=\"p\">:<\/span>\n        <span class=\"n\">operations<\/span> <span class=\"o\">=<\/span> <span class=\"p\">{<\/span>\n            <span class=\"sh\">\"<\/span><span class=\"s\">train<\/span><span class=\"sh\">\"<\/span><span class=\"p\">:<\/span> <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">_train<\/span><span class=\"p\">,<\/span>\n            <span class=\"sh\">\"<\/span><span class=\"s\">predict<\/span><span class=\"sh\">\"<\/span><span class=\"p\">:<\/span> <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">_predict<\/span>\n        <span class=\"p\">}<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">operations<\/span><span class=\"p\">.<\/span><span class=\"nf\">get<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">stage<\/span><span class=\"p\">,<\/span> <span class=\"k\">lambda<\/span> <span class=\"n\">d<\/span><span class=\"p\">:<\/span> <span class=\"n\">d<\/span><span class=\"p\">)(<\/span><span class=\"n\">data<\/span><span class=\"p\">)<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">_train<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">data<\/span><span class=\"p\">:<\/span> <span class=\"n\">pd<\/span><span class=\"p\">.<\/span><span class=\"n\">DataFrame<\/span><span class=\"p\">):<\/span>\n        <span class=\"n\">cleaned_data<\/span> <span class=\"o\">=<\/span> <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">transformer<\/span><span class=\"p\">.<\/span><span class=\"nf\">fit_transform<\/span><span class=\"p\">(<\/span><span class=\"n\">data<\/span><span class=\"p\">)<\/span>\n        <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">transformer<\/span> <span class=\"o\">=<\/span> <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">_transformer<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">cleaned_data<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">_predict<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">data<\/span><span class=\"p\">:<\/span> <span class=\"n\">pd<\/span><span class=\"p\">.<\/span><span class=\"n\">DataFrame<\/span><span class=\"p\">):<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">transformer<\/span><span class=\"p\">.<\/span><span class=\"nf\">transform<\/span><span class=\"p\">(<\/span><span class=\"n\">data<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Test we must, again young Padawan ...<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"n\">sklearn.compose<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">make_column_transformer<\/span>\n<span class=\"kn\">from<\/span> <span class=\"n\">sklearn.preprocessing<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">OneHotEncoder<\/span>\n<span class=\"kn\">from<\/span> <span class=\"n\">sklearn<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">set_config<\/span>\n\n<span class=\"kn\">from<\/span> <span class=\"n\">srp<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">config<\/span>\n\n<span class=\"nf\">set_config<\/span><span class=\"p\">(<\/span><span class=\"n\">transform_output<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">pandas<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n\n\n<span class=\"n\">transformer<\/span> <span class=\"o\">=<\/span> <span class=\"nf\">make_column_transformer<\/span><span class=\"p\">(<\/span>\n    <span class=\"p\">(<\/span>\n        <span class=\"nc\">OneHotEncoder<\/span><span class=\"p\">(<\/span><span class=\"n\">sparse_output<\/span><span class=\"o\">=<\/span><span class=\"bp\">False<\/span><span class=\"p\">),[<\/span>\n            <span class=\"n\">config<\/span><span class=\"p\">.<\/span><span class=\"n\">COLUMNS_TO_ONEHOTENCODE<\/span><span class=\"p\">,<\/span>\n        <span class=\"p\">],<\/span>\n    <span class=\"p\">),<\/span>\n    <span class=\"n\">verbose_feature_names_out<\/span><span class=\"o\">=<\/span><span class=\"bp\">False<\/span><span class=\"p\">,<\/span>\n    <span class=\"n\">remainder<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">passthrough<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span>\n<span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Oops, now test, we can \ud83e\udd79<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"n\">pathlib<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Path<\/span>\n<span class=\"kn\">import<\/span> <span class=\"n\">pytest<\/span>\n<span class=\"kn\">import<\/span> <span class=\"n\">pandas<\/span> <span class=\"k\">as<\/span> <span class=\"n\">pd<\/span>\n<span class=\"kn\">from<\/span> <span class=\"n\">tasks<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">TransformerTask<\/span> \n<span class=\"kn\">from<\/span> <span class=\"n\">preprocessors<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">transformer<\/span> \n\n\n\n<span class=\"c1\"># URI for the dataset\n<\/span><span class=\"n\">URI<\/span> <span class=\"o\">=<\/span> <span class=\"sh\">\"<\/span><span class=\"s\">https:\/\/raw.githubusercontent.com\/mwaskom\/seaborn-data\/master\/penguins.csv<\/span><span class=\"sh\">\"<\/span>\n\n<span class=\"c1\"># Model File path\n<\/span><span class=\"n\">MODEL_FILE<\/span> <span class=\"o\">=<\/span> <span class=\"nc\">Path<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">models\/transformer.pkl<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">cleanup<\/span><span class=\"p\">(<\/span><span class=\"nb\">file<\/span><span class=\"o\">=<\/span><span class=\"n\">MODEL_FILE<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">if<\/span> <span class=\"n\">MODEL_FILE<\/span><span class=\"p\">.<\/span><span class=\"nf\">exists<\/span><span class=\"p\">():<\/span>\n        <span class=\"n\">MODEL_FILE<\/span><span class=\"p\">.<\/span><span class=\"nf\">unlink<\/span><span class=\"p\">()<\/span> \n\n<span class=\"nd\">@pytest.fixture<\/span><span class=\"p\">(<\/span><span class=\"n\">autouse<\/span><span class=\"o\">=<\/span><span class=\"bp\">True<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">startup_and_teardown<\/span><span class=\"p\">():<\/span>\n    <span class=\"sh\">\"<\/span><span class=\"s\">Execute before and after a test run<\/span><span class=\"sh\">\"<\/span>\n    <span class=\"c1\"># Startup: remove model file\n<\/span>    <span class=\"nf\">cleanup<\/span><span class=\"p\">()<\/span>\n    <span class=\"k\">yield<\/span>\n    <span class=\"c1\"># Teardown : remove model file\n<\/span>    <span class=\"nf\">cleanup<\/span><span class=\"p\">()<\/span>\n\n<span class=\"nd\">@pytest.fixture<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">load_data<\/span><span class=\"p\">():<\/span>\n    <span class=\"k\">yield<\/span> <span class=\"n\">pd<\/span><span class=\"p\">.<\/span><span class=\"nf\">read_csv<\/span><span class=\"p\">(<\/span><span class=\"n\">URI<\/span><span class=\"p\">)<\/span>\n\n\n<span class=\"nd\">@pytest.fixture<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">transformer_task<\/span><span class=\"p\">():<\/span>\n    <span class=\"k\">yield<\/span> <span class=\"nc\">TransformerTask<\/span><span class=\"p\">(<\/span><span class=\"n\">transformer<\/span><span class=\"p\">,<\/span> <span class=\"n\">stage<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">train<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n\n\n<span class=\"k\">def<\/span> <span class=\"nf\">test_transformer_train<\/span><span class=\"p\">(<\/span><span class=\"n\">load_data<\/span><span class=\"p\">,<\/span> <span class=\"n\">transformer_task<\/span><span class=\"p\">):<\/span>\n    <span class=\"sh\">\"<\/span><span class=\"s\">Test the training stage<\/span><span class=\"sh\">\"<\/span>\n\n    <span class=\"n\">result<\/span> <span class=\"o\">=<\/span> <span class=\"n\">transformer_task<\/span><span class=\"p\">.<\/span><span class=\"nf\">run<\/span><span class=\"p\">(<\/span><span class=\"n\">load_data<\/span><span class=\"p\">)<\/span> \n    <span class=\"k\">assert<\/span> <span class=\"ow\">not<\/span> <span class=\"n\">result<\/span><span class=\"p\">.<\/span><span class=\"n\">empty<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"n\">MODEL_FILE<\/span><span class=\"p\">.<\/span><span class=\"nf\">exists<\/span><span class=\"p\">()<\/span>\n\n\n<span class=\"k\">def<\/span> <span class=\"nf\">test_transformer_predict<\/span><span class=\"p\">(<\/span><span class=\"n\">transformer_task<\/span><span class=\"p\">,<\/span> <span class=\"n\">load_data<\/span><span class=\"p\">):<\/span>\n    <span class=\"sh\">\"<\/span><span class=\"s\">Test the prediction stage<\/span><span class=\"sh\">\"<\/span>\n    <span class=\"c1\"># First, run the training stage to ensure the transformer is fitted and saved\n<\/span>    <span class=\"n\">transformer_task<\/span><span class=\"p\">.<\/span><span class=\"nf\">run<\/span><span class=\"p\">(<\/span><span class=\"n\">load_data<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">transformer_task<\/span><span class=\"p\">.<\/span><span class=\"n\">stage<\/span> <span class=\"o\">=<\/span> <span class=\"sh\">\"<\/span><span class=\"s\">predict<\/span><span class=\"sh\">\"<\/span>\n\n    <span class=\"c1\"># Testing prediction stage\n<\/span>    <span class=\"k\">assert<\/span> <span class=\"n\">MODEL_FILE<\/span><span class=\"p\">.<\/span><span class=\"nf\">exists<\/span><span class=\"p\">()<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"n\">transformer_task<\/span><span class=\"p\">.<\/span><span class=\"n\">stage<\/span> <span class=\"o\">==<\/span> <span class=\"sh\">\"<\/span><span class=\"s\">predict<\/span><span class=\"sh\">\"<\/span> \n\n    <span class=\"n\">result<\/span> <span class=\"o\">=<\/span> <span class=\"n\">transformer_task<\/span><span class=\"p\">.<\/span><span class=\"nf\">run<\/span><span class=\"p\">(<\/span><span class=\"n\">load_data<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">assert<\/span> <span class=\"ow\">not<\/span> <span class=\"n\">result<\/span><span class=\"p\">.<\/span><span class=\"n\">empty<\/span>\n\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<h3>\n  \n  \n  Reflect and Contemplate:\n<\/h3>\n\n<p>Wield this tool with a balance to avert the shadows of accidental recursion or the echoes of unwanted side effects.<\/p>\n\n\n\n\n<h2>\n  \n  \n  3. <strong>Decorators: The Enchanters<\/strong>\n<\/h2>\n\n<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fis5votm48pq4ag8qltxc.jpeg\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fis5votm48pq4ag8qltxc.jpeg\" alt=\"russian doll\" width=\"800\" height=\"800\"><\/a><\/p>\n\n<p>Summon the might of Decorators, enchanters that weave their spells to transform the behaviour of your functions or methods, bestowing upon them new realms of possibilities.<\/p>\n\n<p>Create a decorator, we must. Allow it will, to profile bottlenecks within our functions\/methods, hmmm.<\/p>\n\n<p>Start simple, we shall. Expand later, we will, young Padawan.<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"n\">typing<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Callable<\/span>\n<span class=\"kn\">import<\/span> <span class=\"n\">numpy<\/span> <span class=\"k\">as<\/span> <span class=\"n\">np<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">regression<\/span><span class=\"p\">(<\/span><span class=\"n\">func<\/span><span class=\"p\">:<\/span> <span class=\"n\">Callable<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">Callable<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">func<\/span><span class=\"p\">.<\/span><span class=\"n\">kind<\/span> <span class=\"o\">=<\/span> <span class=\"sh\">\"<\/span><span class=\"s\">regression<\/span><span class=\"sh\">\"<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">func<\/span>\n\n<span class=\"nd\">@regression<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">mse<\/span><span class=\"p\">(<\/span><span class=\"n\">y_true<\/span><span class=\"p\">,<\/span> <span class=\"n\">y_pred<\/span><span class=\"p\">):<\/span>\n    <span class=\"nf\">return <\/span><span class=\"p\">((<\/span><span class=\"n\">y_true<\/span> <span class=\"o\">-<\/span> <span class=\"n\">y_pred<\/span><span class=\"p\">)<\/span><span class=\"o\">**<\/span><span class=\"mi\">2<\/span><span class=\"p\">).<\/span><span class=\"nf\">mean<\/span><span class=\"p\">()<\/span>\n\n<span class=\"nf\">print<\/span><span class=\"p\">((<\/span><span class=\"n\">mse<\/span><span class=\"p\">.<\/span><span class=\"n\">__name__<\/span><span class=\"p\">,<\/span> <span class=\"n\">mse<\/span><span class=\"p\">.<\/span><span class=\"n\">kind<\/span><span class=\"p\">))<\/span>\n<span class=\"c1\"># ('mse', 'regression')\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Let's level up, doing a hyperspace jump<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"n\">cProfile<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Profile<\/span>\n<span class=\"kn\">from<\/span> <span class=\"n\">functools<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">wraps<\/span>\n<span class=\"kn\">import<\/span> <span class=\"n\">pstats<\/span>\n\n<span class=\"k\">def<\/span> <span class=\"nf\">profiler<\/span><span class=\"p\">(<\/span><span class=\"n\">func<\/span><span class=\"p\">):<\/span>\n    <span class=\"nd\">@wraps<\/span><span class=\"p\">(<\/span><span class=\"n\">func<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">wrapper<\/span><span class=\"p\">(<\/span><span class=\"o\">*<\/span><span class=\"n\">args<\/span><span class=\"p\">,<\/span> <span class=\"o\">**<\/span><span class=\"n\">kwargs<\/span><span class=\"p\">):<\/span>\n        <span class=\"k\">with<\/span> <span class=\"nc\">Profile<\/span><span class=\"p\">()<\/span> <span class=\"k\">as<\/span> <span class=\"n\">p<\/span><span class=\"p\">:<\/span>\n            <span class=\"c1\"># run the function and collect profile data\n<\/span>            <span class=\"n\">results<\/span> <span class=\"o\">=<\/span> <span class=\"n\">p<\/span><span class=\"p\">.<\/span><span class=\"nf\">runcall<\/span><span class=\"p\">(<\/span><span class=\"n\">func<\/span><span class=\"p\">,<\/span> <span class=\"o\">*<\/span><span class=\"n\">args<\/span><span class=\"p\">,<\/span> <span class=\"o\">**<\/span><span class=\"n\">kwargs<\/span><span class=\"p\">)<\/span>\n\n            <span class=\"c1\"># sort =&gt; name, time, file https:\/\/docs.python.org\/3\/library\/profile.html\n<\/span>            <span class=\"n\">ps<\/span> <span class=\"o\">=<\/span> <span class=\"n\">pstats<\/span><span class=\"p\">.<\/span><span class=\"nc\">Stats<\/span><span class=\"p\">(<\/span><span class=\"n\">p<\/span><span class=\"p\">,<\/span> <span class=\"n\">stream<\/span><span class=\"o\">=<\/span><span class=\"bp\">None<\/span><span class=\"p\">).<\/span><span class=\"nf\">sort_stats<\/span><span class=\"p\">(<\/span><span class=\"sh\">'<\/span><span class=\"s\">cumulative<\/span><span class=\"sh\">'<\/span><span class=\"p\">)<\/span> \n            <span class=\"n\">ps<\/span><span class=\"p\">.<\/span><span class=\"nf\">print_stats<\/span><span class=\"p\">(<\/span><span class=\"mi\">5<\/span><span class=\"p\">)<\/span> <span class=\"c1\"># restriction of print\n<\/span>        <span class=\"k\">return<\/span> <span class=\"n\">results<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">wrapper<\/span>\n\n\n<span class=\"nd\">@profiler<\/span>\n<span class=\"nd\">@regression<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">mse<\/span><span class=\"p\">(<\/span><span class=\"n\">y_true<\/span><span class=\"p\">,<\/span> <span class=\"n\">y_pred<\/span><span class=\"p\">):<\/span>\n    <span class=\"nf\">return <\/span><span class=\"p\">((<\/span><span class=\"n\">y_true<\/span> <span class=\"o\">-<\/span> <span class=\"n\">y_pred<\/span><span class=\"p\">)<\/span><span class=\"o\">**<\/span><span class=\"mi\">2<\/span><span class=\"p\">).<\/span><span class=\"nf\">mean<\/span><span class=\"p\">()<\/span>\n\n<span class=\"k\">if<\/span> <span class=\"n\">__name__<\/span> <span class=\"o\">==<\/span> <span class=\"sh\">\"<\/span><span class=\"s\">__main__<\/span><span class=\"sh\">\"<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">y_true<\/span> <span class=\"o\">=<\/span> <span class=\"n\">np<\/span><span class=\"p\">.<\/span><span class=\"nf\">array<\/span><span class=\"p\">([<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"mi\">3<\/span><span class=\"p\">,<\/span> <span class=\"mi\">4<\/span><span class=\"p\">,<\/span> <span class=\"mi\">5<\/span><span class=\"p\">])<\/span>\n    <span class=\"n\">y_pred<\/span> <span class=\"o\">=<\/span> <span class=\"n\">np<\/span><span class=\"p\">.<\/span><span class=\"nf\">array<\/span><span class=\"p\">([<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">3<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"mi\">3<\/span><span class=\"p\">,<\/span> <span class=\"mi\">5<\/span><span class=\"p\">])<\/span>\n    <span class=\"nf\">print<\/span><span class=\"p\">((<\/span><span class=\"n\">mse<\/span><span class=\"p\">.<\/span><span class=\"n\">__name__<\/span><span class=\"p\">,<\/span> <span class=\"n\">mse<\/span><span class=\"p\">.<\/span><span class=\"n\">kind<\/span><span class=\"p\">,<\/span> <span class=\"n\">results<\/span><span class=\"p\">))<\/span>\n    <span class=\"c1\"># some profile logs\n<\/span>    <span class=\"c1\"># ('mse', 'regression', 0.6)\n<\/span>\n\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Another way of writing the same decorator, using <code>class<\/code>, would look like this:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">class<\/span> <span class=\"nc\">profiler<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">__init__<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">func<\/span><span class=\"p\">):<\/span>\n        <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">func<\/span> <span class=\"o\">=<\/span> <span class=\"n\">func<\/span>\n        <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">__name__<\/span> <span class=\"o\">=<\/span> <span class=\"n\">func<\/span><span class=\"p\">.<\/span><span class=\"n\">__name__<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">__call__<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">,<\/span> <span class=\"o\">*<\/span><span class=\"n\">args<\/span><span class=\"p\">,<\/span> <span class=\"o\">**<\/span><span class=\"n\">kwargs<\/span><span class=\"p\">):<\/span>\n        <span class=\"k\">with<\/span> <span class=\"nc\">Profile<\/span><span class=\"p\">()<\/span> <span class=\"k\">as<\/span> <span class=\"n\">p<\/span><span class=\"p\">:<\/span>\n            <span class=\"n\">results<\/span> <span class=\"o\">=<\/span> <span class=\"n\">p<\/span><span class=\"p\">.<\/span><span class=\"nf\">runcall<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">func<\/span><span class=\"p\">,<\/span> <span class=\"o\">*<\/span><span class=\"n\">args<\/span><span class=\"p\">,<\/span> <span class=\"o\">**<\/span><span class=\"n\">kwargs<\/span><span class=\"p\">)<\/span>\n\n            <span class=\"n\">ps<\/span> <span class=\"o\">=<\/span> <span class=\"n\">pstats<\/span><span class=\"p\">.<\/span><span class=\"nc\">Stats<\/span><span class=\"p\">(<\/span><span class=\"n\">p<\/span><span class=\"p\">,<\/span> <span class=\"n\">stream<\/span><span class=\"o\">=<\/span><span class=\"bp\">None<\/span><span class=\"p\">).<\/span><span class=\"nf\">sort_stats<\/span><span class=\"p\">(<\/span><span class=\"sh\">'<\/span><span class=\"s\">cumulative<\/span><span class=\"sh\">'<\/span><span class=\"p\">)<\/span>\n            <span class=\"n\">ps<\/span><span class=\"p\">.<\/span><span class=\"nf\">print_stats<\/span><span class=\"p\">(<\/span><span class=\"mi\">5<\/span><span class=\"p\">)<\/span> \n\n        <span class=\"k\">return<\/span> <span class=\"n\">results<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Hark! A disturbance in the Force. Hardcoded sorting and stats printing lines. A desire to inject them, I sense. Embark on this quest, we must. Bring balance to the code, we shall. Doing that, let us.<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"c1\"># functional way\n<\/span><span class=\"k\">def<\/span> <span class=\"nf\">profiler<\/span><span class=\"p\">(<\/span><span class=\"n\">sort_by<\/span><span class=\"o\">=<\/span><span class=\"sh\">'<\/span><span class=\"s\">cumulative<\/span><span class=\"sh\">'<\/span><span class=\"p\">,<\/span> <span class=\"n\">restriction<\/span><span class=\"o\">=<\/span><span class=\"mi\">5<\/span><span class=\"p\">,<\/span> <span class=\"n\">streams<\/span><span class=\"o\">=<\/span><span class=\"bp\">None<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">inner_function<\/span><span class=\"p\">(<\/span><span class=\"n\">func<\/span><span class=\"p\">):<\/span>\n        <span class=\"nd\">@wraps<\/span><span class=\"p\">(<\/span><span class=\"n\">func<\/span><span class=\"p\">)<\/span>\n        <span class=\"k\">def<\/span> <span class=\"nf\">wrapper<\/span><span class=\"p\">(<\/span><span class=\"o\">*<\/span><span class=\"n\">args<\/span><span class=\"p\">,<\/span> <span class=\"o\">**<\/span><span class=\"n\">kwargs<\/span><span class=\"p\">):<\/span>\n            <span class=\"k\">with<\/span> <span class=\"nc\">Profile<\/span><span class=\"p\">()<\/span> <span class=\"k\">as<\/span> <span class=\"n\">p<\/span><span class=\"p\">:<\/span>\n                <span class=\"n\">result<\/span> <span class=\"o\">=<\/span> <span class=\"n\">p<\/span><span class=\"p\">.<\/span><span class=\"nf\">runcall<\/span><span class=\"p\">(<\/span><span class=\"n\">func<\/span><span class=\"p\">,<\/span> <span class=\"o\">*<\/span><span class=\"n\">args<\/span><span class=\"p\">,<\/span> <span class=\"o\">**<\/span><span class=\"n\">kwargs<\/span><span class=\"p\">)<\/span>  \n                <span class=\"n\">ps<\/span> <span class=\"o\">=<\/span> <span class=\"n\">pstats<\/span><span class=\"p\">.<\/span><span class=\"nc\">Stats<\/span><span class=\"p\">(<\/span><span class=\"n\">p<\/span><span class=\"p\">,<\/span> <span class=\"n\">stream<\/span><span class=\"o\">=<\/span><span class=\"n\">streams<\/span><span class=\"p\">).<\/span><span class=\"nf\">sort_stats<\/span><span class=\"p\">(<\/span><span class=\"n\">sort_by<\/span><span class=\"p\">)<\/span>  \n                <span class=\"n\">ps<\/span><span class=\"p\">.<\/span><span class=\"nf\">print_stats<\/span><span class=\"p\">(<\/span><span class=\"n\">restriction<\/span><span class=\"p\">)<\/span>  \n            <span class=\"k\">return<\/span> <span class=\"n\">result<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">wrapper<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">inner_function<\/span>\n\n<span class=\"c1\"># class way\n<\/span><span class=\"k\">class<\/span> <span class=\"nc\">profiler<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">__init__<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">sort_by<\/span><span class=\"o\">=<\/span><span class=\"sh\">'<\/span><span class=\"s\">cumulative<\/span><span class=\"sh\">'<\/span><span class=\"p\">,<\/span> <span class=\"n\">restriction<\/span><span class=\"o\">=<\/span><span class=\"mi\">5<\/span><span class=\"p\">,<\/span> <span class=\"n\">streams<\/span><span class=\"o\">=<\/span><span class=\"bp\">None<\/span><span class=\"p\">):<\/span>\n        <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">sort_by<\/span> <span class=\"o\">=<\/span> <span class=\"n\">sort_by<\/span>\n        <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">restriction<\/span> <span class=\"o\">=<\/span> <span class=\"n\">restriction<\/span>\n        <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">streams<\/span> <span class=\"o\">=<\/span> <span class=\"n\">streams<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">__call__<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">func<\/span><span class=\"p\">):<\/span>\n        <span class=\"k\">def<\/span> <span class=\"nf\">wrapper<\/span><span class=\"p\">(<\/span><span class=\"o\">*<\/span><span class=\"n\">args<\/span><span class=\"p\">,<\/span> <span class=\"o\">**<\/span><span class=\"n\">kwargs<\/span><span class=\"p\">):<\/span>\n            <span class=\"k\">with<\/span> <span class=\"nc\">Profile<\/span><span class=\"p\">()<\/span> <span class=\"k\">as<\/span> <span class=\"n\">p<\/span><span class=\"p\">:<\/span>\n                <span class=\"n\">result<\/span> <span class=\"o\">=<\/span> <span class=\"n\">p<\/span><span class=\"p\">.<\/span><span class=\"nf\">runcall<\/span><span class=\"p\">(<\/span><span class=\"n\">func<\/span><span class=\"p\">,<\/span> <span class=\"o\">*<\/span><span class=\"n\">args<\/span><span class=\"p\">,<\/span> <span class=\"o\">**<\/span><span class=\"n\">kwargs<\/span><span class=\"p\">)<\/span>  \n                <span class=\"n\">ps<\/span> <span class=\"o\">=<\/span> <span class=\"n\">pstats<\/span><span class=\"p\">.<\/span><span class=\"nc\">Stats<\/span><span class=\"p\">(<\/span><span class=\"n\">p<\/span><span class=\"p\">,<\/span> <span class=\"n\">stream<\/span><span class=\"o\">=<\/span><span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">streams<\/span><span class=\"p\">).<\/span><span class=\"nf\">sort_stats<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">sort_by<\/span><span class=\"p\">)<\/span>  \n                <span class=\"n\">ps<\/span><span class=\"p\">.<\/span><span class=\"nf\">print_stats<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">restriction<\/span><span class=\"p\">)<\/span>  \n            <span class=\"k\">return<\/span> <span class=\"n\">result<\/span>\n        <span class=\"n\">wrapper<\/span><span class=\"p\">.<\/span><span class=\"n\">__name__<\/span> <span class=\"o\">=<\/span> <span class=\"n\">func<\/span><span class=\"p\">.<\/span><span class=\"n\">__name__<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">wrapper<\/span>\n\n<span class=\"nd\">@regression<\/span>\n<span class=\"nd\">@profiler<\/span><span class=\"p\">(<\/span><span class=\"n\">sort_by<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">time<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">restriction<\/span><span class=\"o\">=<\/span><span class=\"mi\">3<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">mse<\/span><span class=\"p\">(<\/span><span class=\"n\">y_true<\/span><span class=\"p\">,<\/span> <span class=\"n\">y_pred<\/span><span class=\"p\">):<\/span>\n    <span class=\"nf\">return <\/span><span class=\"p\">((<\/span><span class=\"n\">y_true<\/span> <span class=\"o\">-<\/span> <span class=\"n\">y_pred<\/span><span class=\"p\">)<\/span><span class=\"o\">**<\/span><span class=\"mi\">2<\/span><span class=\"p\">).<\/span><span class=\"nf\">mean<\/span><span class=\"p\">()<\/span>\n\n<span class=\"k\">if<\/span> <span class=\"n\">__name__<\/span> <span class=\"o\">==<\/span> <span class=\"sh\">\"<\/span><span class=\"s\">__main__<\/span><span class=\"sh\">\"<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">y_true<\/span> <span class=\"o\">=<\/span> <span class=\"n\">np<\/span><span class=\"p\">.<\/span><span class=\"nf\">array<\/span><span class=\"p\">([<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"mi\">3<\/span><span class=\"p\">,<\/span> <span class=\"mi\">4<\/span><span class=\"p\">,<\/span> <span class=\"mi\">5<\/span><span class=\"p\">])<\/span>\n    <span class=\"n\">y_pred<\/span> <span class=\"o\">=<\/span> <span class=\"n\">np<\/span><span class=\"p\">.<\/span><span class=\"nf\">array<\/span><span class=\"p\">([<\/span><span class=\"mi\">1<\/span><span class=\"p\">,<\/span> <span class=\"mi\">3<\/span><span class=\"p\">,<\/span> <span class=\"mi\">2<\/span><span class=\"p\">,<\/span> <span class=\"mi\">3<\/span><span class=\"p\">,<\/span> <span class=\"mi\">5<\/span><span class=\"p\">])<\/span>\n    <span class=\"n\">results<\/span> <span class=\"o\">=<\/span> <span class=\"nf\">mse<\/span><span class=\"p\">(<\/span><span class=\"n\">y_true<\/span><span class=\"p\">,<\/span> <span class=\"n\">y_pred<\/span><span class=\"p\">)<\/span>\n\n    <span class=\"nf\">print<\/span><span class=\"p\">((<\/span><span class=\"n\">mse<\/span><span class=\"p\">.<\/span><span class=\"n\">__name__<\/span><span class=\"p\">,<\/span> <span class=\"n\">mse<\/span><span class=\"p\">.<\/span><span class=\"n\">kind<\/span><span class=\"p\">,<\/span> <span class=\"n\">results<\/span><span class=\"p\">))<\/span>\n    <span class=\"c1\"># some profile logs are sorted by time and show 3 lines\n<\/span>    <span class=\"c1\"># ('mse', 'regression', 0.6)\n<\/span><\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Ah, a wise query emerges amidst the stars! Speak of adorning methods within a cosmic class, do you? Fear not, for a class decorator we shall craft. Bestow our decorator upon the class methods, it shall. In unity, traverse the celestial pathways of Python, we will!<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"k\">def<\/span> <span class=\"nf\">profilerx<\/span><span class=\"p\">(<\/span><span class=\"n\">decorate<\/span><span class=\"o\">=<\/span><span class=\"bp\">None<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">if<\/span> <span class=\"n\">decorate<\/span> <span class=\"ow\">is<\/span> <span class=\"bp\">None<\/span><span class=\"p\">:<\/span>\n        <span class=\"n\">decorate<\/span> <span class=\"o\">=<\/span> <span class=\"k\">lambda<\/span> <span class=\"n\">d<\/span><span class=\"p\">:<\/span> <span class=\"n\">d<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">wrapper<\/span><span class=\"p\">(<\/span><span class=\"n\">cls<\/span><span class=\"p\">):<\/span>\n        <span class=\"n\">name_functions<\/span> <span class=\"o\">=<\/span> <span class=\"p\">{<\/span><span class=\"n\">name<\/span><span class=\"p\">:<\/span><span class=\"n\">func<\/span> <span class=\"k\">for<\/span> <span class=\"n\">name<\/span><span class=\"p\">,<\/span> <span class=\"n\">func<\/span> <span class=\"ow\">in<\/span> <span class=\"nf\">vars<\/span><span class=\"p\">(<\/span><span class=\"n\">cls<\/span><span class=\"p\">).<\/span><span class=\"nf\">items<\/span><span class=\"p\">()<\/span> <span class=\"k\">if<\/span> <span class=\"ow\">not<\/span> <span class=\"n\">name<\/span><span class=\"p\">.<\/span><span class=\"nf\">startswith<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">__<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)}<\/span>\n\n        <span class=\"k\">for<\/span> <span class=\"n\">name<\/span><span class=\"p\">,<\/span> <span class=\"n\">func<\/span> <span class=\"ow\">in<\/span> <span class=\"nf\">vars<\/span><span class=\"p\">(<\/span><span class=\"n\">cls<\/span><span class=\"p\">).<\/span><span class=\"nf\">items<\/span><span class=\"p\">():<\/span>\n            <span class=\"k\">if<\/span> <span class=\"nf\">callable<\/span><span class=\"p\">(<\/span><span class=\"n\">func<\/span><span class=\"p\">):<\/span>\n                <span class=\"nf\">setattr<\/span><span class=\"p\">(<\/span><span class=\"n\">cls<\/span><span class=\"p\">,<\/span> <span class=\"n\">name<\/span><span class=\"p\">,<\/span> <span class=\"nf\">decorate<\/span><span class=\"p\">(<\/span><span class=\"n\">func<\/span><span class=\"p\">))<\/span>\n\n        <span class=\"k\">return<\/span> <span class=\"n\">cls<\/span>\n        <span class=\"n\">wrapper<\/span><span class=\"p\">.<\/span><span class=\"n\">__name__<\/span> <span class=\"o\">=<\/span> <span class=\"n\">cls<\/span><span class=\"p\">.<\/span><span class=\"n\">__name__<\/span>\n\n    <span class=\"k\">return<\/span> <span class=\"n\">wrapper<\/span>\n\n<span class=\"c1\"># Decorate our class, we can \u263a\ufe0f\n<\/span>\n<span class=\"nd\">@profilerx<\/span><span class=\"p\">(<\/span><span class=\"n\">decorate<\/span><span class=\"o\">=<\/span><span class=\"nf\">profiler<\/span><span class=\"p\">(<\/span><span class=\"n\">sort_by<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">cumulative<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">restriction<\/span><span class=\"o\">=<\/span><span class=\"mi\">5<\/span><span class=\"p\">))<\/span>\n<span class=\"k\">class<\/span> <span class=\"nc\">Circle<\/span><span class=\"p\">:<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">__init__<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">radius<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span><span class=\"o\">|<\/span><span class=\"nb\">float<\/span><span class=\"p\">):<\/span>\n        <span class=\"n\">self<\/span><span class=\"p\">.<\/span><span class=\"n\">_radius<\/span> <span class=\"o\">=<\/span> <span class=\"nf\">radius_validator<\/span><span class=\"p\">(<\/span><span class=\"n\">radius<\/span><span class=\"p\">)<\/span>\n   <span class=\"bp\">...<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<h3>\n  \n  \n  Reflect and Contemplate:\n<\/h3>\n\n<p>Embrace this magic with mindfulness, for within its allure, lies the labyrinth of complexity, whispering tales of code entangled in its own enchantment.<\/p>\n\n\n\n\n<h2>\n  \n  \n  4. <strong>Metaprogramming: The Arcane Arts<\/strong>\n<\/h2>\n\n<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fqofx2qxiamjmor1ntsfs.jpeg\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fqofx2qxiamjmor1ntsfs.jpeg\" alt=\"meta\" width=\"800\" height=\"800\"><\/a><\/p>\n\n<p>As we traverse the realms of decorators, we find ourselves at the gateway to Metaprogramming, a universe where code breathes life into more code. Imagine a scenario where the reins of our crafted code slip out of our grasp, handed over to realms and teams beyond our dominion. In this abyss, where control eludes our touch, how do we ensure our shields are not tampered with? Behold the luminescence of metaprogramming, a beacon in the shadows of constraint and order.<\/p>\n\n<p>Envision a creation of our own, a code entity named <code>RejectPrint<\/code>, destined to embark upon a voyage to distant teams and uses. As it journeys beyond our realm, we yearn to engrave a solemn vow upon its essence \u2013 the exile of the <code>print<\/code> command, guarding the sanctity of silence amidst its ventures in the unknown.<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"c1\"># our modulex.py\n<\/span><span class=\"kn\">import<\/span> <span class=\"n\">logging<\/span>\n<span class=\"kn\">import<\/span> <span class=\"n\">inspect<\/span>\n<span class=\"kn\">from<\/span> <span class=\"n\">typing<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Callable<\/span>\n\n\n<span class=\"k\">def<\/span> <span class=\"nf\">check_bad_usage<\/span><span class=\"p\">(<\/span><span class=\"n\">name<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span><span class=\"p\">,<\/span> <span class=\"n\">obj<\/span><span class=\"p\">:<\/span> <span class=\"n\">Callable<\/span><span class=\"p\">)<\/span><span class=\"o\">-&gt;<\/span> <span class=\"nb\">bool<\/span><span class=\"p\">:<\/span>\n    <span class=\"sh\">\"\"\"<\/span><span class=\"s\">\n    checks if an object has a function with certain name\n    <\/span><span class=\"sh\">\"\"\"<\/span>\n\n    <span class=\"n\">func_names<\/span> <span class=\"o\">=<\/span> <span class=\"n\">obj<\/span><span class=\"p\">.<\/span><span class=\"n\">__code__<\/span><span class=\"p\">.<\/span><span class=\"n\">co_names<\/span>\n    <span class=\"n\">is_bad_usage<\/span> <span class=\"o\">=<\/span> <span class=\"bp\">False<\/span>\n\n    <span class=\"k\">if<\/span> <span class=\"n\">name<\/span> <span class=\"ow\">not<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">func_names<\/span><span class=\"p\">:<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">is_bad_usage<\/span>\n\n    <span class=\"n\">is_bad_usage<\/span> <span class=\"o\">=<\/span> <span class=\"bp\">True<\/span>\n    <span class=\"k\">for<\/span> <span class=\"n\">func_name<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">func_names<\/span><span class=\"p\">:<\/span>\n\n\n        <span class=\"k\">if<\/span> <span class=\"n\">name<\/span> <span class=\"o\">==<\/span> <span class=\"n\">func_name<\/span><span class=\"p\">:<\/span>\n            <span class=\"c1\"># echo the code with bad usage\n<\/span>            <span class=\"n\">logging<\/span><span class=\"p\">.<\/span><span class=\"nf\">error<\/span><span class=\"p\">(<\/span><span class=\"n\">inspect<\/span><span class=\"p\">.<\/span><span class=\"nf\">getsource<\/span><span class=\"p\">(<\/span><span class=\"n\">obj<\/span><span class=\"p\">.<\/span><span class=\"n\">__code__<\/span><span class=\"p\">))<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">is_bad_usage<\/span>\n\n\n\n<span class=\"k\">class<\/span> <span class=\"nc\">RejectPrintMeta<\/span><span class=\"p\">(<\/span><span class=\"nb\">type<\/span><span class=\"p\">):<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">__new__<\/span><span class=\"p\">(<\/span><span class=\"n\">cls<\/span><span class=\"p\">,<\/span> <span class=\"n\">name<\/span><span class=\"p\">,<\/span> <span class=\"n\">bases<\/span><span class=\"p\">,<\/span> <span class=\"n\">body<\/span><span class=\"p\">):<\/span>\n\n        <span class=\"n\">callable_obj<\/span> <span class=\"o\">=<\/span> <span class=\"p\">(<\/span><span class=\"n\">v<\/span> <span class=\"k\">for<\/span> <span class=\"n\">v<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">body<\/span><span class=\"p\">.<\/span><span class=\"nf\">values<\/span><span class=\"p\">()<\/span> <span class=\"k\">if<\/span> <span class=\"nf\">callable<\/span><span class=\"p\">(<\/span><span class=\"n\">v<\/span><span class=\"p\">))<\/span>\n\n        <span class=\"k\">for<\/span> <span class=\"n\">obj<\/span> <span class=\"ow\">in<\/span> <span class=\"n\">callable_obj<\/span><span class=\"p\">:<\/span>\n            <span class=\"n\">is_bad_usage<\/span> <span class=\"o\">=<\/span> <span class=\"nf\">check_bad_usage<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">print<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">obj<\/span><span class=\"p\">)<\/span>\n            <span class=\"k\">if<\/span> <span class=\"n\">is_bad_usage<\/span><span class=\"p\">:<\/span>\n                <span class=\"k\">raise<\/span> <span class=\"nc\">UserWarning<\/span><span class=\"p\">(<\/span><span class=\"sa\">f<\/span><span class=\"sh\">\"<\/span><span class=\"s\">No way, `<\/span><span class=\"si\">{<\/span><span class=\"n\">obj<\/span><span class=\"p\">.<\/span><span class=\"n\">__name__<\/span><span class=\"si\">}<\/span><span class=\"s\">` contains `print` function!<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n\n        <span class=\"k\">return<\/span> <span class=\"nf\">super<\/span><span class=\"p\">().<\/span><span class=\"nf\">__new__<\/span><span class=\"p\">(<\/span><span class=\"n\">cls<\/span><span class=\"p\">,<\/span> <span class=\"n\">name<\/span><span class=\"p\">,<\/span> <span class=\"n\">bases<\/span><span class=\"p\">,<\/span> <span class=\"n\">body<\/span><span class=\"p\">)<\/span>\n\n\n\n<span class=\"k\">class<\/span> <span class=\"nc\">RejectPrint<\/span><span class=\"p\">(<\/span><span class=\"n\">metaclass<\/span><span class=\"o\">=<\/span><span class=\"n\">RejectPrintMeta<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">pass<\/span>\n\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>As users employ our package, a silent guardian emerges at the birth of their classes. The use of <code>print<\/code> is gently yet firmly barred, ensuring unbroken harmony in the world of our tool\u2019s operation.<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"c1\"># example.py\n# from modulex import RejectPrint\n<\/span>\n<span class=\"k\">class<\/span> <span class=\"nc\">UserPrint<\/span><span class=\"p\">(<\/span><span class=\"n\">RejectPrint<\/span><span class=\"p\">):<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">no_print_used<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">):<\/span>\n        <span class=\"k\">pass<\/span>\n\n    <span class=\"k\">def<\/span> <span class=\"nf\">print_used<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">):<\/span>\n        <span class=\"nf\">print<\/span><span class=\"p\">(<\/span><span class=\"sh\">'<\/span><span class=\"s\">using print :-o<\/span><span class=\"sh\">'<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F95kobdm1aui37cyo2s78.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F95kobdm1aui37cyo2s78.png\" alt=\"beautiful error message\" width=\"800\" height=\"403\"><\/a><\/p>\n\n<h3>\n  \n  \n  Reflect and Contemplate:\n<\/h3>\n\n<p>Tread with care in the realm of metaprogramming, for the powers it unleashes, while potent, whisper the tales of complexity and enigma.<\/p>\n\n\n\n\n<h2>\n  \n  \n  Conclusion: Beginning of the Start\n<\/h2>\n\n<p>I incorporated these techniques in my repertoire, as a young Padawan. I watched the doors to Python mastery unfold before me. It is my hope that you continue to explore, learn, and adapt, and let the Force guide your own enlightening journey.<\/p>\n\n<p>And so, the odyssey through the realms of advanced Python concludes yet begins. May your code flourish in elegance and efficiency, nurtured by the seeds of wisdom sown today. <\/p>\n\n<p><code>Until<\/code> then, may the force keep you coding.<\/p>\n\n<p><em>Note:<\/em> Generators, context managers, and plugins are additional Jedi tools that I've had to set aside for this post to avoid making it overly lengthy. If your curiosity is piqued and you desire to explore these realms further, do let me know. A sequel awaits the beckoning of your interest, ready to guide you further on this path to Python mastery.<\/p>\n\n","category":["python","programming","tutorial"]},{"title":"Unexpected Expected Thriller: A Tale of Coding Curiosity","pubDate":"Sun, 10 Sep 2023 16:23:45 +0000","link":"https:\/\/dev.to\/proteusiq\/unexpected-expected-thriller-a-tale-of-coding-curiosity-f6e","guid":"https:\/\/dev.to\/proteusiq\/unexpected-expected-thriller-a-tale-of-coding-curiosity-f6e","description":"<p>Do you remember the last time you dived deep into a coding session, losing track of time, just for the sheer fun of it? It's not about deadlines, efficiency, or even perfection. It's the allure of experimentation, breaking, and rebuilding, all while discovering new paths. <\/p>\n\n<p>Today, I'm going to take you on a thrilling coding adventure inspired by a LinkedIn code snippet, where I tangled with <a href=\"https:\/\/fastapi.tiangolo.com\/\" rel=\"noopener noreferrer\">FastAPI<\/a>, <a href=\"https:\/\/riverml.xyz\/\" rel=\"noopener noreferrer\">River<\/a>, <a href=\"https:\/\/github.com\/gorakhargosh\/watchdog\" rel=\"noopener noreferrer\">Watchdog<\/a>, and <a href=\"https:\/\/github.com\/jd\/tenacity\" rel=\"noopener noreferrer\">Tenacity<\/a>. Ready? Buckle up!<\/p>\n\n<p><strong>\u26a0\ufe0f <em>Disclaimer:<\/em><\/strong><br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight shell\"><code>For the adventurous spirit: Don<span class=\"se\">\\'<\/span>t attempt this at home <span class=\"se\">\\`<\/span><span class=\"o\">{<\/span>home <span class=\"o\">==<\/span> production<span class=\"o\">}<\/span><span class=\"se\">\\`<\/span><span class=\"nb\">.<\/span> For clarity, the code has been condensed into three files.\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<h3>\n  \n  \n  <strong>Once Upon a FastAPI...<\/strong>\n<\/h3>\n\n<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fx3cybqbllwmz5cnoypy0.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fx3cybqbllwmz5cnoypy0.png\" alt=\"LinkedInPost\" width=\"800\" height=\"609\"><\/a><\/p>\n\n<p>It all started when my eyes caught a captivating MantisNLP's LinkedIn post. The post showcased <code>FastAPI<\/code> code snippets intended for educational purposes. What intrigued me was the swift model reloading. In the snippet, which carried out an ML prediction, the authors first loaded the model and then executed the prediction.<\/p>\n\n<p>This stirred a question in me: Could I modify an API's model state based on an event change? If achievable, could we serve a machine learning model that's a shell or semi-trained and continues learning over time?<\/p>\n\n<p>Using the penguins classification dataset as my canvas, I envisioned something like this:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"n\">pydantic<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">BaseModel<\/span>\n<span class=\"kn\">from<\/span> <span class=\"n\">fastapi<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">FastAPI<\/span>\n\n\n<span class=\"n\">app<\/span> <span class=\"o\">=<\/span> <span class=\"nc\">FastAPI<\/span><span class=\"p\">(<\/span><span class=\"n\">title<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">\ud83d\ude02 Pure Joy<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">Attributes<\/span><span class=\"p\">(<\/span><span class=\"n\">BaseModel<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">island<\/span><span class=\"p\">:<\/span><span class=\"nb\">str<\/span>\n    <span class=\"n\">bill_length_mm<\/span><span class=\"p\">:<\/span> <span class=\"nb\">float<\/span> <span class=\"o\">|<\/span> <span class=\"bp\">None<\/span>\n    <span class=\"n\">bill_depth_mm<\/span><span class=\"p\">:<\/span> <span class=\"nb\">float<\/span> <span class=\"o\">|<\/span> <span class=\"bp\">None<\/span>\n    <span class=\"n\">flipper_length_mm<\/span><span class=\"p\">:<\/span> <span class=\"nb\">float<\/span> <span class=\"o\">|<\/span> <span class=\"bp\">None<\/span>\n    <span class=\"n\">body_mass_g<\/span><span class=\"p\">:<\/span> <span class=\"nb\">float<\/span> <span class=\"o\">|<\/span> <span class=\"bp\">None<\/span>\n    <span class=\"n\">sex<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span> <span class=\"o\">|<\/span> <span class=\"bp\">None<\/span>\n\n<span class=\"k\">class<\/span> <span class=\"nc\">LearnAttributes<\/span><span class=\"p\">(<\/span><span class=\"n\">BaseModel<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">attributes<\/span><span class=\"p\">:<\/span> <span class=\"n\">Attributes<\/span>\n    <span class=\"n\">species<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span>\n\n<span class=\"nd\">@app.on_event<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">startup<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">startup_event<\/span><span class=\"p\">():<\/span>\n    <span class=\"c1\"># check and use if the ml model exists else create one\n<\/span>    <span class=\"bp\">...<\/span>\n\n\n<span class=\"nd\">@app.post<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">\/predict<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">predict<\/span><span class=\"p\">(<\/span><span class=\"n\">attributes<\/span><span class=\"p\">:<\/span> <span class=\"n\">Attributes<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">dict<\/span><span class=\"p\">:<\/span>\n\n    <span class=\"c1\"># predict\n<\/span>    <span class=\"bp\">...<\/span>\n\n<span class=\"nd\">@app.post<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">\/learn<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">learn<\/span><span class=\"p\">(<\/span><span class=\"n\">learn_attributes<\/span><span class=\"p\">:<\/span> <span class=\"n\">LearnAttributes<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">dict<\/span>\n    <span class=\"c1\"># learn and update the ml model\n<\/span>    <span class=\"bp\">...<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<h3>\n  \n  \n  <strong>Swimming the River of Learning<\/strong>\n<\/h3>\n\n<p>But just tweaking the API felt too straightforward. I wanted both the API and the machine learning model to be dynamic, evolving in tandem. Enter <code>River<\/code>, a tool tailor-made for online machine learning. As new data streamed in, the model continually refines its skills.<\/p>\n\n<p>Here's a basic model:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"c1\"># ml.py\n<\/span><span class=\"kn\">from<\/span> <span class=\"n\">river<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">compose<\/span>\n<span class=\"kn\">from<\/span> <span class=\"n\">river<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">preprocessing<\/span><span class=\"p\">,<\/span> <span class=\"n\">stats<\/span>\n<span class=\"kn\">from<\/span> <span class=\"n\">river<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">naive_bayes<\/span>\n\n\n<span class=\"k\">def<\/span> <span class=\"nf\">penguins_model<\/span><span class=\"p\">():<\/span>\n    <span class=\"n\">island_transformation<\/span> <span class=\"o\">=<\/span> <span class=\"n\">compose<\/span><span class=\"p\">.<\/span><span class=\"nc\">Select<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">island<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span> <span class=\"o\">|<\/span> <span class=\"n\">preprocessing<\/span><span class=\"p\">.<\/span><span class=\"nc\">OneHotEncoder<\/span><span class=\"p\">(<\/span>\n        <span class=\"n\">drop_first<\/span><span class=\"o\">=<\/span><span class=\"bp\">True<\/span>\n    <span class=\"p\">)<\/span>\n\n    <span class=\"n\">sex_transformation<\/span> <span class=\"o\">=<\/span> <span class=\"p\">(<\/span>\n        <span class=\"n\">compose<\/span><span class=\"p\">.<\/span><span class=\"nc\">Select<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">sex<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n        <span class=\"o\">|<\/span> <span class=\"n\">preprocessing<\/span><span class=\"p\">.<\/span><span class=\"nc\">StatImputer<\/span><span class=\"p\">((<\/span><span class=\"sh\">\"<\/span><span class=\"s\">sex<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">stats<\/span><span class=\"p\">.<\/span><span class=\"nc\">Mode<\/span><span class=\"p\">()))<\/span>\n        <span class=\"o\">|<\/span> <span class=\"n\">preprocessing<\/span><span class=\"p\">.<\/span><span class=\"nc\">OneHotEncoder<\/span><span class=\"p\">(<\/span><span class=\"n\">drop_first<\/span><span class=\"o\">=<\/span><span class=\"bp\">True<\/span><span class=\"p\">)<\/span>\n    <span class=\"p\">)<\/span>\n\n    <span class=\"n\">numeric_transformation<\/span> <span class=\"o\">=<\/span> <span class=\"n\">compose<\/span><span class=\"p\">.<\/span><span class=\"nc\">Select<\/span><span class=\"p\">(<\/span>\n        <span class=\"sh\">\"<\/span><span class=\"s\">bill_length_mm<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span>\n        <span class=\"sh\">\"<\/span><span class=\"s\">bill_depth_mm<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span>\n        <span class=\"sh\">\"<\/span><span class=\"s\">flipper_length_mm<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span>\n        <span class=\"sh\">\"<\/span><span class=\"s\">body_mass_g<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span>\n    <span class=\"p\">)<\/span> <span class=\"o\">|<\/span> <span class=\"n\">preprocessing<\/span><span class=\"p\">.<\/span><span class=\"nc\">StatImputer<\/span><span class=\"p\">(<\/span>\n        <span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">bill_length_mm<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">stats<\/span><span class=\"p\">.<\/span><span class=\"nc\">Mean<\/span><span class=\"p\">()),<\/span>\n        <span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">bill_depth_mm<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">stats<\/span><span class=\"p\">.<\/span><span class=\"nc\">Mean<\/span><span class=\"p\">()),<\/span>\n        <span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">flipper_length_mm<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">stats<\/span><span class=\"p\">.<\/span><span class=\"nc\">Mean<\/span><span class=\"p\">()),<\/span>\n        <span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">body_mass_g<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">stats<\/span><span class=\"p\">.<\/span><span class=\"nc\">Mean<\/span><span class=\"p\">()),<\/span>\n    <span class=\"p\">)<\/span>\n\n    <span class=\"n\">model<\/span> <span class=\"o\">=<\/span> <span class=\"p\">(<\/span>\n        <span class=\"n\">island_transformation<\/span> <span class=\"o\">+<\/span> <span class=\"n\">sex_transformation<\/span> <span class=\"o\">+<\/span> <span class=\"n\">numeric_transformation<\/span>\n        <span class=\"o\">|<\/span> <span class=\"n\">naive_bayes<\/span><span class=\"p\">.<\/span><span class=\"nc\">MultinomialNB<\/span><span class=\"p\">(<\/span><span class=\"n\">alpha<\/span><span class=\"o\">=<\/span><span class=\"mi\">1<\/span><span class=\"p\">)<\/span>\n    <span class=\"p\">)<\/span>\n\n    <span class=\"k\">return<\/span> <span class=\"n\">model<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fjoc7hsmw2p78ye5szvyt.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fjoc7hsmw2p78ye5szvyt.png\" alt=\"modelflow\" width=\"800\" height=\"410\"><\/a><\/p>\n\n<p>Putting the Lego pieces together into a complete, testable code.<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"kn\">from<\/span> <span class=\"n\">pathlib<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Path<\/span>\n<span class=\"kn\">import<\/span> <span class=\"n\">pickle<\/span>\n<span class=\"bp\">...<\/span>\n\n<span class=\"n\">MODEL_FILE<\/span> <span class=\"o\">=<\/span> <span class=\"sh\">\"<\/span><span class=\"s\">model\/naive.pickle<\/span><span class=\"sh\">\"<\/span>\n<span class=\"n\">app<\/span> <span class=\"o\">=<\/span> <span class=\"nc\">FastAPI<\/span><span class=\"p\">(<\/span><span class=\"n\">title<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">\ud83d\ude02 Pure Joy<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n\n<span class=\"bp\">...<\/span>\n\n<span class=\"nd\">@app.on_event<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">startup<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">startup_event<\/span><span class=\"p\">():<\/span>\n    <span class=\"n\">model_file<\/span> <span class=\"o\">=<\/span> <span class=\"nc\">Path<\/span><span class=\"p\">(<\/span><span class=\"n\">MODEL_FILE<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">if<\/span> <span class=\"ow\">not<\/span> <span class=\"n\">model_file<\/span><span class=\"p\">.<\/span><span class=\"nf\">exists<\/span><span class=\"p\">():<\/span>\n        <span class=\"kn\">from<\/span> <span class=\"n\">ml<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">penguins_model<\/span>\n\n        <span class=\"n\">app<\/span><span class=\"p\">.<\/span><span class=\"n\">state<\/span><span class=\"p\">.<\/span><span class=\"n\">ml<\/span> <span class=\"o\">=<\/span> <span class=\"nf\">penguins_model<\/span><span class=\"p\">()<\/span>\n    <span class=\"k\">else<\/span><span class=\"p\">:<\/span>\n        <span class=\"n\">app<\/span><span class=\"p\">.<\/span><span class=\"n\">state<\/span><span class=\"p\">.<\/span><span class=\"n\">ml<\/span> <span class=\"o\">=<\/span> <span class=\"n\">pickle<\/span><span class=\"p\">.<\/span><span class=\"nf\">loads<\/span><span class=\"p\">(<\/span><span class=\"n\">model_file<\/span><span class=\"p\">.<\/span><span class=\"nf\">read_bytes<\/span><span class=\"p\">())<\/span>\n\n\n<span class=\"nd\">@app.post<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">\/predict<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">predict<\/span><span class=\"p\">(<\/span><span class=\"n\">attributes<\/span><span class=\"p\">:<\/span> <span class=\"n\">Attributes<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">str<\/span> <span class=\"o\">|<\/span> <span class=\"bp\">None<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">X<\/span> <span class=\"o\">=<\/span> <span class=\"n\">attributes<\/span><span class=\"p\">.<\/span><span class=\"nf\">model_dump<\/span><span class=\"p\">()<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">app<\/span><span class=\"p\">.<\/span><span class=\"n\">state<\/span><span class=\"p\">.<\/span><span class=\"n\">ml<\/span><span class=\"p\">.<\/span><span class=\"nf\">predict_one<\/span><span class=\"p\">(<\/span><span class=\"n\">X<\/span><span class=\"p\">)<\/span>\n\n\n<span class=\"nd\">@app.post<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">\/learn<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">learn<\/span><span class=\"p\">(<\/span><span class=\"n\">learn_attributes<\/span><span class=\"p\">:<\/span> <span class=\"n\">LearnAttributes<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">dict<\/span><span class=\"p\">[<\/span><span class=\"nb\">str<\/span><span class=\"p\">,<\/span> <span class=\"nb\">str<\/span><span class=\"p\">]:<\/span>\n    <span class=\"n\">X<\/span> <span class=\"o\">=<\/span> <span class=\"n\">learn_attributes<\/span><span class=\"p\">.<\/span><span class=\"n\">attributes<\/span><span class=\"p\">.<\/span><span class=\"nf\">model_dump<\/span><span class=\"p\">()<\/span>\n    <span class=\"n\">y<\/span> <span class=\"o\">=<\/span> <span class=\"n\">learn_attributes<\/span><span class=\"p\">.<\/span><span class=\"n\">species<\/span>\n\n    <span class=\"n\">y_pred<\/span> <span class=\"o\">=<\/span> <span class=\"n\">app<\/span><span class=\"p\">.<\/span><span class=\"n\">state<\/span><span class=\"p\">.<\/span><span class=\"n\">ml<\/span><span class=\"p\">.<\/span><span class=\"nf\">predict_one<\/span><span class=\"p\">(<\/span><span class=\"n\">X<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">app<\/span><span class=\"p\">.<\/span><span class=\"n\">state<\/span><span class=\"p\">.<\/span><span class=\"n\">ml<\/span><span class=\"p\">.<\/span><span class=\"nf\">learn_one<\/span><span class=\"p\">(<\/span><span class=\"n\">X<\/span><span class=\"p\">,<\/span> <span class=\"n\">y<\/span><span class=\"p\">)<\/span>\n\n    <span class=\"nc\">Path<\/span><span class=\"p\">(<\/span><span class=\"n\">MODEL_FILE<\/span><span class=\"p\">).<\/span><span class=\"nf\">write_bytes<\/span><span class=\"p\">(<\/span><span class=\"n\">pickle<\/span><span class=\"p\">.<\/span><span class=\"nf\">dumps<\/span><span class=\"p\">(<\/span><span class=\"n\">app<\/span><span class=\"p\">.<\/span><span class=\"n\">state<\/span><span class=\"p\">.<\/span><span class=\"n\">ml<\/span><span class=\"p\">))<\/span>\n\n    <span class=\"k\">return<\/span> <span class=\"p\">{<\/span><span class=\"sh\">\"<\/span><span class=\"s\">status<\/span><span class=\"sh\">\"<\/span><span class=\"p\">:<\/span> <span class=\"sa\">f<\/span><span class=\"sh\">\"<\/span><span class=\"s\">we learned <\/span><span class=\"si\">{<\/span><span class=\"n\">y<\/span><span class=\"si\">}<\/span><span class=\"s\">. We initially predicted <\/span><span class=\"si\">{<\/span><span class=\"n\">y_pred<\/span><span class=\"si\">}<\/span><span class=\"sh\">\"<\/span><span class=\"p\">}<\/span>\n\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Boom! That was easy. Let's run<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight shell\"><code>uvicorn app:app <span class=\"nt\">--reload<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>\ud83c\udf89 Hurrah ... The code worked seamlessly. That is, until multiple workers entered the fray, leading to unexpected pandemonium. Why? The reason lies in <code>uvicorn<\/code>'s workers: each operates independently, causing utter chaos, especially when executing:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight shell\"><code>uvicorn app:app <span class=\"nt\">--reload<\/span> <span class=\"nt\">--workers<\/span> 4\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>To better illustrate the conundrum, let's introduce counters for both predictions and learnings:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"bp\">...<\/span>\n\n<span class=\"nd\">@app.on_event<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">startup<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">startup_event<\/span><span class=\"p\">():<\/span>\n    <span class=\"n\">model_file<\/span> <span class=\"o\">=<\/span> <span class=\"nc\">Path<\/span><span class=\"p\">(<\/span><span class=\"n\">MODEL_FILE<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">if<\/span> <span class=\"ow\">not<\/span> <span class=\"n\">model_file<\/span><span class=\"p\">.<\/span><span class=\"nf\">exists<\/span><span class=\"p\">():<\/span>\n       <span class=\"bp\">...<\/span>\n        <span class=\"n\">ml<\/span><span class=\"p\">.<\/span><span class=\"n\">meta<\/span> <span class=\"o\">=<\/span> <span class=\"p\">{<\/span><span class=\"sh\">\"<\/span><span class=\"s\">predicted<\/span><span class=\"sh\">\"<\/span><span class=\"p\">:<\/span> <span class=\"mi\">0<\/span><span class=\"p\">,<\/span>\n                   <span class=\"sh\">\"<\/span><span class=\"s\">learned<\/span><span class=\"sh\">\"<\/span><span class=\"p\">:<\/span> <span class=\"mi\">0<\/span><span class=\"p\">}<\/span>\n\n        <span class=\"n\">app<\/span><span class=\"p\">.<\/span><span class=\"n\">state<\/span><span class=\"p\">.<\/span><span class=\"n\">ml<\/span> <span class=\"o\">=<\/span> <span class=\"n\">ml<\/span>\n    <span class=\"k\">else<\/span><span class=\"p\">:<\/span>\n        <span class=\"n\">app<\/span><span class=\"p\">.<\/span><span class=\"n\">state<\/span><span class=\"p\">.<\/span><span class=\"n\">ml<\/span> <span class=\"o\">=<\/span> <span class=\"n\">pickle<\/span><span class=\"p\">.<\/span><span class=\"nf\">loads<\/span><span class=\"p\">(<\/span><span class=\"n\">model_file<\/span><span class=\"p\">.<\/span><span class=\"nf\">read_bytes<\/span><span class=\"p\">())<\/span>\n\n\n<span class=\"nd\">@app.post<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">\/predict<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">predict<\/span><span class=\"p\">(<\/span><span class=\"n\">attributes<\/span><span class=\"p\">:<\/span> <span class=\"n\">Attributes<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">dict<\/span><span class=\"p\">[<\/span><span class=\"nb\">str<\/span><span class=\"p\">,<\/span> <span class=\"nb\">str<\/span> <span class=\"o\">|<\/span> <span class=\"nb\">int<\/span> <span class=\"o\">|<\/span> <span class=\"bp\">None<\/span><span class=\"p\">]:<\/span>\n    <span class=\"n\">X<\/span> <span class=\"o\">=<\/span> <span class=\"n\">attributes<\/span><span class=\"p\">.<\/span><span class=\"nf\">model_dump<\/span><span class=\"p\">()<\/span>\n    <span class=\"n\">y_pred<\/span>  <span class=\"o\">=<\/span> <span class=\"n\">app<\/span><span class=\"p\">.<\/span><span class=\"n\">state<\/span><span class=\"p\">.<\/span><span class=\"n\">ml<\/span><span class=\"p\">.<\/span><span class=\"nf\">predict_one<\/span><span class=\"p\">(<\/span><span class=\"n\">X<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">app<\/span><span class=\"p\">.<\/span><span class=\"n\">state<\/span><span class=\"p\">.<\/span><span class=\"n\">ml<\/span><span class=\"p\">.<\/span><span class=\"n\">meta<\/span><span class=\"p\">[<\/span><span class=\"sh\">\"<\/span><span class=\"s\">predicted<\/span><span class=\"sh\">\"<\/span><span class=\"p\">]<\/span> <span class=\"o\">+=<\/span> <span class=\"mi\">1<\/span>\n\n    <span class=\"k\">return<\/span> <span class=\"p\">{<\/span><span class=\"sh\">\"<\/span><span class=\"s\">predicted<\/span><span class=\"sh\">\"<\/span><span class=\"p\">:<\/span> <span class=\"n\">y_pred<\/span><span class=\"p\">,<\/span>\n            <span class=\"o\">**<\/span><span class=\"n\">app<\/span><span class=\"p\">.<\/span><span class=\"n\">state<\/span><span class=\"p\">.<\/span><span class=\"n\">ml<\/span><span class=\"p\">.<\/span><span class=\"n\">meta<\/span><span class=\"p\">,}<\/span>\n\n\n<span class=\"nd\">@app.post<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">\/learn<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">learn<\/span><span class=\"p\">(<\/span><span class=\"n\">learn_features<\/span><span class=\"p\">:<\/span> <span class=\"n\">LearnAttributes<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">dict<\/span><span class=\"p\">[<\/span><span class=\"nb\">str<\/span><span class=\"p\">,<\/span> <span class=\"nb\">str<\/span><span class=\"o\">|<\/span><span class=\"nb\">int<\/span><span class=\"p\">]:<\/span>\n    <span class=\"bp\">...<\/span>\n\n    <span class=\"n\">app<\/span><span class=\"p\">.<\/span><span class=\"n\">state<\/span><span class=\"p\">.<\/span><span class=\"n\">ml<\/span><span class=\"p\">.<\/span><span class=\"n\">meta<\/span><span class=\"p\">[<\/span><span class=\"sh\">\"<\/span><span class=\"s\">learned<\/span><span class=\"sh\">\"<\/span><span class=\"p\">]<\/span> <span class=\"o\">+=<\/span> <span class=\"mi\">1<\/span>\n\n    <span class=\"nc\">Path<\/span><span class=\"p\">(<\/span><span class=\"n\">MODEL_FILE<\/span><span class=\"p\">).<\/span><span class=\"nf\">write_bytes<\/span><span class=\"p\">(<\/span><span class=\"n\">pickle<\/span><span class=\"p\">.<\/span><span class=\"nf\">dumps<\/span><span class=\"p\">(<\/span><span class=\"n\">app<\/span><span class=\"p\">.<\/span><span class=\"n\">state<\/span><span class=\"p\">.<\/span><span class=\"n\">ml<\/span><span class=\"p\">)<\/span>\n\n    <span class=\"k\">return<\/span> <span class=\"p\">{<\/span><span class=\"sh\">\"<\/span><span class=\"s\">status<\/span><span class=\"sh\">\"<\/span><span class=\"p\">:<\/span> <span class=\"sa\">f<\/span><span class=\"sh\">\"<\/span><span class=\"s\">we learned <\/span><span class=\"si\">{<\/span><span class=\"n\">y<\/span><span class=\"si\">}<\/span><span class=\"s\">. We initially predicted <\/span><span class=\"si\">{<\/span><span class=\"n\">y_pred<\/span><span class=\"si\">}<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span>\n            <span class=\"o\">**<\/span><span class=\"n\">app<\/span><span class=\"p\">.<\/span><span class=\"n\">state<\/span><span class=\"p\">.<\/span><span class=\"n\">ml<\/span><span class=\"p\">.<\/span><span class=\"n\">meta<\/span><span class=\"p\">,}<\/span>\n\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F99r1ma6txretu2t5ms0k.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F99r1ma6txretu2t5ms0k.png\" alt=\"messy\" width=\"800\" height=\"422\"><\/a><\/p>\n\n<blockquote>\n<p>The tally for predictions and learnings is inexplicably out of sync.<\/p>\n<\/blockquote>\n\n<p>\ud83d\ude2d My joy short lived. The core issue was that our ML model's knowledge wasn't uniformly distributed across workers. It felt like having several chefs in a kitchen, each adding their unique flavour, unaware of the others' contributions.<\/p>\n\n\n\n\n<h3>\n  \n  \n  <strong>The Watchdog's Bark and Bite<\/strong>\n<\/h3>\n\n<p>That's when <code>Watchdog<\/code> barked into the scene. Its event handling capabilities ensured machine learning model updates happened seamlessly based on file changes across workers. Let's add the modification.<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"bp\">...<\/span>\n<span class=\"kn\">from<\/span> <span class=\"n\">watchdog.observers<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Observer<\/span>\n<span class=\"kn\">from<\/span> <span class=\"n\">watchdog.events<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">FileSystemEventHandler<\/span>\n\n\n<span class=\"n\">MODEL_FILE<\/span> <span class=\"o\">=<\/span> <span class=\"sh\">\"<\/span><span class=\"s\">model\/naive.pickle<\/span><span class=\"sh\">\"<\/span>\n<span class=\"n\">app<\/span> <span class=\"o\">=<\/span> <span class=\"nc\">FastAPI<\/span><span class=\"p\">(<\/span><span class=\"n\">title<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">\ud83d\ude02 Pure Joy<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n\n\n<span class=\"k\">class<\/span> <span class=\"nc\">FileHandler<\/span><span class=\"p\">(<\/span><span class=\"n\">FileSystemEventHandler<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">on_modified<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">event<\/span><span class=\"p\">):<\/span>\n        <span class=\"nf\">print<\/span><span class=\"p\">(<\/span><span class=\"sa\">f<\/span><span class=\"sh\">\"<\/span><span class=\"s\">path=<\/span><span class=\"si\">{<\/span><span class=\"n\">event<\/span><span class=\"p\">.<\/span><span class=\"n\">src_path<\/span><span class=\"si\">}<\/span><span class=\"s\"> event=<\/span><span class=\"si\">{<\/span><span class=\"n\">event<\/span><span class=\"p\">.<\/span><span class=\"n\">event_type<\/span><span class=\"si\">}<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n        <span class=\"n\">app<\/span><span class=\"p\">.<\/span><span class=\"n\">state<\/span><span class=\"p\">.<\/span><span class=\"n\">ml<\/span> <span class=\"o\">=<\/span> <span class=\"n\">pickle<\/span><span class=\"p\">.<\/span><span class=\"nf\">loads<\/span><span class=\"p\">(<\/span><span class=\"nc\">Path<\/span><span class=\"p\">(<\/span><span class=\"n\">MODEL_FILE<\/span><span class=\"p\">).<\/span><span class=\"nf\">read_bytes<\/span><span class=\"p\">())<\/span>\n\n<span class=\"bp\">...<\/span>\n\n<span class=\"n\">handler<\/span> <span class=\"o\">=<\/span> <span class=\"nc\">FileHandler<\/span><span class=\"p\">()<\/span>\n<span class=\"n\">observer<\/span> <span class=\"o\">=<\/span> <span class=\"nc\">Observer<\/span><span class=\"p\">()<\/span>\n\n\n<span class=\"nd\">@app.on_event<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">startup<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">startup_event<\/span><span class=\"p\">():<\/span>\n    <span class=\"n\">model_file<\/span> <span class=\"o\">=<\/span> <span class=\"nc\">Path<\/span><span class=\"p\">(<\/span><span class=\"n\">MODEL_FILE<\/span><span class=\"p\">)<\/span>\n    <span class=\"k\">if<\/span> <span class=\"ow\">not<\/span> <span class=\"n\">model_file<\/span><span class=\"p\">.<\/span><span class=\"nf\">exists<\/span><span class=\"p\">():<\/span>\n        <span class=\"bp\">...<\/span>\n    <span class=\"k\">else<\/span><span class=\"p\">:<\/span>\n        <span class=\"bp\">...<\/span>\n\n    <span class=\"n\">observer<\/span><span class=\"p\">.<\/span><span class=\"nf\">schedule<\/span><span class=\"p\">(<\/span><span class=\"n\">handler<\/span><span class=\"p\">,<\/span> <span class=\"n\">path<\/span><span class=\"o\">=<\/span><span class=\"n\">MODEL_FILE<\/span><span class=\"p\">,<\/span> <span class=\"n\">recursive<\/span><span class=\"o\">=<\/span><span class=\"bp\">False<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">observer<\/span><span class=\"p\">.<\/span><span class=\"nf\">start<\/span><span class=\"p\">()<\/span>\n\n\n<span class=\"nd\">@app.on_event<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">shutdown<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">shutdown_event<\/span><span class=\"p\">():<\/span>\n    <span class=\"n\">observer<\/span><span class=\"p\">.<\/span><span class=\"nf\">stop<\/span><span class=\"p\">()<\/span>\n    <span class=\"n\">observer<\/span><span class=\"p\">.<\/span><span class=\"nf\">join<\/span><span class=\"p\">()<\/span>\n\n\n<span class=\"nd\">@app.post<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">\/predict<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">predict<\/span><span class=\"p\">(<\/span><span class=\"n\">attributes<\/span><span class=\"p\">:<\/span> <span class=\"n\">Attributes<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">dict<\/span><span class=\"p\">[<\/span><span class=\"nb\">str<\/span><span class=\"p\">,<\/span> <span class=\"nb\">str<\/span> <span class=\"o\">|<\/span> <span class=\"nb\">int<\/span> <span class=\"o\">|<\/span> <span class=\"bp\">None<\/span><span class=\"p\">]:<\/span>\n   <span class=\"bp\">...<\/span>\n\n\n<span class=\"nd\">@app.post<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">\/learn<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">learn<\/span><span class=\"p\">(<\/span><span class=\"n\">learn_attributes<\/span><span class=\"p\">:<\/span> <span class=\"n\">LearnAttributes<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"nb\">dict<\/span><span class=\"p\">[<\/span><span class=\"nb\">str<\/span><span class=\"p\">,<\/span> <span class=\"nb\">str<\/span> <span class=\"o\">|<\/span> <span class=\"nb\">int<\/span><span class=\"p\">]:<\/span>\n   <span class=\"bp\">...<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p><a href=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F5qkhx7syomuqgs10y6mq.png\" class=\"article-body-image-wrapper\"><img src=\"https:\/\/media2.dev.to\/dynamic\/image\/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto\/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F5qkhx7syomuqgs10y6mq.png\" alt=\"unmessy\" width=\"800\" height=\"422\"><\/a><\/p>\n\n<p>\ud83c\udf89 Hurrah! Again \u2026 It worked. But <code>watchdog<\/code> had its quirks. It bite back. Workers still occasionally collided, each clamouring to update the model and sometimes finding the file \"in use\" error. It is chaotic, to say the least. But isn't that what coding for fun is all about? Facing challenges and finding creative solutions.<\/p>\n\n\n\n\n<h3>\n  \n  \n  <strong>Tenacity to the Rescue!<\/strong>\n<\/h3>\n\n<p>That's where <code>Tenacity<\/code> strutted in. Instead of letting workers clash and give up, Tenacity made them... well, tenacious! It equipped them with smart retries. If a worker faced an \"in-use\" error, it patiently waited and retried, ensuring that every update was eventually accounted for.<\/p>\n\n<p>We are going to update our machine learning file to have the loading and saving in order to add the tenaciousness. More over, we can move code arround to cleanup a little. The final code looks like:<\/p>\n\n<p>Getting schemas out<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"c1\"># schemas.py\n<\/span><span class=\"kn\">from<\/span> <span class=\"n\">pydantic<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">BaseModel<\/span>\n\n\n<span class=\"k\">class<\/span> <span class=\"nc\">Attributes<\/span><span class=\"p\">(<\/span><span class=\"n\">BaseModel<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">island<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span>\n    <span class=\"n\">bill_length_mm<\/span><span class=\"p\">:<\/span> <span class=\"nb\">float<\/span> <span class=\"o\">|<\/span> <span class=\"bp\">None<\/span>\n    <span class=\"n\">bill_depth_mm<\/span><span class=\"p\">:<\/span> <span class=\"nb\">float<\/span> <span class=\"o\">|<\/span> <span class=\"bp\">None<\/span>\n    <span class=\"n\">flipper_length_mm<\/span><span class=\"p\">:<\/span> <span class=\"nb\">float<\/span> <span class=\"o\">|<\/span> <span class=\"bp\">None<\/span>\n    <span class=\"n\">body_mass_g<\/span><span class=\"p\">:<\/span> <span class=\"nb\">float<\/span> <span class=\"o\">|<\/span> <span class=\"bp\">None<\/span>\n    <span class=\"n\">sex<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span> <span class=\"o\">|<\/span> <span class=\"bp\">None<\/span>\n\n\n<span class=\"k\">class<\/span> <span class=\"nc\">LearnAttributes<\/span><span class=\"p\">(<\/span><span class=\"n\">BaseModel<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">attributes<\/span><span class=\"p\">:<\/span> <span class=\"n\">Attributes<\/span>\n    <span class=\"n\">species<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span>\n\n\n<span class=\"k\">class<\/span> <span class=\"nc\">Predictions<\/span><span class=\"p\">(<\/span><span class=\"n\">BaseModel<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">species<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span> <span class=\"o\">|<\/span> <span class=\"bp\">None<\/span>\n    <span class=\"n\">predicted<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span>\n    <span class=\"n\">learned<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span>\n\n\n<span class=\"k\">class<\/span> <span class=\"nc\">Learnings<\/span><span class=\"p\">(<\/span><span class=\"n\">BaseModel<\/span><span class=\"p\">):<\/span>\n    <span class=\"n\">status<\/span><span class=\"p\">:<\/span> <span class=\"nb\">str<\/span>\n    <span class=\"n\">predicted<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span>\n    <span class=\"n\">learned<\/span><span class=\"p\">:<\/span> <span class=\"nb\">int<\/span>\n\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>Adding model io(input and out) function that is tenacious<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"c1\"># ml.py\n<\/span><span class=\"kn\">import<\/span> <span class=\"n\">pickle<\/span>\n<span class=\"kn\">from<\/span> <span class=\"n\">pathlib<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Path<\/span>\n<span class=\"kn\">from<\/span> <span class=\"n\">typing<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Literal<\/span>\n\n<span class=\"bp\">...<\/span>\n<span class=\"kn\">from<\/span> <span class=\"n\">tenacity<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">retry<\/span><span class=\"p\">,<\/span> <span class=\"n\">retry_if_exception_type<\/span>\n\n<span class=\"bp\">...<\/span>\n\n<span class=\"nd\">@retry<\/span><span class=\"p\">(<\/span><span class=\"n\">retry<\/span><span class=\"o\">=<\/span><span class=\"nf\">retry_if_exception_type<\/span><span class=\"p\">(<\/span><span class=\"nb\">EOFError<\/span><span class=\"p\">))<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">ml_io<\/span><span class=\"p\">(<\/span>\n    <span class=\"n\">model_file<\/span><span class=\"p\">:<\/span> <span class=\"n\">Path<\/span><span class=\"p\">,<\/span>\n    <span class=\"n\">mode<\/span><span class=\"p\">:<\/span> <span class=\"n\">Literal<\/span><span class=\"p\">[<\/span><span class=\"sh\">\"<\/span><span class=\"s\">wb<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span> <span class=\"sh\">\"<\/span><span class=\"s\">rb<\/span><span class=\"sh\">\"<\/span><span class=\"p\">]<\/span> <span class=\"o\">=<\/span> <span class=\"sh\">\"<\/span><span class=\"s\">rb<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span>\n    <span class=\"n\">ml_object<\/span><span class=\"p\">:<\/span> <span class=\"n\">compose<\/span><span class=\"p\">.<\/span><span class=\"n\">Pipeline<\/span> <span class=\"o\">|<\/span> <span class=\"bp\">None<\/span> <span class=\"o\">=<\/span> <span class=\"bp\">None<\/span><span class=\"p\">,<\/span>\n<span class=\"p\">):<\/span>\n    <span class=\"k\">if<\/span> <span class=\"n\">mode<\/span> <span class=\"o\">==<\/span> <span class=\"sh\">\"<\/span><span class=\"s\">rb<\/span><span class=\"sh\">\"<\/span> <span class=\"ow\">and<\/span> <span class=\"ow\">not<\/span> <span class=\"n\">model_file<\/span><span class=\"p\">.<\/span><span class=\"nf\">exists<\/span><span class=\"p\">():<\/span>\n        <span class=\"n\">ml<\/span> <span class=\"o\">=<\/span> <span class=\"nf\">penguins_model<\/span><span class=\"p\">()<\/span>\n        <span class=\"n\">ml<\/span><span class=\"p\">.<\/span><span class=\"n\">meta<\/span> <span class=\"o\">=<\/span> <span class=\"p\">{<\/span>\n            <span class=\"sh\">\"<\/span><span class=\"s\">predicted<\/span><span class=\"sh\">\"<\/span><span class=\"p\">:<\/span> <span class=\"mi\">0<\/span><span class=\"p\">,<\/span>\n            <span class=\"sh\">\"<\/span><span class=\"s\">learned<\/span><span class=\"sh\">\"<\/span><span class=\"p\">:<\/span> <span class=\"mi\">0<\/span><span class=\"p\">,<\/span>\n        <span class=\"p\">}<\/span>\n        <span class=\"c1\"># save the first local copy (god like ...)\n<\/span>        <span class=\"n\">model_file<\/span><span class=\"p\">.<\/span><span class=\"nf\">write_bytes<\/span><span class=\"p\">(<\/span><span class=\"n\">pickle<\/span><span class=\"p\">.<\/span><span class=\"nf\">dumps<\/span><span class=\"p\">(<\/span><span class=\"n\">ml_object<\/span><span class=\"p\">))<\/span>\n\n        <span class=\"k\">return<\/span> <span class=\"n\">ml<\/span>\n\n    <span class=\"k\">elif<\/span> <span class=\"n\">mode<\/span> <span class=\"o\">==<\/span> <span class=\"sh\">\"<\/span><span class=\"s\">rb<\/span><span class=\"sh\">\"<\/span> <span class=\"ow\">and<\/span> <span class=\"n\">model_file<\/span><span class=\"p\">.<\/span><span class=\"nf\">exists<\/span><span class=\"p\">():<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">pickle<\/span><span class=\"p\">.<\/span><span class=\"nf\">loads<\/span><span class=\"p\">(<\/span><span class=\"n\">model_file<\/span><span class=\"p\">.<\/span><span class=\"nf\">read_bytes<\/span><span class=\"p\">())<\/span>\n\n    <span class=\"k\">elif<\/span> <span class=\"n\">mode<\/span> <span class=\"o\">==<\/span> <span class=\"sh\">\"<\/span><span class=\"s\">wb<\/span><span class=\"sh\">\"<\/span><span class=\"p\">:<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">model_file<\/span><span class=\"p\">.<\/span><span class=\"nf\">write_bytes<\/span><span class=\"p\">(<\/span><span class=\"n\">pickle<\/span><span class=\"p\">.<\/span><span class=\"nf\">dumps<\/span><span class=\"p\">(<\/span><span class=\"n\">ml_object<\/span><span class=\"p\">))<\/span>\n    <span class=\"k\">else<\/span><span class=\"p\">:<\/span>\n        <span class=\"nc\">NotImplemented<\/span><span class=\"p\">(<\/span><span class=\"sa\">f<\/span><span class=\"sh\">\"<\/span><span class=\"s\">mode can only be `wb` or `rb`<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n<p>And for our application:<br>\n<\/p>\n\n<div class=\"highlight js-code-highlight\">\n<pre class=\"highlight python\"><code><span class=\"c1\"># app.py\n<\/span><span class=\"kn\">from<\/span> <span class=\"n\">pathlib<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Path<\/span>\n\n<span class=\"kn\">from<\/span> <span class=\"n\">fastapi<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">FastAPI<\/span>\n<span class=\"kn\">from<\/span> <span class=\"n\">watchdog.observers<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Observer<\/span>\n<span class=\"kn\">from<\/span> <span class=\"n\">watchdog.events<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">FileSystemEventHandler<\/span>\n\n<span class=\"kn\">from<\/span> <span class=\"n\">ml<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">ml_io<\/span>\n<span class=\"kn\">from<\/span> <span class=\"n\">schemas<\/span> <span class=\"kn\">import<\/span> <span class=\"n\">Attributes<\/span><span class=\"p\">,<\/span> <span class=\"n\">LearnAttributes<\/span><span class=\"p\">,<\/span> <span class=\"n\">Predictions<\/span><span class=\"p\">,<\/span> <span class=\"n\">Learnings<\/span>\n\n\n<span class=\"n\">MODEL_FILE<\/span> <span class=\"o\">=<\/span> <span class=\"sh\">\"<\/span><span class=\"s\">model\/naive.pickle<\/span><span class=\"sh\">\"<\/span>\n<span class=\"n\">app<\/span> <span class=\"o\">=<\/span> <span class=\"nc\">FastAPI<\/span><span class=\"p\">(<\/span><span class=\"n\">title<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">\ud83d\ude02 Pure Joy<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n\n\n<span class=\"k\">class<\/span> <span class=\"nc\">FileHandler<\/span><span class=\"p\">(<\/span><span class=\"n\">FileSystemEventHandler<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">on_modified<\/span><span class=\"p\">(<\/span><span class=\"n\">self<\/span><span class=\"p\">,<\/span> <span class=\"n\">event<\/span><span class=\"p\">):<\/span>\n        <span class=\"c1\"># print(f\"path={event.src_path} event={event.event_type}\")\n<\/span>        <span class=\"n\">app<\/span><span class=\"p\">.<\/span><span class=\"n\">state<\/span><span class=\"p\">.<\/span><span class=\"n\">ml<\/span> <span class=\"o\">=<\/span> <span class=\"nf\">ml_io<\/span><span class=\"p\">(<\/span><span class=\"n\">model_file<\/span><span class=\"o\">=<\/span><span class=\"nc\">Path<\/span><span class=\"p\">(<\/span><span class=\"n\">MODEL_FILE<\/span><span class=\"p\">),<\/span> <span class=\"n\">mode<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">rb<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n\n\n<span class=\"n\">handler<\/span> <span class=\"o\">=<\/span> <span class=\"nc\">FileHandler<\/span><span class=\"p\">()<\/span>\n<span class=\"n\">observer<\/span> <span class=\"o\">=<\/span> <span class=\"nc\">Observer<\/span><span class=\"p\">()<\/span>\n\n\n<span class=\"nd\">@app.on_event<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">startup<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">startup_event<\/span><span class=\"p\">():<\/span>\n    <span class=\"n\">model_file<\/span> <span class=\"o\">=<\/span> <span class=\"nc\">Path<\/span><span class=\"p\">(<\/span><span class=\"n\">MODEL_FILE<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">app<\/span><span class=\"p\">.<\/span><span class=\"n\">state<\/span><span class=\"p\">.<\/span><span class=\"n\">ml<\/span> <span class=\"o\">=<\/span> <span class=\"nf\">ml_io<\/span><span class=\"p\">(<\/span><span class=\"n\">model_file<\/span><span class=\"o\">=<\/span><span class=\"n\">model_file<\/span><span class=\"p\">,<\/span> <span class=\"n\">mode<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">rb<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n\n    <span class=\"n\">observer<\/span><span class=\"p\">.<\/span><span class=\"nf\">schedule<\/span><span class=\"p\">(<\/span><span class=\"n\">handler<\/span><span class=\"p\">,<\/span> <span class=\"n\">path<\/span><span class=\"o\">=<\/span><span class=\"n\">MODEL_FILE<\/span><span class=\"p\">,<\/span> <span class=\"n\">recursive<\/span><span class=\"o\">=<\/span><span class=\"bp\">False<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">observer<\/span><span class=\"p\">.<\/span><span class=\"nf\">start<\/span><span class=\"p\">()<\/span>\n\n\n<span class=\"nd\">@app.on_event<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">shutdown<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">shutdown_event<\/span><span class=\"p\">():<\/span>\n    <span class=\"n\">observer<\/span><span class=\"p\">.<\/span><span class=\"nf\">stop<\/span><span class=\"p\">()<\/span>\n    <span class=\"n\">observer<\/span><span class=\"p\">.<\/span><span class=\"nf\">join<\/span><span class=\"p\">()<\/span>\n\n\n<span class=\"nd\">@app.post<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">\/predict<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">predict<\/span><span class=\"p\">(<\/span><span class=\"n\">attributes<\/span><span class=\"p\">:<\/span> <span class=\"n\">Attributes<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">Predictions<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">X<\/span> <span class=\"o\">=<\/span> <span class=\"n\">attributes<\/span><span class=\"p\">.<\/span><span class=\"nf\">model_dump<\/span><span class=\"p\">()<\/span>\n    <span class=\"n\">y_pred<\/span> <span class=\"o\">=<\/span> <span class=\"n\">app<\/span><span class=\"p\">.<\/span><span class=\"n\">state<\/span><span class=\"p\">.<\/span><span class=\"n\">ml<\/span><span class=\"p\">.<\/span><span class=\"nf\">predict_one<\/span><span class=\"p\">(<\/span><span class=\"n\">X<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">app<\/span><span class=\"p\">.<\/span><span class=\"n\">state<\/span><span class=\"p\">.<\/span><span class=\"n\">ml<\/span><span class=\"p\">.<\/span><span class=\"n\">meta<\/span><span class=\"p\">[<\/span><span class=\"sh\">\"<\/span><span class=\"s\">predicted<\/span><span class=\"sh\">\"<\/span><span class=\"p\">]<\/span> <span class=\"o\">+=<\/span> <span class=\"mi\">1<\/span>\n\n    <span class=\"k\">return<\/span> <span class=\"p\">{<\/span>\n        <span class=\"sh\">\"<\/span><span class=\"s\">species<\/span><span class=\"sh\">\"<\/span><span class=\"p\">:<\/span> <span class=\"n\">y_pred<\/span><span class=\"p\">,<\/span>\n        <span class=\"o\">**<\/span><span class=\"n\">app<\/span><span class=\"p\">.<\/span><span class=\"n\">state<\/span><span class=\"p\">.<\/span><span class=\"n\">ml<\/span><span class=\"p\">.<\/span><span class=\"n\">meta<\/span><span class=\"p\">,<\/span>\n    <span class=\"p\">}<\/span>\n\n\n<span class=\"nd\">@app.post<\/span><span class=\"p\">(<\/span><span class=\"sh\">\"<\/span><span class=\"s\">\/learn<\/span><span class=\"sh\">\"<\/span><span class=\"p\">)<\/span>\n<span class=\"k\">def<\/span> <span class=\"nf\">learn<\/span><span class=\"p\">(<\/span><span class=\"n\">learn_attributes<\/span><span class=\"p\">:<\/span> <span class=\"n\">LearnAttributes<\/span><span class=\"p\">)<\/span> <span class=\"o\">-&gt;<\/span> <span class=\"n\">Learnings<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">X<\/span> <span class=\"o\">=<\/span> <span class=\"n\">learn_attributes<\/span><span class=\"p\">.<\/span><span class=\"n\">attributes<\/span><span class=\"p\">.<\/span><span class=\"nf\">model_dump<\/span><span class=\"p\">()<\/span>\n    <span class=\"n\">y<\/span> <span class=\"o\">=<\/span> <span class=\"n\">learn_attributes<\/span><span class=\"p\">.<\/span><span class=\"n\">species<\/span>\n\n    <span class=\"n\">y_pred<\/span> <span class=\"o\">=<\/span> <span class=\"n\">app<\/span><span class=\"p\">.<\/span><span class=\"n\">state<\/span><span class=\"p\">.<\/span><span class=\"n\">ml<\/span><span class=\"p\">.<\/span><span class=\"nf\">predict_one<\/span><span class=\"p\">(<\/span><span class=\"n\">X<\/span><span class=\"p\">)<\/span>\n    <span class=\"n\">app<\/span><span class=\"p\">.<\/span><span class=\"n\">state<\/span><span class=\"p\">.<\/span><span class=\"n\">ml<\/span><span class=\"p\">.<\/span><span class=\"nf\">learn_one<\/span><span class=\"p\">(<\/span><span class=\"n\">X<\/span><span class=\"p\">,<\/span> <span class=\"n\">y<\/span><span class=\"p\">)<\/span>\n\n    <span class=\"n\">app<\/span><span class=\"p\">.<\/span><span class=\"n\">state<\/span><span class=\"p\">.<\/span><span class=\"n\">ml<\/span><span class=\"p\">.<\/span><span class=\"n\">meta<\/span><span class=\"p\">[<\/span><span class=\"sh\">\"<\/span><span class=\"s\">learned<\/span><span class=\"sh\">\"<\/span><span class=\"p\">]<\/span> <span class=\"o\">+=<\/span> <span class=\"mi\">1<\/span>\n\n    <span class=\"nf\">ml_io<\/span><span class=\"p\">(<\/span><span class=\"n\">model_file<\/span><span class=\"o\">=<\/span><span class=\"nc\">Path<\/span><span class=\"p\">(<\/span><span class=\"n\">MODEL_FILE<\/span><span class=\"p\">),<\/span> <span class=\"n\">mode<\/span><span class=\"o\">=<\/span><span class=\"sh\">\"<\/span><span class=\"s\">wb<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span> <span class=\"n\">ml_object<\/span><span class=\"o\">=<\/span><span class=\"n\">app<\/span><span class=\"p\">.<\/span><span class=\"n\">state<\/span><span class=\"p\">.<\/span><span class=\"n\">ml<\/span><span class=\"p\">)<\/span>\n\n    <span class=\"k\">return<\/span> <span class=\"p\">{<\/span>\n        <span class=\"sh\">\"<\/span><span class=\"s\">status<\/span><span class=\"sh\">\"<\/span><span class=\"p\">:<\/span> <span class=\"sa\">f<\/span><span class=\"sh\">\"<\/span><span class=\"s\">learned <\/span><span class=\"si\">{<\/span><span class=\"n\">y<\/span><span class=\"si\">}<\/span><span class=\"s\">. Initially predicted <\/span><span class=\"si\">{<\/span><span class=\"n\">y_pred<\/span><span class=\"si\">}<\/span><span class=\"sh\">\"<\/span><span class=\"p\">,<\/span>\n        <span class=\"o\">**<\/span><span class=\"n\">app<\/span><span class=\"p\">.<\/span><span class=\"n\">state<\/span><span class=\"p\">.<\/span><span class=\"n\">ml<\/span><span class=\"p\">.<\/span><span class=\"n\">meta<\/span><span class=\"p\">,<\/span>\n    <span class=\"p\">}<\/span>\n<\/code><\/pre>\n\n<\/div>\n\n\n\n\n\n\n<h3>\n  \n  \n  <strong>The Sunset of Our Tale<\/strong>\n<\/h3>\n\n<p>This journey was not about creating the code for production. It was a thrilling adventure through challenges and solutions, a testament to the beauty of coding for fun. I experimented, learned, broke things, and eventually built something <em>magnificent<\/em>.<\/p>\n\n<p>The biggest takeaway? Embrace the chaos and curiosity. Revel in the unpredictability. And remember, coding for fun should always be fun. And when in doubt, be tenacious!<\/p>\n\n<p>Until then, keep on coding curiously <\/p>\n\n","category":["python","fastapi","machinelearning","webdev"]}]}}