At OpenSauced, we power over 40,000 generative AI inferences every day, all through our in-house platform ontop of Kubernetes.

The cost of doing this kind of at-scale AI inference with a third party provider API would be astronomic. Thankfully, using Kubernetes, the public cloud, and open-source technologies, we’ve been able to scale with relatively low costs and a lean stack.

In this talk, I walk through the journey of building a production grade generative AI system using open source technologies, open large language models, and Kubernetes. We also explore why OpenSauced chose to build ontop of Kubernetes for our AI workloads over using a third party provider, and how we’re running and managing our AI/ML clusters today. Additionally, I dive into the techniques we used to groom our Retrieval-Augmented-Generation pipelines for efficiency ontop of Kubernetes and other practical tips for deploying your own AI services at-scale

The “.zip” top level domain is an inherently dangerous new route for malicious actors to use and exploit. In mid 2023, John was able to acquire the kubernetes.zip domain: https://twitter.com/johncodezzz/status/1657888452149669888

And through some experimenting and iteration, we were able to serve modified Kubernetes source code through that domain that APPEARS to be from the real Kubernetes GitHub org (where the real source code is available as a zip file).

The real domain for downloading the actual upstream source zipfile is: https://github.com/kubernetes/kubernetes/archive/refs/heads/master.zip

And the malicious one is: https://github.com/kubernetes/kubernetes/archive/refs/heads/@kubernetes.zip

An unsuspecting party could easily download this code, unpack the tarball, and build the bespoke source code with potentially compromised malicious bits.

This talk will include a demo of this exploit, a thorough description of how socially engineered domains fit into the secure software supply-chain, and a call to action for how organizations that consume Kubernetes source code can strengthen their security posture towards these kinds of supply chain attacks (including through verifying signed Kubernetes artifacts, hardened Linux node environments, etc).

In early 2023, I began experimenting with Ollama for local software development. This talk cronicals the expierments, pitfalls, and successes I had in building nvim-llama, a Neovim first, AI coding assistant.

https://github.com/jpmcb/nvim-llama

PancakesCon is a community cybersecurity and hacking conference put on by Stack Overflow. Alongside a technical topic, presenters are encouraged to present on something not tech related.

In this talk, I explore what container escapes are, how to harden against them in your production workflows using tools like Bottlerocket, SE Linux, and defensive security practices. I also discuss the history of funk bass (along with some samples!)

While in Spain for Kubecon EU ‘22, I also appeared on the “Business of Open Source” podcast to discuss maintaining cobra and what startups considering opening sourcing their technologies can learn.

You can listen to the episode here.

Team knowledge, configurations, and infrastructure access can be challenging in a distributed world. Have you ever asked “what’s that command we use?” or “Where are the secrets to access that environment?” - well this talk is for you!

In this talk, I explore using Oh-My-Zsh, a powerful Zsh configuration framework, in order to share team knowledge, distribute common configurations, and share infrastructure access. I also discuss the real world scenario this use case emerged from, challenges faced in this approach, and how you can leverage Oh-My-Zsh for your distributed devops teams.

For years, I’ve maintained Cobra; a Go command line bootstrapping library and core dependency for many CNCF projects, including Kubernetes, Helm, Etcd, Istio, Linkerd, and many more.

In this talk, I discuss the challenges of being a solo maintainer on such an important project, the lottery factor, the need for contributor community, and the secure software supply chain implications this all has for the entire CNCF ecosystem.

Configuring Nvim via Lua is a powerful, native, and fast way to get your Nvim editor environment just how you like it. But what IS Lua?

Well, in this talk, we’ll explore the essentials of this amazing & simple language. You’ll learn how to use it effectively within Nvim, how to create a simple Lua plugin, and how to use it on it’s own via it’s interpreter.