Blog
Intelligent inference request routing for large language models
Today's AI environment is experiencing a surge in specialized Large Language Models (LLMs), each possessing unique abilities and strengths123: Some are strong in reasoning and mathematics, while others may excel in creative writing. Yet most applications resort to a...
Enhancing AI inference security with confidential computing: A path to private data inference with proprietary LLMs
Red Hat's Office of the CTO is collaborating in the upstream project Tinfoil community to explore pioneering a complete, cloud-native solution for Confidential AI. The community is focused on solving one of the toughest AI security challenges facing the enterprise:...
A developer’s guide to PyTorch, containers, and NVIDIA – Solving the puzzle
Starting about 5 years ago, I began moving to container-based operating systems (OS). It started with Bazzite and most recently I have been using Aurora. What's not to love? These OS's make containers first-class citizens, simplifying how to "install" and run...
Understanding Triton Cache: Optimizing GPU Kernel Compilation
The goal of this blog post is to explore Triton’s caching mechanism: how it works, what affects it, how different frameworks leverage it, and how you can optimize it for your specific workloads.
Model authenticity and transparency with Sigstore
What is the Sigstore model transparency project? Sigstore’s Model Transparency project is a Sigstore community project aimed at applying the software supply chain security practice of signing to machine learning (ML) models. Hosted on Github at...
A container-first approach to Triton development
The Triton project from OpenAI is at the forefront of a groundbreaking movement to democratize AI accelerators and GPU kernel programming. It provides a powerful and flexible framework for writing high performance GPU kernels. As AI workloads become increasingly...
SPIFFE/SPIRE and Keylime: Software Identity based on Secure Machine State
It's almost impossible to count the number of security breaches that were caused by a leaked password or API key. Secret management in software is a tricky thing to get right. Securely deploying secrets to only the places where they are needed, securely rotating them...
Getting started with PyTorch and Triton on AMD GPUs using the Red Hat Universal Base Image
In a prior blog post, we provided an overview of the Triton language and its ecosystem. Triton is a Python based DSL (Domain Specific Language), compiler and related tooling designed for writing efficient GPU kernels in a hardware-agnostic manner, offering high-level...
User experience and its importance in adoption of democratized AI
"An intuitive and accessible UI is critical to even the most powerful AI system" As artificial intelligence (AI) continues to evolve, its influence spans across industries, transforming operations and enhancing decision-making processes. At this point in time, I...
Democratizing AI Accelerators and GPU Kernel Programming using Triton
Red Hat’s Emerging Technologies blog includes posts that discuss technologies that are under active development in upstream open source communities and at Red Hat. We believe in sharing early and often the things we’re working on, but we want to note that unless...