How Open Data Hub learners become the teachers

The prospect of true machine learning is a tangible goal for data scientists and researchers. It has been long known that the platform on which such ML apps can run have to be fast and hyper efficient so that learning can be that much faster. This is the motivation for Red Hat engineers in the Office of the CTO who are working to optimize such an open source platform: Open Data Hub.

Open Data Hub is built on Red Hat OpenShift Container Platform, Ceph Object Storage, and Apache Kafka/Strimzi integrated into a collection of open source projects to enable a machine-learning-as-a-service platform. That’s a lot of components to be integrated, and to ensure that their contributions to Open Data Hub perform well, Red Hat engineers have taken the step of creating an Internal Data Hub within Red Hat as a proving ground and learning environment.

Continue reading “How Open Data Hub learners become the teachers”

Machine Learning with Open Source Infrastructure

As machine learning becomes more interesting to technology companies, it is hardly surprising that a company like Red Hat is going to approach the challenges of this aspect of artificial intelligence with an open source methodology in mind.

The immediate benefits to open source machine learning tools are plain as day to anyone familiar with how open source works: lower cost, more flexibility, no vendor lock-in… you know, the usual.

But dig a little deeper and it quickly becomes apparent that open source means more for cutting-edge software than just a faster way to get cheaper software. 

Continue reading “Machine Learning with Open Source Infrastructure”

Exploring Unsupervised Deep Learning

The concept of artificial intelligence, which seemed so much like science fiction a few decades ago, has made real, practical inroads in producing results that organizations can find useful. What’s making those results happen, though, isn’t esoteric pie-in-the-sky theory: it’s creating statistical models that have been trained to make decisions. And trained a lot.

Artificial intelligence itself is a term that, for now, has had less of a focus than the more results-oriented machine learning, where a computer system is given input and output data and then is directed to infer the mathematical rules that govern the transformation of that data.

“It’s like pointing a program to look at the solar system and then have it figure out the laws of motion that govern a planetary system,” explained Sanjay Arora.

Continue reading “Exploring Unsupervised Deep Learning”

Building a Scalable TensorFlow Twitter Bot for Red Hat Summit

Red Hat’s AI Center of Excellence and PerceptiLabs wanted a way to demonstrate a TensorFlow model to the public during the 2019 Red Hat Summit. The plan was for this model to take images as input, and then respond with the likelihood of a Red Hat fedora being in that image. Here’s what we learned during Red Hat Summit.

This application, which we called Fedora Finder Bot, would be featured during Red Hat CTO Chris Wright’s keynote, where PerceptiLabs demoed their AI platform.

Our initial solution for this objective would be a Twitter bot that receives tweets or direct messages and replies with the output from the TensorFlow model. Twitter being a public service, we felt it could make the model available to a large number of users, so that any user could just tweet to the bot with a picture and the bot would respond with the model’s output.

Continue reading “Building a Scalable TensorFlow Twitter Bot for Red Hat Summit”

Anomaly Detection on OpenStack Logs Using Machine Learning

(There’s a great  new conference in the U.S., DevConf.US, returning in 2019 to Boston University (15 to 17 Aug). This highly-technical conference is interested in drawing a diverse group of speakers and attendees, with a specific emphasis on people who are new to speaking and tech conferences in general. Only in its second year, DevConf.US builds on the successful decade-spanning run of DevConf.CZ in Brno, CZ.

This is a session from DevConf.US 2018. The call for proposals to present at DevConf.US 2019 is now open.)

In this session from the CentOS Dojo held as part of DevConf.US, OpenStack technical support engineers Madhur Gupta and Shatadru Bandyopadhyay talk about how to use machine learning for anomaly detection on OpenStack logs. Once an anomaly is detected in the logs, it can be used to automate further action, while helping in root cause analysis.

The challenge with anomaly detection in OpenStack in the first place is that it generates a significant quantity of logs, even in relatively simple production setups. How do you ingest and detect anomalies in all that data?

Continue reading “Anomaly Detection on OpenStack Logs Using Machine Learning”

Red Hat’s Open Source AI Vision

Analytics, Machine Learning, and AI represent a fundamental transformation that over the coming decade will affect every aspect of society, business, and industry. It will fundamentally change, how we interact with computers – and how we develop, maintain, and operate systems. It’s impact will be visible in our part of the universe much sooner than for the analog world. This deeply affects both open source in general, as well as Red Hat, its ecosystem, and customer base.

In this video from the inaugural DevConf.US 2018, Daniel Riek who leads the AI Center of Excellence in Red Hat Office of the CTO, talks about this coming change.

Continue reading “Red Hat’s Open Source AI Vision”

Transforming IT Operations: A Roadmap

Digital transformation is more than just a fancy buzzword. With 85 percent of Global 2000 CEOs believing in digital innovation as a driver of business success, it is estimated that nearly $2.1 trillion will be invested in digital transformation technologies in 2019.

According to Mary Johnston Turner, Director, Management Software BU Evangelism,  the drivers to digital transformation are going to play a significant role in driving IT decision-making for the near-term future. Turner outlined the significant driving factors in her 2018 Summit breakout session “Transforming IT Ops: The future of IT automation & management.”

Continue reading “Transforming IT Operations: A Roadmap”

Machine Learning as a Service

Experimenting with machine learning algorithms or integrating such techniques into an existing environment often presents challenges, like selecting and deploying the right infrastructure, in addition to having the necessary data science background and skills, etc. In this post, we present a service that allows users to train machine learning models, run analyses using trained models, as well as manage data required for such models or analyses. Now machine learning models or the prediction results can be easily integrated in to existing continuous integration (CI) or IT infrastructure using REST API.

Overview

The main components of such a service are Apache OpenWhisk, Red Hat OpenShift, and Ceph Storage. These components are available under AI Library at https://gitlab.com/opendatahub/ai-library. OpenWhisk is a serverless computing platform that provides the interface through which users can submit HTTP requests to train or execute machine learning models. HTTP requests submitted to OpenWhisk are actually targeted to stateless functions, called actions, that run on the platform. Ceph Storage is used for storage of training and prediction data, models and results. Users can submit data in to Ceph backend through OpenWhisk actions provided in our implementation (s3.py) or any custom tools such as RADOS object storage utility that can interact with Ceph storage. The action ‘s3.py’ not only supports Ceph, but also any S3-compatible storage backend.

Continue reading “Machine Learning as a Service”

Seeing the Trees in the Forest: Anomaly Detection with Prometheus

Red Hat’s work within the field of artificial intelligence is primarily taking three directions right now. First, our engineers see the inclusion of AI features as a workload requirement for our platforms, as well as AI being applicable to Red Hat’s existing core business in order to increase open source development and production efficiency. In short, Red Hat thinks AI can be good for our customers and good for us, too.

Second, Red Hat is collaborating with the Mass Open Cloud project to establish the one thing that all AI tools need the most: data. Our team members are working on the Open Data Hub, a cloud platform that lets data scientists spend less time on dealing with infrastructure administration and more time building and running their data models.

The third aspect of Red Hat’s work in AI right now is at the application level. More to the point, how can developers plug in AI tools to applications so that data from those applications can be gathered for storage and later modeling?

Continue reading “Seeing the Trees in the Forest: Anomaly Detection with Prometheus”

The Future of Storage in Container Space: Part 4

The challenges of maintaining persistent storage in environments that are anything but persistent should not be taken lightly. My recent conversation with Ceph founder Sage Weil certainly made that clear. Thus far, the conversation with Sage has highlighted key areas of focus for the Red Hat Storage team as they look to the horizon, including how storage plans are affected by:

  • Hardware trends (examined in Part 1)
  • Software platforms (reviewed in Part 2)
  • Multi-cloud and hybrid cloud (discussed in Part 3)

In the last segment of our interview, Sage focused on technology that’s very much on the horizon: the emerging workloads. Specifically, how will storage work in a world where artificial intelligence and machine learning begins to shape software, hardware, and networking architecture?

Continue reading “The Future of Storage in Container Space: Part 4”

  • Page 1 of 2
  • 1
  • 2
  • >