This document will guide you through the most important use cases, functions and classes in the OpenML Python API. Throughout this document, we will use pandas to format and filter tables.
Installation & Set up¶
The OpenML Python package is a connector to OpenML. It allows you to use and share datasets and tasks, run machine learning algorithms on them and then share the results online.
The following tutorial gives a short introduction on how to install and set up the OpenML Python connector, followed up by a simple example.
The configuration file resides in a directory
.config/openml in the home
directory of the user and is called config (More specifically, it resides in the
configuration directory specified by the XDGB Base Directory Specification).
It consists of
key = value pairs which are separated by newlines.
The following keys are defined:
required to access the server. The Introduction tutorial & Setup describes how to obtain an API key.
http://www.openml.org. Alternatively, use
test.openml.orgfor the test server.
if not given, will default to
if set to
run_flow_on_taskor similar methods are called a lookup is performed to see if there already exists such a run on the server. If so, download those results instead.
if not given, will default to
Defines how to react when the server is unavailable or experiencing high load. It determines both how often to attempt to reconnect and how quickly to do so. Please don’t use
humanin an automated script that you run more than one instance of, it might increase the time to complete your jobs and that of others.
human (default): For people running openml in interactive fashion. Try only a few times, but in quick succession.
robot: For people using openml in an automated fashion. Keep trying to reconnect for a longer time, quickly increasing the time between retries.
number of connection retries
default depends on retry_policy (5 for
human, 50 for
0: normal output
1: info output
2: debug output
This file is easily configurable by the
openml command line interface.
To see where the file is stored, and what its values are, use openml configure none.
Set any field with
openml configure FIELD or even all fields with just
It is also possible to try out the latest development version of
openml-python with docker:
docker run -it openml/openml-python
See the openml-python docker documentation for more information.
OpenML contains several key concepts which it needs to make machine learning research shareable. A machine learning experiment consists of one or several runs, which describe the performance of an algorithm (called a flow in OpenML), its hyperparameter settings (called a setup) on a task. A Task is the combination of a dataset, a split and an evaluation metric. In this user guide we will go through listing and exploring existing tasks to actually running machine learning algorithms on them. In a further user guide we will examine how to search through datasets in order to curate a list of tasks.
A further explanation is given in the OpenML user guide.
Working with tasks¶
You can think of a task as an experimentation protocol, describing how to apply a machine learning model to a dataset in a way that is comparable with the results of others (more on how to do that further down). Tasks are containers, defining which dataset to use, what kind of task we’re solving (regression, classification, clustering, etc…) and which column to predict. Furthermore, it also describes how to split the dataset into a train and test set, whether to use several disjoint train and test splits (cross-validation) and whether this should be repeated several times. Also, the task defines a target metric for which a flow should be optimized.
Below you can find our tutorial regarding tasks and if you want to know more you can read the OpenML guide:
Running machine learning algorithms and uploading results¶
In order to upload and share results of running a machine learning algorithm
on a task, we need to create an
OpenMLRun. A run object can
be created by running a
OpenMLFlow or a scikit-learn compatible
model on a task. We will focus on the simpler example of running a
Flows are descriptions of something runable which does the machine learning. A flow contains all information to set up the necessary machine learning library and its dependencies as well as all possible parameters.
A run is the outcome of running a flow on a task. It contains all parameter settings for the flow, a setup string (most likely a command line call) and all predictions of that run. When a run is uploaded to the server, the server automatically calculates several metrics which can be used to compare the performance of different flows to each other.
So far, the OpenML Python connector works only with estimator objects following the scikit-learn estimator API. Those can be directly run on a task, and a flow will automatically be created or downloaded from the server if it already exists.
The next tutorial covers how to train different machine learning models, how to run machine learning models on OpenML data and how to share the results:
OpenML provides a large collection of datasets and the benchmark “OpenML100” which consists of a curated list of datasets.
You can find the dataset that best fits your requirements by making use of the available metadata. The tutorial which follows explains how to get a list of datasets, how to filter the list to find the dataset that suits your requirements and how to download a dataset:
OpenML is about sharing machine learning results and the datasets they were obtained on. Learn how to share your datasets in the following tutorial:
OpenML-Python provides an extension interface to connect machine learning libraries directly to
the API and ships a
scikit-learn extension. You can find more information in the Section