The successful application of machine learning to understand the behavior of complex real-world systems from healthcare to climate requires robust methods for processing time series data. This type of data is made up of streams of values that change over time, and can represent topics as varied as a patient’s ECG signal in the ICU or a storm system moving across the Earth.
Highly capable multimodal foundation models, such as Gemini Pro, have recently burst onto the scene and are able to reason not only about text, like the large language models (LLMs) that preceded them, but also about other modalities of input, including images. These new models are powerful in their abilities to consume and understand different kinds of data for real-world use cases, such as demonstrating expert medical knowledge or answering physics questions, but haven’t yet been leveraged to make sense of time-series data at scale, despite the clear importance of this type of data. As chat interfaces mature generally across industries and data modalities, products will need the ability to interrogate time series data via natural language to meet user needs. When working with time series data, previous attempts to improve performance of LLMs have included sophisticated prompt tuning and engineering or training a domain specific encoder.
Today we present work from our recent paper, “Plots Unlock Time-Series Understanding in Multimodal Models”, in which we show that for multimodal models, much like for humans, it is easier to make sense of the data visually by looking at plots of the data rather than sifting through the raw time-series values themselves. Importantly, we show that this does not require any expensive additional training, and instead relies on the native multimodal capabilities of these foundation models. Compared to only using a text format for prompting a multimodal model, we demonstrate that using plots of the time series data can increase performance on classification tasks up to 120%.
