Skip to main content

Data Backends in tif1

tif1 supports two DataFrame libraries: pandas (default) and polars (installed with tif1, selected with lib="polars"). This design balances familiarity, ecosystem compatibility, and raw performance for each use case. The backend system is integrated into the tif1 core architecture. When tif1 loads session data, telemetry, or lap data, it constructs DataFrames with the selected backend. All subsequent operations then use that backend. A backend switch needs few code changes, but the column types differ (see Data Contract by Backend).

Why Multiple Backends?

The Formula 1 data analysis landscape presents unique challenges:
  • Volume: A race session has more than 1,000 laps (1,128 for the 2025 Bahrain Grand Prix race). One lap of telemetry holds 700-800 samples of 23 columns, so the telemetry of a race holds millions of values.
  • Variety: Different analysis tasks have different requirements. Exploratory analysis benefits from the pandas ecosystem, while large pipelines can use the polars query engine.
  • Compatibility: Many existing tools and libraries in the Python ecosystem expect pandas DataFrames
By supporting both backends, tif1 allows optimization for specific constraints without loss of functionality.

Backend Architecture Overview

The backend abstraction layer in tif1 handles the complexity of supporting multiple DataFrame libraries. get_session() loads no data. When code reads session attributes such as .laps or .weather, tif1 runs these steps internally.
  1. Fetches data from the CDN or cache (backend-agnostic)
  2. Parses JSON using orjson (backend-agnostic)
  3. Constructs DataFrames using the specified backend
  4. Applies transformations (column renaming, type casting) using backend-specific code
  5. Returns typed objects with the appropriate DataFrame type
This architecture applies backend-specific optimizations at the lowest level and keeps the high-level API consistent.

Backend Comparison Matrix


Pandas Backend (Default)

Pandas is the default backend in tif1 and provides maximum compatibility with the Python data science ecosystem. Pandas has been the de facto standard for tabular data in Python since 2008. Its ecosystem of tools, libraries, and community knowledge is large.

Architecture and Design

Pandas is built on top of NumPy arrays and provides a rich, high-level API for data manipulation. Key architectural characteristics:
  • Index-based operations: Every DataFrame has an index (row labels) that enables powerful alignment and joining
  • Single-threaded execution: Operations run on a single CPU core, making behavior predictable but potentially slower
  • Eager evaluation: Operations execute immediately when called
  • Object dtype flexibility: Can store mixed types in a single column (though this impacts performance)
  • NaN for missing data: Uses NumPy’s NaN for numeric missing values, None for objects

When to Use Pandas

Pandas is the right choice when:
  • Learning tif1: The pandas API is more forgiving and has extensive documentation.
  • Ecosystem compatibility: Libraries like matplotlib, seaborn, scikit-learn, and statsmodels expect pandas DataFrames.
  • tif1 charts: The tif1.charts functions accept lib="pandas" and lib="polars" and draw the same figure. In both cases, the chart loads the session with the pandas backend, because matplotlib and seaborn use pandas frames. Both backends share the same payload cache. Other lib values raise ValueError.
  • Exploratory analysis: The pandas API and Jupyter integration suit interactive work.
  • Small to medium datasets: For datasets under 1 million rows, pandas performance is typically sufficient.
  • The complete pandas API: Some advanced pandas features (MultiIndex, time series resampling, etc.) are not available in polars.
  • Integration with existing code: When a codebase already uses pandas, staying consistent reduces friction.
  • Stability: Pandas has a stable API with strong backward compatibility guarantees.

Performance Characteristics

Pandas performance varies significantly by operation type:
  • Fast operations: Vectorized numeric operations, boolean indexing, simple aggregations
  • Moderate operations: String operations, groupby with multiple aggregations, merges
  • Slow operations: Apply with Python functions, iteration, complex string parsing
Typical tif1 workloads include filtering laps, computing statistics, and grouping by driver. On the laps of one race (about 1,100 rows), each of these operations takes about 1 ms on pandas.

Example Usage

Pandas-Specific Features in tif1

When using the pandas backend, tif1 provides additional functionality:
  • Categorical dtypes: Driver names and compounds are stored as categoricals for memory efficiency
  • Timedelta columns: Lap times are proper timedelta objects, enabling time arithmetic
  • Index: laps keep a RangeIndex plus the FastF1 index column
  • Method chaining: All tif1 DataFrame operations support pandas method chaining

Polars Backend (High-Performance)

Polars is a fast DataFrame library written in Rust and designed for performance and memory efficiency. It incorporates lessons learned from pandas and uses modern hardware capabilities.

Architecture and Design

Polars is built on Apache Arrow and uses a fundamentally different execution model:
  • Multi-threaded execution: Automatically parallelizes operations across all CPU cores
  • Apache Arrow memory format: Columnar, cache-friendly layout with zero-copy operations
  • Query optimization: Analyzes and optimizes query plans before execution (in lazy mode)
  • SIMD vectorization: Uses CPU SIMD instructions
  • Proper null handling: Native null support without NaN confusion
  • Rust implementation: Memory-safe, no GIL limitations, predictable performance

When to Use Polars

Polars is the right choice when:
  • Large datasets: Parallel execution helps most on large tables, such as the telemetry of many sessions.
  • Data pipelines: Lazy evaluation enables query optimization.
  • Memory is constrained: tif1 polars frames use about 25% less memory for laps and 40% less for telemetry.
  • Heavy aggregations: Groupby operations run in parallel on all CPU cores.
  • A different API: The polars API is more explicit but requires learning.

Performance Characteristics

Polars gives the largest gains on large tables, in these areas:
  • Groupby aggregations, joins, sorting, and filters
  • String operations, window functions, and complex expressions
The tables of one session are small (about 1,100 laps, or 700-800 telemetry samples per lap). At that size, the two backends take about the same time. See Detailed Performance Comparison for measured values.

Installation

Polars (polars>=2.0.0,<3) is installed with tif1. If it is missing, for example after a manual uninstall, reinstall it alongside tif1:
tif1 requires Python 3.11 or later (3.11 through 3.15 are tested in CI), which satisfies the Polars requirement. The library is distributed as pre-built wheels for all major platforms (Linux, macOS, Windows) and architectures (x86_64, ARM64).

Example Usage

Polars 2 notes

tif1 requires polars 2.x. tif1 returns eager DataFrame objects. Most polars 2.0 changes do not affect code that only reads them. The changes below matter for code that builds on the returned frames: The complete list is in the polars 2.0 upgrade guide. Telemetry frames that polars 1.x wrote to the cache load unchanged under polars 2.x. The same release changes the telemetry dtypes for DistanceToDriverAhead and DriverAhead, so tif1 rebuilds its cached telemetry frames once. It rebuilds them from the cached payloads, without a network fetch.

Polars-Specific Features in tif1

With the polars backend, code can use these polars capabilities on the returned frames:
  • Lazy evaluation: Call .lazy() on a polars frame to build an optimized query plan
  • Parallel execution: All operations automatically use available CPU cores
  • Expression API: Complex calculations can be expressed as optimized expressions
  • Efficient string operations: Driver name filtering and compound matching are faster
  • Native Arrow: Zero-copy interop with other Arrow-based tools
  • fastf1 methods: session.laps is a PolarsLaps and telemetry is a PolarsTelemetry. Both are pl.DataFrame subclasses with the pick_* and telemetry methods. A filter, select, sort, or with_columns returns a plain pl.DataFrame. Call the methods on it through the df.f1 namespace, for example laps.filter(...).f1.pick_fastest(). Refer to the Models API.

Detailed Performance Comparison

Performance varies by operation type, data size, and hardware. The values below were measured on one machine, with the tables that tif1 returns for one session.

Measurement Method

  • Hardware: Intel Core i9-13900HX (32 threads)
  • Software: Python 3.14, pandas 3.0.6, polars 2.0.0
  • Dataset: 2025 Bahrain Grand Prix race (1,128 laps; one lap of telemetry has 710 samples)
  • Measurement: Median of 25 runs, with the session already in the local cache

Measured Values

Memory is memory_usage(deep=True) on pandas and estimated_size() on polars.
At the size of one session, both backends finish each operation in about 2 ms or less. Neither backend is faster for every operation. Measure the actual workload before a backend switch.

Example Workflows

These code blocks show complete analysis workflows. Run them on both backends to compare the times on the actual hardware.

Workflow 1: Single Session Analysis

Workflow 2: Multi-Session Aggregation

Workflow 3: Telemetry Analysis

The polars engine runs in parallel and is designed for large tables, such as the telemetry of many sessions. For the tables of one session, the difference is small.

Data Contract by Backend

tif1 loads the same payloads on both backends and then applies a dtype contract per backend. The contract decides how time values, text, and missing values are represented. Read this section before writing code that runs on both backends.

Column types by backend

The tables list the types as loaded for the 2025 Bahrain Grand Prix race. Columns not listed have the same type on both backends: Float64 for speeds and weather readings, String for text. Laps (session.laps) Set TIF1_POLARS_LAP_CATEGORICAL=true to get Categorical for the text columns on polars. See Environment variables. Weather (session.weather) Race control messages (session.race_control_messages) Telemetry (lap.telemetry, session.get_fastest_lap_tel()) DriverAhead holds driver numbers such as "12", not driver codes. To find the samples with nobody ahead, use isna() on pandas or is_null() on polars. The column never holds the string "None". For code that runs on both backends, use LapTimeSeconds for lap times. Compare Driver, Team, and Compound as strings. Both work without a cast on either backend.

Missing values and sentinels

The upstream payloads encode a missing value in two ways: JSON null, and the literal string "None". The string form appears in 2026 lap data and in the Lap column of race control messages for every non-race session. It also appears in telemetry when nobody is ahead. tif1 normalizes both forms on both backends before it applies the dtype contract:
  • Laps: the strings "None", "none", "null", "nan", and "" become a missing value. The replacement runs only when a payload contains one of these tokens.
  • Weather and race control messages: on polars, the same tokens become a missing value before the frame is built. A column with one sentinel then keeps its Boolean, Float64, or Int64 type instead of becoming String.
  • Weather on pandas: the same tokens become a missing value before the astype and to_numeric passes.
  • Race control messages on pandas: the exact string "None" becomes None in the text columns and NaN in Sector. Lap goes through a numeric parse, so any token that is not a number becomes 0.
  • Telemetry: the same tokens become NaN on pandas and null on polars. DistanceToDriverAhead and the other numeric channels stay float64 or Float64. DriverAhead stays a text column of driver numbers.
  • Casts are non-strict on both backends. A value that cannot be parsed becomes a missing value; it does not raise.
The representation of a missing value follows the backend. polars uses null for every type. pandas uses NaT for time columns, NaN for floats, <NA> for nullable integers and booleans, and None for text columns. One difference is intentional. The race control Lap column is a plain int64 on pandas (FastF1 convention). A message without a lap has Lap == 0 on pandas and null on polars.

Empty weather or race control tables

Weather and race control messages are optional tables. When a payload cannot be fetched (NetworkError), fails validation, or cannot be converted to the dtype contract, tif1 logs a warning. It returns an empty DataFrame for that table instead of raising. session.load(), session.laps, and telemetry are not affected. The behaviour is the same on both backends. The warning names the table and the session, then gives the error. The event name is decoded, as in session.event_name:
Enable logging to see it, and test for an empty table before plotting:
A session that has no weather file at all logs No weather in session at INFO level and also returns an empty DataFrame.

Comprehensive API Differences

While tif1 abstracts many differences, the API variations still matter for idiomatic code in each backend. This section compares common operations. Each tab continues from the setup tab of the same backend:

Data Selection and Filtering

Column Selection and Manipulation

Grouping and Aggregation

Sorting

Joining and Merging

Missing Data Handling

String Operations

Window Functions

Type Conversion


Converting Between Backends

A frame from one backend converts to the other with the polars functions DataFrame.to_pandas() and pl.from_pandas(). This lets different parts of a workflow use the strengths of each library.
Both conversions use pyarrow. pip install tif1 installs it, so no extra install is necessary.

Basic Conversion

Conversion Performance

Conversion between backends has a cost, so use it strategically. On the machine of the Measurement Method, the laps of one race (1,128 rows) took these times:
Frequent conversions can negate the performance benefits of using polars. Design the workflow to minimize backend switches.

Zero-Copy Conversion (Arrow)

For maximum efficiency, use Apache Arrow as an intermediate format. pyarrow is a tif1 dependency, so it is already installed:
Zero-copy conversions avoid memory duplication and are much faster for large datasets. These conversions require compatible data types.

Hybrid Workflows

Combine backends strategically for optimal performance:

Preserving Data Types

Some data types do not convert exactly between backends:

Setting Default Backend

Configure tif1 to use a preferred backend by default. This avoids the need to specify lib= on every call.

Via Configuration File

Create or edit ~/.tif1rc (JSON format):
After this setting, all sessions use polars by default:

Via Environment Variable

Set the TIF1_LIB environment variable:
Environment variables take precedence over the config file.

Programmatically

Set the backend at runtime using the config API:

Per-Session Override

Override the default on a per-session basis at any time:

Configuration Precedence

When multiple configuration sources are present, tif1 uses this precedence order (highest to lowest):
  1. Explicit parameter: lib="polars" in function call
  2. Runtime call: config.set("lib", "polars") after import
  3. Environment variable: TIF1_LIB=polars, read once when the config loads
  4. Config file: ~/.tif1rc setting
  5. Default: pandas

Backend-Specific Features and Capabilities

Each backend has unique features for specific use cases.

Pandas-Exclusive Features

The examples in this section use a pandas session and its laps frame:

MultiIndex (Hierarchical Indexing)

Time Series Functionality

Flexible Indexing

Rich Ecosystem Integration

seaborn is installed with tif1. scikit-learn and statsmodels are not; install them with pip install scikit-learn statsmodels.

Polars-Exclusive Features

Lazy Evaluation and Query Optimization

The Parquet examples on this page read a file of polars laps. The first block writes it once with write_parquet:
Lazy evaluation allows polars to optimize the entire query plan, potentially eliminating unnecessary operations and reducing memory usage.

Expression API

Parallel String Operations

Native Arrow Integration

Streaming Mode (Out-of-Core Processing)

Feature Comparison Table


Lazy Evaluation Deep Dive (Polars)

Lazy evaluation is a key polars feature. It enables query optimization that can improve performance by a large factor.

Understanding Lazy vs Eager Execution

In polars 2.x, LazyFrame.collect() runs on the streaming engine by default. The streaming engine does not keep row order after group_by, joins, or unpivot. Add .sort(...), or pass maintain_order (for example join(..., maintain_order="left")), when the order matters. Call collect(engine="in-memory") to use the in-memory engine. Eager DataFrame operations, which tif1 returns, are not affected.

Query Optimization Techniques

Polars applies several optimizations automatically:

1. Predicate Pushdown

Filters are pushed down to the data source. The scan applies them while it reads the file:

2. Projection Pushdown

Only requested columns are read from storage:

3. Filter Reordering

Cheap filters run before expensive ones:

Viewing Query Plans

Examine what polars plans to do before execution:
Output with polars 2.0 (the exact plan text depends on the polars version):
The plan shows predicate pushdown (SELECTION) and projection pushdown (PROJECT 4/45 COLUMNS). The scan reads only the rows and columns that the query needs.

When to Use Lazy Evaluation

Use lazy evaluation when:
  • Reading from files: Parquet, CSV, JSON (use scan_* instead of read_*)
  • Complex queries: Multiple filters, joins, aggregations
  • Large datasets: When data does not fit comfortably in memory
  • Production pipelines: Consistent performance is critical
Do not use lazy evaluation when:
  • Interactive exploration: Eager mode gives immediate feedback
  • Small datasets: The optimization overhead exceeds the gain.
  • Simple operations: Single filter or selection is already fast

Lazy Evaluation Patterns

Pattern 1: ETL Pipeline

Pattern 2: Multi-File Aggregation

Pattern 3: Streaming for Huge Datasets

Performance Impact of Lazy Evaluation

Lazy evaluation benefits increase with query complexity and data size. Simple operations may see minimal improvement.

Decision Framework: Choosing the Right Backend

Use this decision tree to select the best backend for a given use case:

Quick Reference Guide

Detailed Decision Criteria

Choose Pandas When:

  1. Ecosystem Integration is Critical
    • The analysis uses libraries that require pandas (seaborn, scikit-learn, statsmodels)
    • The team standardizes on pandas
    • Existing pandas code needs maintenance
  2. Dataset is Small to Medium
    • Under 100,000 rows: Performance difference is negligible
    • Under 1,000,000 rows: Pandas is still performant enough
  3. Specific Pandas Features
    • MultiIndex for hierarchical data
    • Advanced time series resampling
    • Flexible indexing with loc/iloc
    • Pandas-specific methods that the code depends on
  4. Learning and Exploration
    • Data analysis in Python is new work
    • The work is interactive exploration in Jupyter
    • Stack Overflow coverage is a priority

Choose Polars When:

  1. Performance is Critical
    • Data processing runs in production
    • Consistent, predictable performance is required
    • Time constraints are tight
  2. Dataset is Large
    • Over 1,000,000 rows: Polars parallelization performs best
    • Data does not fit comfortably in RAM: Use streaming
  3. Memory is Constrained
    • Running on limited hardware
    • Processing multiple datasets simultaneously
    • Need to minimize cloud computing costs
  4. Building Data Pipelines
    • ETL workflows benefit from lazy evaluation
    • Query optimization reduces complexity
    • Consistent performance is more important than peak performance
  5. Heavy Aggregations or String Operations
    • Complex groupby with multiple aggregations
    • Extensive string parsing or manipulation
    • Window functions over large groups

Hybrid Approach

One backend is not required for the whole project. Use both strategically:

Advanced Patterns and Best Practices

Pattern 1: Processing Multiple Sessions Efficiently

When analyzing multiple sessions, backend choice significantly impacts performance:

Pattern 2: Memory-Efficient Telemetry Analysis

Telemetry data can be very large. This pattern handles it efficiently:

Pattern 3: Incremental Data Processing

Build results incrementally without loading everything at once:

Pattern 4: Optimizing Groupby Operations

Groupby operations are common in F1 analysis. Optimize them:

Pattern 5: Efficient Filtering

Apply filters efficiently to minimize data scanned:

Pattern 6: Caching Expensive Computations

Cache results of expensive operations:

Best Practices Summary

  • Use polars for large datasets (tif1 polars frames use 25-40% less memory)
  • Process data incrementally when possible
  • Use lazy evaluation for multi-step operations
  • Free memory explicitly with del when done with large objects
  • Monitor memory usage with memory_profiler or similar tools
  • Profile before optimizing (use cProfile or line_profiler)
  • Use polars for operations on > 100k rows
  • Combine multiple aggregations in single groupby
  • Use lazy evaluation for complex queries
  • Avoid Python loops over rows (use vectorized operations)
  • Cache expensive computations with @lru_cache
  • Set default backend in config file for consistency
  • Document backend choice in function docstrings
  • Use type hints: pd.DataFrame or pl.DataFrame
  • Create backend-agnostic interfaces when possible
  • Test with both backends if supporting both
  • Use polars for ETL pipelines
  • Save intermediate results to Parquet format
  • Use lazy evaluation for multi-file processing
  • Implement streaming for datasets larger than RAM
  • Monitor pipeline performance with logging
  • Use .explain() to understand polars query plans
  • Compare results between backends for validation
  • Use .describe() to check data distributions
  • Profile memory usage with both backends
  • Test edge cases (empty DataFrames, single row, nulls)

Troubleshooting and Common Issues

Issue 1: Polars Removed From the Environment

Symptom: import polars raises ModuleNotFoundError: No module named 'polars'. A session requested with lib="polars" logs Polars not available, falling back to pandas and returns pandas frames. Polars ships with tif1, so this means it was uninstalled or the environment is broken. Solution:

Issue 2: Type Errors After Backend Switch

Symptom: Code works with pandas but fails with polars (or vice versa) Solution: Check for backend-specific API usage

Issue 3: Memory Issues with Pandas

Symptom: MemoryError or system slowdown with large datasets Solution: Switch to polars or use chunking

Issue 4: Slow Groupby Operations

Symptom: Groupby on a large table takes several seconds with pandas Solution: Use polars, which runs the aggregations in parallel in one pass

Issue 5: Conversion Overhead

Symptom: Frequent conversions between backends slow down code Solution: Minimize conversions, do them once

Issue 6: Lazy Evaluation Confusion

Symptom: A polars lazy query does not return data Solution: Remember to call .collect()

Issue 7: Incompatible Library Expectations

Symptom: Library expects pandas DataFrame but receives polars Solution: Convert before passing to library

Issue 8: Column Name Conflicts

Symptom: Errors with column names containing spaces or special characters Solution: Select such columns by string; query() and attribute access need valid identifiers

Issue 9: Performance Not Improving with Polars

Symptom: Polars is not faster than pandas for the given use case Possible Causes:
  1. Dataset is too small (< 10k rows)
  2. Using eager mode instead of lazy
  3. Bottleneck is I/O, not computation
  4. Single-threaded operations (some string ops)
Solution: Profile and optimize

Issue 10: Type Inference Differences

Symptom: The same column has a different type on each backend. LapTime is timedelta64[ns] on pandas and Duration("ns") on polars; Driver is category on pandas and String on polars. See Column types by backend for the full table. Solution: Use the columns that have the same type on both backends; LapTimeSeconds is a float on both. Alternatively, normalise once after loading:

Common Use Cases

Performance guide.

Performance

Optimization.

Installation

Install tif1 (polars included).

Summary and Key Takeaways

Quick Summary

  • Pandas: Default backend, maximum compatibility, gentle learning curve
  • Polars: Parallel query engine, lazy evaluation, 25-40% less memory for tif1 frames
  • Choice: Depends on dataset size, performance needs, and ecosystem requirements
  • Flexibility: Use both in the same project, convert as needed
  • Recommendation: Start with pandas, migrate to polars for performance-critical code

Key Performance Insights

  1. Polars performs best with large datasets: on one session, both backends take about 1 ms per operation
  2. Memory efficiency matters: tif1 polars frames use 25-40% less memory
  3. Lazy evaluation helps on files: predicate and projection pushdown read less data
  4. Conversion has cost: Minimize backend switches
  5. Ecosystem compatibility: Pandas still dominates for visualization and ML

When to Use Each Backend

Use Pandas for:
  • Learning and exploration
  • Small to medium datasets (< 100k rows)
  • Ecosystem integration (seaborn, scikit-learn)
  • Time series analysis
  • Hierarchical data (MultiIndex)
Use Polars for:
  • Large datasets (> 100k rows)
  • Production pipelines
  • Memory-constrained environments
  • Heavy aggregations
  • String-intensive operations

Best Practices Checklist

  • Set default backend in ~/.tif1rc for consistency
  • Profile code before optimizing
  • Use polars for data processing, pandas for visualization
  • Minimize conversions between backends
  • Use lazy evaluation for complex polars queries
  • Cache expensive computations
  • Test with both backends if supporting both
  • Document backend choice in code comments
  • Monitor memory usage for large datasets
  • Use type hints for DataFrame types

Next Steps

1

Experiment

Try both backends with typical workloads to measure the actual performance differences.
2

Profile

Use profiling tools to identify bottlenecks in the code.
3

Optimize

Apply backend-specific optimizations (lazy evaluation, vectorization).
4

Monitor

Track performance and memory usage in production.
5

Iterate

Continuously refine the backend choice based on real-world results.

Installation Guide

Install tif1; polars is included.

Performance Guide

Optimization techniques and best practices.

API Reference

Complete API documentation.

Data Flow

The tif1 data architecture.

Caching Strategy

How tif1 caches data efficiently.

Common Use Cases

Real-world examples and patterns.

Additional Resources

External Documentation

Performance Benchmarks

Community Resources


This documentation is maintained as part of the tif1 project. For questions or suggestions, please open an issue on GitHub.
Last modified on October 30, 2026