Data Backends in tif1
tif1 supports two DataFrame libraries: pandas (default) and polars (installed with tif1, selected withlib="polars"). This design balances familiarity, ecosystem compatibility, and raw performance for each use case.
The backend system is integrated into the tif1 core architecture. When tif1 loads session data, telemetry, or lap data, it constructs DataFrames with the selected backend. All subsequent operations then use that backend. A backend switch needs few code changes, but the column types differ (see Data Contract by Backend).
Why Multiple Backends?
The Formula 1 data analysis landscape presents unique challenges:- Volume: A race session has more than 1,000 laps (1,128 for the 2025 Bahrain Grand Prix race). One lap of telemetry holds 700-800 samples of 23 columns, so the telemetry of a race holds millions of values.
- Variety: Different analysis tasks have different requirements. Exploratory analysis benefits from the pandas ecosystem, while large pipelines can use the polars query engine.
- Compatibility: Many existing tools and libraries in the Python ecosystem expect pandas DataFrames
Backend Architecture Overview
The backend abstraction layer in tif1 handles the complexity of supporting multiple DataFrame libraries.get_session() loads no data. When code reads session attributes such as .laps or .weather, tif1 runs these steps internally.
- Fetches data from the CDN or cache (backend-agnostic)
- Parses JSON using orjson (backend-agnostic)
- Constructs DataFrames using the specified backend
- Applies transformations (column renaming, type casting) using backend-specific code
- Returns typed objects with the appropriate DataFrame type
Backend Comparison Matrix
Pandas Backend (Default)
Pandas is the default backend in tif1 and provides maximum compatibility with the Python data science ecosystem. Pandas has been the de facto standard for tabular data in Python since 2008. Its ecosystem of tools, libraries, and community knowledge is large.Architecture and Design
Pandas is built on top of NumPy arrays and provides a rich, high-level API for data manipulation. Key architectural characteristics:- Index-based operations: Every DataFrame has an index (row labels) that enables powerful alignment and joining
- Single-threaded execution: Operations run on a single CPU core, making behavior predictable but potentially slower
- Eager evaluation: Operations execute immediately when called
- Object dtype flexibility: Can store mixed types in a single column (though this impacts performance)
- NaN for missing data: Uses NumPy’s NaN for numeric missing values, None for objects
When to Use Pandas
Pandas is the right choice when:- Learning tif1: The pandas API is more forgiving and has extensive documentation.
- Ecosystem compatibility: Libraries like matplotlib, seaborn, scikit-learn, and statsmodels expect pandas DataFrames.
- tif1 charts: The
tif1.chartsfunctions acceptlib="pandas"andlib="polars"and draw the same figure. In both cases, the chart loads the session with the pandas backend, because matplotlib and seaborn use pandas frames. Both backends share the same payload cache. Otherlibvalues raiseValueError. - Exploratory analysis: The pandas API and Jupyter integration suit interactive work.
- Small to medium datasets: For datasets under 1 million rows, pandas performance is typically sufficient.
- The complete pandas API: Some advanced pandas features (MultiIndex, time series resampling, etc.) are not available in polars.
- Integration with existing code: When a codebase already uses pandas, staying consistent reduces friction.
- Stability: Pandas has a stable API with strong backward compatibility guarantees.
Performance Characteristics
Pandas performance varies significantly by operation type:- Fast operations: Vectorized numeric operations, boolean indexing, simple aggregations
- Moderate operations: String operations, groupby with multiple aggregations, merges
- Slow operations: Apply with Python functions, iteration, complex string parsing
Example Usage
Pandas-Specific Features in tif1
When using the pandas backend, tif1 provides additional functionality:- Categorical dtypes: Driver names and compounds are stored as categoricals for memory efficiency
- Timedelta columns: Lap times are proper timedelta objects, enabling time arithmetic
- Index: laps keep a
RangeIndexplus the FastF1indexcolumn - Method chaining: All tif1 DataFrame operations support pandas method chaining
Polars Backend (High-Performance)
Polars is a fast DataFrame library written in Rust and designed for performance and memory efficiency. It incorporates lessons learned from pandas and uses modern hardware capabilities.Architecture and Design
Polars is built on Apache Arrow and uses a fundamentally different execution model:- Multi-threaded execution: Automatically parallelizes operations across all CPU cores
- Apache Arrow memory format: Columnar, cache-friendly layout with zero-copy operations
- Query optimization: Analyzes and optimizes query plans before execution (in lazy mode)
- SIMD vectorization: Uses CPU SIMD instructions
- Proper null handling: Native null support without NaN confusion
- Rust implementation: Memory-safe, no GIL limitations, predictable performance
When to Use Polars
Polars is the right choice when:- Large datasets: Parallel execution helps most on large tables, such as the telemetry of many sessions.
- Data pipelines: Lazy evaluation enables query optimization.
- Memory is constrained: tif1 polars frames use about 25% less memory for laps and 40% less for telemetry.
- Heavy aggregations: Groupby operations run in parallel on all CPU cores.
- A different API: The polars API is more explicit but requires learning.
Performance Characteristics
Polars gives the largest gains on large tables, in these areas:- Groupby aggregations, joins, sorting, and filters
- String operations, window functions, and complex expressions
Installation
Polars (polars>=2.0.0,<3) is installed with tif1. If it is missing, for example after a manual uninstall, reinstall it alongside tif1:
tif1 requires Python 3.11 or later (3.11 through 3.15 are tested in CI), which satisfies the Polars requirement. The library is distributed as pre-built wheels for all major platforms (Linux, macOS, Windows) and architectures (x86_64, ARM64).Example Usage
Polars 2 notes
tif1 requires polars 2.x. tif1 returns eagerDataFrame objects. Most polars 2.0 changes do not affect code that only reads them. The changes below matter for code that builds on the returned frames:
The complete list is in the polars 2.0 upgrade guide. Telemetry frames that polars 1.x wrote to the cache load unchanged under polars 2.x. The same release changes the telemetry dtypes for
DistanceToDriverAhead and DriverAhead, so tif1 rebuilds its cached telemetry frames once. It rebuilds them from the cached payloads, without a network fetch.
Polars-Specific Features in tif1
With the polars backend, code can use these polars capabilities on the returned frames:- Lazy evaluation: Call
.lazy()on a polars frame to build an optimized query plan - Parallel execution: All operations automatically use available CPU cores
- Expression API: Complex calculations can be expressed as optimized expressions
- Efficient string operations: Driver name filtering and compound matching are faster
- Native Arrow: Zero-copy interop with other Arrow-based tools
- fastf1 methods:
session.lapsis aPolarsLapsand telemetry is aPolarsTelemetry. Both arepl.DataFramesubclasses with thepick_*and telemetry methods. Afilter,select,sort, orwith_columnsreturns a plainpl.DataFrame. Call the methods on it through thedf.f1namespace, for examplelaps.filter(...).f1.pick_fastest(). Refer to the Models API.
Detailed Performance Comparison
Performance varies by operation type, data size, and hardware. The values below were measured on one machine, with the tables that tif1 returns for one session.Measurement Method
- Hardware: Intel Core i9-13900HX (32 threads)
- Software: Python 3.14, pandas 3.0.6, polars 2.0.0
- Dataset: 2025 Bahrain Grand Prix race (1,128 laps; one lap of telemetry has 710 samples)
- Measurement: Median of 25 runs, with the session already in the local cache
Measured Values
Memory is
memory_usage(deep=True) on pandas and estimated_size() on polars.
At the size of one session, both backends finish each operation in about 2 ms or less. Neither backend is faster for every operation. Measure the actual workload before a backend switch.
Example Workflows
These code blocks show complete analysis workflows. Run them on both backends to compare the times on the actual hardware.Workflow 1: Single Session Analysis
Workflow 2: Multi-Session Aggregation
Workflow 3: Telemetry Analysis
Data Contract by Backend
tif1 loads the same payloads on both backends and then applies a dtype contract per backend. The contract decides how time values, text, and missing values are represented. Read this section before writing code that runs on both backends.Column types by backend
The tables list the types as loaded for the 2025 Bahrain Grand Prix race. Columns not listed have the same type on both backends:Float64 for speeds and weather readings, String for text.
Laps (session.laps)
Set
TIF1_POLARS_LAP_CATEGORICAL=true to get Categorical for the text columns on polars. See Environment variables.
Weather (session.weather)
Race control messages (
session.race_control_messages)
Telemetry (
lap.telemetry, session.get_fastest_lap_tel())
DriverAhead holds driver numbers such as "12", not driver codes. To find the samples with nobody ahead, use isna() on pandas or is_null() on polars. The column never holds the string "None".
For code that runs on both backends, use LapTimeSeconds for lap times. Compare Driver, Team, and Compound as strings. Both work without a cast on either backend.
Missing values and sentinels
The upstream payloads encode a missing value in two ways: JSONnull, and the literal string "None". The string form appears in 2026 lap data and in the Lap column of race control messages for every non-race session. It also appears in telemetry when nobody is ahead. tif1 normalizes both forms on both backends before it applies the dtype contract:
- Laps: the strings
"None","none","null","nan", and""become a missing value. The replacement runs only when a payload contains one of these tokens. - Weather and race control messages: on polars, the same tokens become a missing value before the frame is built. A column with one sentinel then keeps its
Boolean,Float64, orInt64type instead of becomingString. - Weather on pandas: the same tokens become a missing value before the
astypeandto_numericpasses. - Race control messages on pandas: the exact string
"None"becomesNonein the text columns andNaNinSector.Lapgoes through a numeric parse, so any token that is not a number becomes0. - Telemetry: the same tokens become
NaNon pandas andnullon polars.DistanceToDriverAheadand the other numeric channels stayfloat64orFloat64.DriverAheadstays a text column of driver numbers. - Casts are non-strict on both backends. A value that cannot be parsed becomes a missing value; it does not raise.
null for every type. pandas uses NaT for time columns, NaN for floats, <NA> for nullable integers and booleans, and None for text columns. One difference is intentional. The race control Lap column is a plain int64 on pandas (FastF1 convention). A message without a lap has Lap == 0 on pandas and null on polars.
Empty weather or race control tables
Weather and race control messages are optional tables. When a payload cannot be fetched (NetworkError), fails validation, or cannot be converted to the dtype contract, tif1 logs a warning. It returns an empty DataFrame for that table instead of raising. session.load(), session.laps, and telemetry are not affected. The behaviour is the same on both backends.
The warning names the table and the session, then gives the error. The event name is decoded, as in session.event_name:
No weather in session at INFO level and also returns an empty DataFrame.
Comprehensive API Differences
While tif1 abstracts many differences, the API variations still matter for idiomatic code in each backend. This section compares common operations. Each tab continues from the setup tab of the same backend:Data Selection and Filtering
Column Selection and Manipulation
Grouping and Aggregation
Sorting
Joining and Merging
Missing Data Handling
String Operations
Window Functions
Type Conversion
Converting Between Backends
A frame from one backend converts to the other with the polars functionsDataFrame.to_pandas() and pl.from_pandas(). This lets different parts of a workflow use the strengths of each library.
Both conversions use
pyarrow. pip install tif1 installs it, so no extra install is necessary.Basic Conversion
Conversion Performance
Conversion between backends has a cost, so use it strategically. On the machine of the Measurement Method, the laps of one race (1,128 rows) took these times:Zero-Copy Conversion (Arrow)
For maximum efficiency, use Apache Arrow as an intermediate format.pyarrow is a tif1 dependency, so it is already installed:
Hybrid Workflows
Combine backends strategically for optimal performance:Preserving Data Types
Some data types do not convert exactly between backends:Setting Default Backend
Configure tif1 to use a preferred backend by default. This avoids the need to specifylib= on every call.
Via Configuration File
Create or edit~/.tif1rc (JSON format):
Via Environment Variable
Set theTIF1_LIB environment variable:
Programmatically
Set the backend at runtime using the config API:Per-Session Override
Override the default on a per-session basis at any time:Configuration Precedence
When multiple configuration sources are present, tif1 uses this precedence order (highest to lowest):- Explicit parameter:
lib="polars"in function call - Runtime call:
config.set("lib", "polars")after import - Environment variable:
TIF1_LIB=polars, read once when the config loads - Config file:
~/.tif1rcsetting - Default:
pandas
Backend-Specific Features and Capabilities
Each backend has unique features for specific use cases.Pandas-Exclusive Features
The examples in this section use a pandas session and its laps frame:MultiIndex (Hierarchical Indexing)
Time Series Functionality
Flexible Indexing
Rich Ecosystem Integration
seaborn is installed with tif1. scikit-learn and statsmodels are not; install them withpip install scikit-learn statsmodels.
Polars-Exclusive Features
Lazy Evaluation and Query Optimization
The Parquet examples on this page read a file of polars laps. The first block writes it once withwrite_parquet:
Expression API
Parallel String Operations
Native Arrow Integration
Streaming Mode (Out-of-Core Processing)
Feature Comparison Table
Lazy Evaluation Deep Dive (Polars)
Lazy evaluation is a key polars feature. It enables query optimization that can improve performance by a large factor.Understanding Lazy vs Eager Execution
In polars 2.x,
LazyFrame.collect() runs on the streaming engine by default. The streaming engine does not keep row order after group_by, joins, or unpivot. Add .sort(...), or pass maintain_order (for example join(..., maintain_order="left")), when the order matters. Call collect(engine="in-memory") to use the in-memory engine. Eager DataFrame operations, which tif1 returns, are not affected.Query Optimization Techniques
Polars applies several optimizations automatically:1. Predicate Pushdown
Filters are pushed down to the data source. The scan applies them while it reads the file:2. Projection Pushdown
Only requested columns are read from storage:3. Filter Reordering
Cheap filters run before expensive ones:Viewing Query Plans
Examine what polars plans to do before execution:SELECTION) and projection pushdown (PROJECT 4/45 COLUMNS). The scan reads only the rows and columns that the query needs.
When to Use Lazy Evaluation
Use lazy evaluation when:- Reading from files: Parquet, CSV, JSON (use
scan_*instead ofread_*) - Complex queries: Multiple filters, joins, aggregations
- Large datasets: When data does not fit comfortably in memory
- Production pipelines: Consistent performance is critical
- Interactive exploration: Eager mode gives immediate feedback
- Small datasets: The optimization overhead exceeds the gain.
- Simple operations: Single filter or selection is already fast
Lazy Evaluation Patterns
Pattern 1: ETL Pipeline
Pattern 2: Multi-File Aggregation
Pattern 3: Streaming for Huge Datasets
Performance Impact of Lazy Evaluation
Lazy evaluation benefits increase with query complexity and data size. Simple operations may see minimal improvement.
Decision Framework: Choosing the Right Backend
Use this decision tree to select the best backend for a given use case:Quick Reference Guide
Detailed Decision Criteria
Choose Pandas When:
-
Ecosystem Integration is Critical
- The analysis uses libraries that require pandas (seaborn, scikit-learn, statsmodels)
- The team standardizes on pandas
- Existing pandas code needs maintenance
-
Dataset is Small to Medium
- Under 100,000 rows: Performance difference is negligible
- Under 1,000,000 rows: Pandas is still performant enough
-
Specific Pandas Features
- MultiIndex for hierarchical data
- Advanced time series resampling
- Flexible indexing with loc/iloc
- Pandas-specific methods that the code depends on
-
Learning and Exploration
- Data analysis in Python is new work
- The work is interactive exploration in Jupyter
- Stack Overflow coverage is a priority
Choose Polars When:
-
Performance is Critical
- Data processing runs in production
- Consistent, predictable performance is required
- Time constraints are tight
-
Dataset is Large
- Over 1,000,000 rows: Polars parallelization performs best
- Data does not fit comfortably in RAM: Use streaming
-
Memory is Constrained
- Running on limited hardware
- Processing multiple datasets simultaneously
- Need to minimize cloud computing costs
-
Building Data Pipelines
- ETL workflows benefit from lazy evaluation
- Query optimization reduces complexity
- Consistent performance is more important than peak performance
-
Heavy Aggregations or String Operations
- Complex groupby with multiple aggregations
- Extensive string parsing or manipulation
- Window functions over large groups
Hybrid Approach
One backend is not required for the whole project. Use both strategically:Advanced Patterns and Best Practices
Pattern 1: Processing Multiple Sessions Efficiently
When analyzing multiple sessions, backend choice significantly impacts performance:Pattern 2: Memory-Efficient Telemetry Analysis
Telemetry data can be very large. This pattern handles it efficiently:Pattern 3: Incremental Data Processing
Build results incrementally without loading everything at once:Pattern 4: Optimizing Groupby Operations
Groupby operations are common in F1 analysis. Optimize them:Pattern 5: Efficient Filtering
Apply filters efficiently to minimize data scanned:Pattern 6: Caching Expensive Computations
Cache results of expensive operations:Best Practices Summary
Memory Management
Memory Management
- Use polars for large datasets (tif1 polars frames use 25-40% less memory)
- Process data incrementally when possible
- Use lazy evaluation for multi-step operations
- Free memory explicitly with
delwhen done with large objects - Monitor memory usage with
memory_profileror similar tools
Performance Optimization
Performance Optimization
- Profile before optimizing (use
cProfileorline_profiler) - Use polars for operations on > 100k rows
- Combine multiple aggregations in single groupby
- Use lazy evaluation for complex queries
- Avoid Python loops over rows (use vectorized operations)
- Cache expensive computations with
@lru_cache
Code Organization
Code Organization
- Set default backend in config file for consistency
- Document backend choice in function docstrings
- Use type hints:
pd.DataFrameorpl.DataFrame - Create backend-agnostic interfaces when possible
- Test with both backends if supporting both
Data Pipeline Design
Data Pipeline Design
- Use polars for ETL pipelines
- Save intermediate results to Parquet format
- Use lazy evaluation for multi-file processing
- Implement streaming for datasets larger than RAM
- Monitor pipeline performance with logging
Debugging and Testing
Debugging and Testing
- Use
.explain()to understand polars query plans - Compare results between backends for validation
- Use
.describe()to check data distributions - Profile memory usage with both backends
- Test edge cases (empty DataFrames, single row, nulls)
Troubleshooting and Common Issues
Issue 1: Polars Removed From the Environment
Symptom:import polars raises ModuleNotFoundError: No module named 'polars'. A session requested with lib="polars" logs Polars not available, falling back to pandas and returns pandas frames. Polars ships with tif1, so this means it was uninstalled or the environment is broken.
Solution:
Issue 2: Type Errors After Backend Switch
Symptom: Code works with pandas but fails with polars (or vice versa) Solution: Check for backend-specific API usageIssue 3: Memory Issues with Pandas
Symptom:MemoryError or system slowdown with large datasets
Solution: Switch to polars or use chunking
Issue 4: Slow Groupby Operations
Symptom: Groupby on a large table takes several seconds with pandas Solution: Use polars, which runs the aggregations in parallel in one passIssue 5: Conversion Overhead
Symptom: Frequent conversions between backends slow down code Solution: Minimize conversions, do them onceIssue 6: Lazy Evaluation Confusion
Symptom: A polars lazy query does not return data Solution: Remember to call.collect()
Issue 7: Incompatible Library Expectations
Symptom: Library expects pandas DataFrame but receives polars Solution: Convert before passing to libraryIssue 8: Column Name Conflicts
Symptom: Errors with column names containing spaces or special characters Solution: Select such columns by string;query() and attribute access need valid identifiers
Issue 9: Performance Not Improving with Polars
Symptom: Polars is not faster than pandas for the given use case Possible Causes:- Dataset is too small (< 10k rows)
- Using eager mode instead of lazy
- Bottleneck is I/O, not computation
- Single-threaded operations (some string ops)
Issue 10: Type Inference Differences
Symptom: The same column has a different type on each backend.LapTime is timedelta64[ns] on pandas and Duration("ns") on polars; Driver is category on pandas and String on polars. See Column types by backend for the full table.
Solution: Use the columns that have the same type on both backends; LapTimeSeconds is a float on both. Alternatively, normalise once after loading:
Related Pages
Common Use Cases
Performance guide.
Performance
Optimization.
Installation
Install tif1 (polars included).
Summary and Key Takeaways
Quick Summary
- Pandas: Default backend, maximum compatibility, gentle learning curve
- Polars: Parallel query engine, lazy evaluation, 25-40% less memory for tif1 frames
- Choice: Depends on dataset size, performance needs, and ecosystem requirements
- Flexibility: Use both in the same project, convert as needed
- Recommendation: Start with pandas, migrate to polars for performance-critical code
Key Performance Insights
- Polars performs best with large datasets: on one session, both backends take about 1 ms per operation
- Memory efficiency matters: tif1 polars frames use 25-40% less memory
- Lazy evaluation helps on files: predicate and projection pushdown read less data
- Conversion has cost: Minimize backend switches
- Ecosystem compatibility: Pandas still dominates for visualization and ML
When to Use Each Backend
Use Pandas for:- Learning and exploration
- Small to medium datasets (< 100k rows)
- Ecosystem integration (seaborn, scikit-learn)
- Time series analysis
- Hierarchical data (MultiIndex)
- Large datasets (> 100k rows)
- Production pipelines
- Memory-constrained environments
- Heavy aggregations
- String-intensive operations
Best Practices Checklist
- Set default backend in
~/.tif1rcfor consistency - Profile code before optimizing
- Use polars for data processing, pandas for visualization
- Minimize conversions between backends
- Use lazy evaluation for complex polars queries
- Cache expensive computations
- Test with both backends if supporting both
- Document backend choice in code comments
- Monitor memory usage for large datasets
- Use type hints for DataFrame types
Next Steps
1
Experiment
Try both backends with typical workloads to measure the actual performance differences.
2
Profile
Use profiling tools to identify bottlenecks in the code.
3
Optimize
Apply backend-specific optimizations (lazy evaluation, vectorization).
4
Monitor
Track performance and memory usage in production.
5
Iterate
Continuously refine the backend choice based on real-world results.
Related Documentation
Installation Guide
Install tif1; polars is included.
Performance Guide
Optimization techniques and best practices.
API Reference
Complete API documentation.
Data Flow
The tif1 data architecture.
Caching Strategy
How tif1 caches data efficiently.
Common Use Cases
Real-world examples and patterns.
Additional Resources
External Documentation
- Pandas Documentation - Official pandas documentation
- Polars Documentation - Official polars documentation
- Polars User Guide - Complete polars guide
- Apache Arrow - Arrow format specification
Performance Benchmarks
- Polars Benchmarks - Official polars benchmarks
- Database-like Ops Benchmark - Independent benchmarks (maintained by DuckDB Labs)
Community Resources
- Polars Discord - Active polars community
- Stack Overflow - Q&A for both libraries
This documentation is maintained as part of the tif1 project. For questions or suggestions, please open an issue on GitHub.