The
core_utils package provides the foundational utilities that power tif1’s high-performance data processing pipeline. While these are primarily internal APIs, they expose capabilities for advanced users. Advanced users gain fine-grained control over data transformations, backend conversions, and performance optimization.This module is the performance engine of tif1. It implements zero-copy operations, optimized JSON parsing, and DataFrame transformations. These features let the library process millions of telemetry rows with minimal overhead.Overview
The core utilities package (tif1.core_utils) is organized into five specialized modules. Each module addresses a critical aspect of the library’s performance and functionality:
- Backend Conversion (
backend_conversion.py) - Zero-copy DataFrame conversion between pandas and polars using Apache Arrow - JSON Utilities (
json_utils.py) - High-performance JSON parsing with orjson, providing 2-3x faster deserialization - Helper Functions (
helpers.py) - DataFrame manipulation, validation, and data transformation utilities - Constants (
constants.py) - Column name mappings, rename dictionaries, and configuration constants - Resource Manager (
resource_manager.py) - Context manager for guaranteed resource cleanup with LIFO ordering
tif1 to achieve its performance goals while maintaining compatibility with both pandas and polars backends. The design philosophy emphasizes:
- Zero-copy operations wherever possible to minimize memory overhead
- Lazy evaluation and deferred imports to reduce startup time
- Type safety with full validation and error handling
- Performance-first design with optimized hot paths
- Backend agnostic APIs that support both pandas and polars
Module Import Structure
Performance Philosophy
The core_utils module is designed with performance as the primary concern:- Lazy imports: polars loads only on first use (
_ensure_polars_available()) - Arrow interchange: Numeric columns move between pandas and polars without a copy
- orjson: JSON parsing and serialization in Rust
- Memoization:
_encode_url_componentkeeps an LRU cache of encoded names - Categorical dtypes: Low-cardinality lap columns use less memory
Backend Conversion
The backend conversion module provides zero-copy DataFrame transformations between pandas and polars using Apache Arrow as the interchange format. This enables switching between backends without the performance penalty of traditional serialization/deserialization.Architecture
The conversion system uses:- Apache Arrow as the zero-copy interchange format
- PyArrow extension arrays for pandas to maintain Arrow memory layout
- Lazy polars imports to avoid dependency requirements when not needed
- No fallback: A failed conversion raises
ValueError
tif1 itself does not call these functions. They are helpers for analysis code that mixes the two backends.
pandas_to_polars
pl.from_pandas() with Arrow as the interchange format, avoiding memory copies when possible.
Parameters:
df(pd.DataFrame): pandas DataFrame to convert. Can contain any pandas-supported dtype including nullable types, categoricals, and datetime types.rechunk(bool, optional): Whether to rechunk the resulting polars DataFrame for optimal memory layout. Default isFalseto preserve zero-copy semantics. Set toTrueto perform many operations on the polars DataFrame with contiguous memory.
pl.DataFrame: polars DataFrame with equivalent data and schema
ImportError: If polars is not installed in the environmentValueError: If the conversion fails due to incompatible data types or corrupted data
- Numeric columns: Converted without a copy (shares memory with the source DataFrame via Arrow)
- String and categorical columns: Copied, O(n)
- Typical performance (one machine, 1M rows): under 1 ms for numeric columns. About 12 ms with a string and a categorical column
polars_to_pandas
df.to_pandas() with PyArrow extension arrays to maintain Arrow memory layout in pandas, enabling zero-copy semantics. pyarrow is a runtime dependency of tif1, so this conversion needs no extra install.
Parameters:
df(pl.DataFrame): polars DataFrame to convert. Supports all polars data types including nested types (List, Struct), categoricals, and temporal types.use_pyarrow(bool, optional): Whether to use PyArrow extension arrays in the resulting pandas DataFrame. Default isTruefor zero-copy conversion. Set toFalseto convert to native pandas dtypes (slower but more compatible with legacy pandas code).
pd.DataFrame: pandas DataFrame with equivalent data and schema
ImportError: If polars is not installed in the environmentValueError: If the conversion fails due to incompatible data types
- With PyArrow arrays (
use_pyarrow=True): Numeric columns are not copied - Native pandas dtypes (
use_pyarrow=False): Every column is converted, O(n) - Typical performance (one machine, 1M rows with a string column): about 7 ms with PyArrow arrays. About 10 ms with native dtypes
convert_backend
df(DataFrame): DataFrame to convert. Can be eitherpd.DataFrameorpl.DataFrame.target_backend(str): Target backend name. Must be either"pandas"or"polars"(case-sensitive).
DataFrame: DataFrame in the target backend format. If already in the target backend, returns the input DataFrame unchanged (no-op).
ValueError: Iftarget_backendis not “pandas” or “polars”, or if the conversion failsImportError: If polars is not installed and target is “polars”
- No-op detection: O(1) type check to avoid unnecessary conversions
- Conversion time: Same as
pandas_to_polarsorpolars_to_pandas - Memory efficient: Uses zero-copy conversion internally
JSON Utilities
The JSON utilities module provides high-performance JSON parsing and serialization. It uses orjson, a fast, correct JSON library for Python written in Rust. This module is critical fortif1’s performance because it parses large JSON payloads from the CDN. These payloads contain lap data, telemetry, and race control messages.
Why orjson?
orjson provides significant advantages over Python’s standard libraryjson module:
- 2-3x faster parsing: Rust-based implementation with SIMD optimizations
- Faster serialization: About 9x faster than stdlib
json.dumpson an 8 MB payload of numbers (one machine) - Native bytes support: Parse directly from HTTP response bodies without decoding
- NumPy support: orjson serializes NumPy arrays only with its
OPT_SERIALIZE_NUMPYoption, andjson_dumpsdoes not set it - Strict correctness: Validates JSON spec compliance
Architecture
The JSON utilities module implements a fallback strategy:- Primary: Use orjson for maximum performance
- Fallback: Use stdlib json if orjson fails (rare edge cases)
- Automatic: No user configuration required
json_loads
payload(str | bytes | bytearray | memoryview): JSON data to parse. Accepts multiple input types for flexibility:str: Standard JSON string (UTF-8 encoded)bytes: Raw bytes from HTTP responses (most efficient)bytearray: Mutable byte arraymemoryview: Zero-copy view of bytes (converted to bytes internally)
Any: Parsed Python object. Common return types:dict: JSON objects{}list: JSON arrays[]str,int,float,bool,None: JSON primitives
json.JSONDecodeError(aValueErrorsubclass): If the payload is not valid JSON. orjson fails first, then the stdlib fallback raises this error.
- Time complexity: O(n) where n is the payload size
- Typical performance (one machine):
- Small payloads (<1KB): about 1μs
- Medium payloads (100KB): about 0.2ms
- Large payloads (10MB): about 30ms
json_dumps
data(Any): Python object to serialize. Supported types:dict,list: Collectionsstr,int,float,bool,None: Primitivesdatetime,date,time: Temporal types (ISO 8601 format)UUID: Converted to stringnumpy.float64scalars: orjson rejects them, so the stdlib fallback serializes them. A payload that also holds adatetimeordatethen raisesTypeError. Convert NumPy values withfloat(),int()or.tolist()first.dataclasses: Serialized to dict. Callmodel_dump()on pydantic models first.
str: JSON string representation of the data
TypeError: If the data contains non-serializable types (for example, NumPy arrays, NumPy integers or pydantic models)
- Time complexity: O(n) where n is the data size
- Typical performance (one machine):
- Small objects (<1KB): about 1μs
- Medium objects (100KB): about 0.1ms
- Large objects (10MB): about 15ms
json_dumps uses orjson, which serialized about 9x faster than stdlib json in the measurement above. It handles datetime objects, UUIDs and dataclasses without custom encoders. Convert NumPy values to Python types first.parse_response_json
response(Any): HTTP response object. Should have either:.contentattribute (bytes): Preferred for performance.json()method: Fallback for compatibility
Any: Parsed Python object from the response JSON body
json.JSONDecodeError(or a subclass from the HTTP library): If the response body is not valid JSON. A bytes parse that fails falls back to.json(), which raises the errorAttributeError: If the response object has neither.contentnor.json()
- Optimized path: Parse the
.contentbytes with orjson, with no decode to text - Fallback path: Use the
.json()method - Speedup: Small with niquests, about 1.05-1.1x on 18 KB and 154 KB CDN files (one machine). niquests also parses with orjson when it is installed, after it decodes the body to text
JSON Utilities Best Practices
- Use bytes when possible: Pass
response.contenttojson_loadsinstead ofresponse.text - Prefer parse_response_json: Use
parse_response_json()for HTTP responses - Trust the fallback: The automatic fallback to stdlib json ensures reliability
- Benchmark each use case: Profile the specific JSON payloads to measure impact
- Handle errors gracefully: Catch
json.JSONDecodeErrorfor malformed JSON
- ❌
json.loads(response.text)- Slow (decode + stdlib parse) - ⚠️
response.json()- Decodes to text first. niquests then parses with orjson;requestsuses stdlib json or simplejson - ✅
json_loads(response.content)- Fast (direct bytes parse) - ✅
parse_response_json(response)- Fast (direct bytes parse,.json()fallback)
Helper Functions
The helpers module (helpers.py) provides utility functions for DataFrame manipulation, validation, and data transformation. These functions carry most of tif1’s data processing work. They handle input validation and complex DataFrame operations across both pandas and polars backends.
Design Philosophy
The helper functions follow these core principles:- Backend Agnostic: All functions work with both pandas and polars DataFrames
- Zero-Copy Optimization: Minimize memory allocations and avoid unnecessary data copies
- Type Safety: Full validation with clear error messages
- Performance First: Optimized hot paths for common operations
- Defensive Programming: Handle edge cases gracefully with fallback strategies
Validation Functions
Input validation is critical for data integrity and user experience. The validation helpers provide complete checks with informative error messages._validate_year
year(int): Year to validate. Must be an integer representing a calendar year.min_year(int): Minimum supported year (inclusive).tif1passesMIN_YEAR(2018).max_year(int): Maximum supported year (inclusive).tif1passesMAX_YEAR(2100).
ValueError: If year is outside the range[min_year, max_year]. Error message includes the valid range and the invalid value provided.
- Time complexity: O(1) - simple integer comparison
- Typical execution: <1μs
The
Session constructor calls this validation before any network request. Early validation fails fast and provides clear feedback._validate_drivers_list
drivers(list[str] | None): List of driver codes to validate. Each code should be a non-empty string (typically 3-letter abbreviations like “VER”, “HAM”, “LEC”). Can beNoneto indicate no filtering.
TypeError: Ifdriversis not a list or None. Error message includes the actual type received.ValueError: If the list is empty, or if any element is not a non-empty string. Error messages are specific to the validation failure.
- Must be a list type (not tuple, set, or other iterable)
- Cannot be an empty list (use
Noneinstead to indicate “all drivers”) - All elements must be strings
- All strings must be non-empty (a whitespace-only string such as
" "passes)
- Time complexity: O(n) where n is the number of drivers
- Typical execution: <10μs for 20 drivers
_validate_lap_number
lap_number(int): Lap number to validate. Must be a positive integer (>= 1).
TypeError: Iflap_numberis not an integer. Error message includes the actual type received.ValueError: Iflap_numberis zero or negative. Error message includes the invalid value.
- Must be an integer type (not float, string, or other numeric type)
- Must be positive (>= 1)
- Time complexity: O(1) - simple type check and comparison
- Typical execution: <1μs
_validate_string_param
param(str): String parameter to validate. Must be a non-empty string with at least one non-whitespace character.param_name(str): Human-readable parameter name for error messages. Used to provide context in error messages (for example, “gp”, “session_type”, “driver”).
TypeError: Ifparamis not a string. Error message includes the parameter name and actual type received.ValueError: Ifparamis empty or contains only whitespace. Error message includes the parameter name.
- Must be a string type (not int, None, or other type)
- Cannot be empty string
"" - Cannot be whitespace-only (for example,
" ","\t","\n")
- Time complexity: O(n) where n is the string length (for whitespace check)
- Typical execution: <5μs for typical parameter lengths
This function uses
.strip() to check for whitespace-only strings, ensuring that parameters like " " are rejected even though they have non-zero length.URL Encoding
_encode_url_component
component(str): String to encode. Can contain any Unicode characters, spaces, or special characters.
str: URL-encoded string with all special characters percent-encoded (for example, space becomes%20,&becomes%26).
- Uses
@lru_cache(maxsize=1024)for memoization - Repeated calls with the same input return cached results instantly
- Cache size of 1024 is sufficient for typical usage (GP names, session types, etc.)
- Time complexity: O(n) for first call, O(1) for cached calls
- Typical execution (one machine):
- Uncached call: about 1μs (the first call of a process takes about 30μs)
- Cached call: <0.1μs (cache lookup)
DataFrame Utility Functions
These functions provide backend-agnostic operations for DataFrame manipulation with both pandas and polars._is_empty_df
df: DataFrame-like object to check. Can bepd.DataFrame,pl.DataFrame, or any object with.emptyor.is_empty()attributes.lib(str): Backend library name ("pandas"or"polars"). Used as a hint for optimization, but the function also performs runtime type checking.
bool:Trueif the DataFrame is empty (zero rows),Falseotherwise.
- Type-based detection: Check
isinstance(df, pd.DataFrame)orisinstance(df, pl.DataFrame) - Attribute-based detection: Check for
.empty(pandas) or.is_empty()(polars) - Fallback: Use
len(df) == 0as last resort
- Time complexity: O(1) - all checks are constant time
- Typical execution: <1μs
This function prefers concrete type checking over the
lib parameter. Some code paths can surface pandas DataFrames even when the configured backend is polars (for example, during backend conversion)._create_empty_df
lib(str): Backend library name. Must be either"pandas"or"polars".
pd.DataFrameiflib == "pandas"pl.DataFrameiflib == "polars"and polars is availablepd.DataFrameas fallback if polars is requested but not installed
- Time complexity: O(1) - creates empty structure
- Typical execution (one machine): about 20μs for pandas, about 1μs for polars
_filter_valid_laptimes
LapTimeSeconds column for analysis. It removes rows whose LapTime is missing (null or NaN). It does not detect pit laps, out laps or deleted laps that have a lap time.
Parameters:
laps: Laps DataFrame (pandas or polars). Must contain aLapTimecolumn.lib(str): Backend library name ("pandas"or"polars").
- Filtered DataFrame with:
- Only rows where
LapTimeis valid (not null/NaN) - New
LapTimeSecondscolumn containing lap time as float (seconds) - For pandas:
LapTimeconverted totimedelta64[ns]dtype - For polars:
LapTimekept as original type,LapTimeSecondsadded as Float64
- Only rows where
- Pandas: Converts
LapTimetotimedelta64[ns]and createsLapTimeSecondsas float - Polars: Casts
LapTimeto Float64 (non-strict) and aliases asLapTimeSeconds - Optimization: Minimizes copies by filtering before copying (pandas) or with one
with_columnsand onefiltercall (polars)
- Time complexity: O(n) where n is the number of laps
- Memory overhead: Minimal (single column addition)
- Typical execution (one machine): under 1ms for 1000 laps
Session loading adds
LapTimeSeconds itself. This function runs in the fastest-lap lookups: Laps.pick_fastest(), Driver.get_fastest_lap() and Session.get_fastest_laps()._rename_columns
df: DataFrame to rename (pandas or polars)rename_map(dict): Mapping of old column names to new names. UseNoneas the value to drop a column.lib(str): Backend library name ("pandas"or"polars")
- DataFrame with renamed columns. Columns mapped to
Noneare dropped.
- Duplicate prevention: Skips renames that would create duplicate column names
- Drop columns: Columns mapped to
Noneare removed from the DataFrame - No-op renames: Skips renames where source == target (for example,
{"Driver": "Driver"}) - Existing columns: Avoids conflicts when target name already exists independently
- Time complexity: O(n) where n is the number of columns
- Memory overhead: Minimal (column metadata only, no data copy)
- Typical execution: <1ms for typical DataFrames
_apply_categorical
df: DataFrame to modify (pandas or polars)cols(list): List of column names to convert to categoricallib(str): Backend library name ("pandas"or"polars")
- DataFrame with specified columns converted to categorical dtype
- Memory savings: 50-90% reduction for columns with low cardinality
- Faster operations: Comparisons and groupby operations are faster
- Preserved semantics: String operations still work as expected
- Time complexity: O(n × m) where n is rows and m is number of categorical columns
- Memory savings: About 90% for the Driver, Team, Compound and TrackStatus columns of a race laps frame
- Typical execution (one machine): about 1ms for 1000 laps with 4 categorical columns
On the pandas backend, session loading applies the categorical dtype to the
Driver, Team, Compound and TrackStatus columns. On the polars backend, these columns stay String unless the polars_lap_categorical option is True.Constants
The constants module (constants.py) defines all column name mappings, configuration values, and standard column orders used throughout the library. These constants ensure consistency across the codebase and provide FastF1 compatibility.
Year Range Constants
F1_API_FIRST_YEAR (2018, the earliest TracingInsights data repository). The upper bound is 2100, allowing for future scheduled races.
Example Usage:
Cache Configuration
LRUCache and the size of the two process-global lap caches (pandas and polars). Each lap cache keeps at most 100 lap DataFrames, which bounds memory growth.
Column Rename Maps
The library defines complete column rename mappings to transform CDN data format to user-facing FastF1-compatible format. These mappings handle both the long key names of validated payloads and the short keys of raw CDN payloads.Lap Data Rename Map
Telemetry Rename Map
Race Control Messages Rename Map
Weather Data Rename Map
Categorical Columns
str):
- Driver: 89% reduction (20 unique values)
- Team: 93% reduction (10 unique values)
- Compound: 92% reduction (3 unique values)
- TrackStatus: 88% reduction (6 unique values)
Standard Column Names
FastF1 Column Order
- Identification columns (index, driver, time)
- Core lap data (lap time, number, stint)
- Detailed timing (sectors, speed traps)
- Metadata (tyres, position, flags)
- Weather data
- tif1-specific additions
Resource Manager
The resource manager module (resource_manager.py) provides a pattern for managing resources with guaranteed cleanup. The cleanup runs even when initialization fails partway through. This is critical because it prevents resource leaks in error scenarios.
Architecture
TheResourceManager class implements the context manager protocol and tracks resources in a LIFO (Last-In-First-Out) stack. When cleanup occurs, resources are cleaned up in reverse order of creation, ensuring that dependencies are respected.
Key Features:
- LIFO cleanup order: Resources cleaned up in reverse order of registration
- Error resilience: Cleanup continues even if individual resources fail to close
- Complete logging: All cleanup operations are logged for debugging
- Multiple cleanup methods: Supports both
.close()and.shutdown()methods
ResourceManager
Methods
_register_resource
name(str): Human-readable name for the resource (used in logging). Should be descriptive for debugging purposes.resource(Any): The resource object to track. Should have a.close()or.shutdown()method for cleanup.
_cleanup_resources
- Iterate through resources in reverse order
- For each resource, try
.close()method first - If no
.close(), try.shutdown(wait=True)method - If neither exists, skip the resource
- Log any errors but continue cleanup
- Clear the resource list after all cleanup work finishes
Usage Patterns
Basic Context Manager Usage
Error Handling During Initialization
Manual Cleanup
Logging
The ResourceManager provides detailed logging for debugging. This example uses theSessionManager class from the context manager example above:
Error Resilience
If cleanup fails for one resource, other resources are still cleaned up:tif1 does not use
ResourceManager internally. Its HTTP session and cache manage their own cleanup. The class is available as a base class for custom resource management.Performance Considerations
JSON Parsing
The library uses orjson for JSON parsing, which provides:- 2-3x faster parsing than stdlib json
- Native support for bytes input
- NumPy values need conversion to Python types before
json_dumps
Lib Conversion
When converting between backends:- pandas → polars: Uses PyArrow for zero-copy when possible
- polars → pandas: Uses PyArrow by default for efficiency
- Rechunking: Optional for polars to optimize memory layout
Advanced Usage
Custom lib conversion
Custom JSON Processing
Best Practices
- Use orjson for JSON: Always use
json_loads/json_dumpsfor performance - Prefer PyArrow conversion: Keep
use_pyarrow=Truefor lib conversion - Validate early: Use validation helpers to catch errors early
- Register resources early: In a
ResourceManagersubclass, register each resource as soon as it exists - Use constants for column names: Reference standard column names from constants
Summary
The core_utils package provides:- High-performance JSON parsing with orjson
- Efficient lib conversion (pandas ↔ polars)
- Data validation utilities
- Column name standardization
- Resource management
- Internal helpers for DataFrame operations