We got there eventually and I think we got there in a way that still looks nice. There is now a DataFrame.Typed API that tracks the entire schema of the dataframe - column names, misapplied operations etc are now compile time failures and you can easily move between exploratory and pipeline work. This is in large part thanks to maxigit and mcoady (Github user names) for their feedback.

$(DT.deriveSchemaFromCsvFile "Housing" "./data/housing.csv")
    
main :: IO ()
main = do
    df <- D.readCsv "./data/housing.csv"
    let df' = either (error . show) id (DT.freezeWithError @Housing df)
    let df'' =
            df'
                & DT.derive @"rooms_per_household" (DT.col @"total_rooms" / DT.col @"households")
                & DT.impute @"total_bedrooms" 0
                & DT.derive @"bedrooms_per_household"
                    (DT.col @"total_bedrooms" / DT.col @"households")
                & DT.derive @"population_per_household"
                    (DT.col @"population" / DT.col @"households")

    print df''

There’s an implementation of Apache Arrow’s C Data interface along with an example of how to pass dataframes between polars and haskell.

You can explore huggingface datasets. Example:

df <- D.readParquet "hf://datasets/Rafmiggonpaz/spain_and_japan_economic_data/data/train-00000-of-00001.parquet"

Introduced more lenient operators that make happy path computation much easier. E.g:

D.derive "bmi" (F.lift2 (\m h -> (/) <$> m <*> fmap ((^2) . (/100). realToFrac) h) mass height) df

Is now instead:

D.derive "bmi" (mass ./ (height ./ 100) .^ 2) df

Connectors! BigQuery, Snowflake, s3 buckets etc. Formats! Parquet, Iceberg, DuckDB, a custom dataframe format with full data provenance. Moving from small in memory demos to querying large data lakes is the goal.

Also, since a new era is upon us, some integration with ai agents to do type-guided data exploration.