perf: improve membership check performance in column filtering

[allrob23]

• closes PERF: Optimize membership check in column filtering for better performance #61045
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[allrob23]

pre-commit.ci autofix

[mroeschke]

Do you have an example benchmark where this PR improves the performance of read_csv?

[allrob23]

This optimization was flagged by a tool I’m developing, which performs code inspection to identify potential performance improvements. However, it doesn’t measure execution times, so I haven’t benchmarked the actual impact yet.

From a theoretical perspective, the change makes sense since the previous implementation performed lookups in a list (O(n)) while the new approach uses a set (O(1)).

Would you be able to help me create a proper benchmark for this case?

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.

[allrob23]

I'm busy with other things right now, but I'll try to create a benchmark as soon as possible.

[allrob23]

Hello @mroeschke , sorry for the delay, I managed to do the benchmark and would like your opinion and review.

Benchmark Explanation

This benchmark script measures the performance of pandas.read_csv with usecols filtering after a code modification that optimizes column selection using a set lookup.

Since the change affects only specific code paths when usecols is provided, the benchmark was designed to:

1. Vary the number of columns (10 to 50,000) to simulate different dataset sizes.
2. Select different percentages of columns (1%, 10%, 50%, 90%) to trigger and measure the filtering logic.
3. Run multiple iterations (5 by default, easily adjustable) to collect stable mean and median read times.

Benachmark results

This is the results in my branch:

 
 Number of Columns Select Percentage  Mean Time (s)  Median Time (s)
                10                1%         0.0008           0.0008
                10               10%         0.0008           0.0008
                10               50%         0.0013           0.0013
                10               90%         0.0018           0.0018
               100                1%         0.0009           0.0009
               100               10%         0.0019           0.0019
               100               50%         0.0062           0.0063
               100               90%         0.0106           0.0106
              1000                1%         0.0025           0.0025
              1000               10%         0.0155           0.0118
              1000               50%         0.0533           0.0527
              1000               90%         0.0958           0.0935
             10000                1%         0.0176           0.0173
             10000               10%         0.1173           0.1120
             10000               50%         0.5866           0.5692
             10000               90%         1.0830           1.0885
             50000                1%         0.0913           0.0907
             50000               10%         0.6784           0.7101
             50000               50%         3.3062           3.2970
             50000               90%         5.9495           5.8902

this is the results in Main branch

 Number of Columns Select Percentage  Mean Time (s)  Median Time (s)
                10                1%         0.0009           0.0009
                10               10%         0.0008           0.0008
                10               50%         0.0013           0.0013
                10               90%         0.0017           0.0017
               100                1%         0.0009           0.0009
               100               10%         0.0020           0.0020
               100               50%         0.0065           0.0064
               100               90%         0.0110           0.0110
              1000                1%         0.0024           0.0024
              1000               10%         0.0160           0.0118
              1000               50%         0.0548           0.0547
              1000               90%         0.1046           0.1014
             10000                1%         0.0181           0.0174
             10000               10%         0.1259           0.1203
             10000               50%         0.8275           0.8285
             10000               90%         1.8634           1.8587
             50000                1%         0.0924           0.0917
             50000               10%         0.9305           0.9910
             50000               50%         9.2642           9.2397
             50000               90%        24.1806          24.3157

The results show a performance improvement with modification, especially as the number of columns and selected percentage increase.

In small datasets (10–100 columns), performance is similar between the branches, as expected the overhead of column filtering is minimal.

However, as the number of columns grows (1000, 10000, 50000), the modification consistently outperforms the main branch, especially for larger filter percentage values (50%, 90%).

For example:

• With 10,000 columns and 90% selected, the mean time dropped from 1.86s (main) to 1.08s (new).
• With 50,000 columns and 90% selected, the mean time dropped from 24.18s (main) to 5.94s (new).

How I ran the benchmark

1. Synchronize branches

git remote add allrob23-fork https://github.com/allrob23/pandas.git
git fetch allrob23-fork

2. Create bench.py script with code

get the code from this gist https://gist.github.com/allrob23/dea86dba17bbfaa988e2b5a08b27db79

3. Run the code

python3 bench.py

4. Get the results printed at the end

5. Change to this PR branch

git checkout perf-columns-set

6. Run the code again

python3 bench.py

7. Get the results printed at the end

I ran this using docker + compile pandas with reasons:

# git
git clone https://github.com/pandas-dev/pandas/
cd pandas

# docker
docker build -t pandas-dev .
docker run -it --rm -v ${PWD}:/home/pandas pandas-dev

# compile with reasons
python -m pip install -ve . --no-build-isolation -Ceditable-verbose=true

[mroeschke]

Thanks @allrob23