perf: improve membership check performance in column filtering (#61046)

allrob23 · pre-commit-ci[bot] · web-flow · commit 44c56135688e · 2025-04-26T11:10:39.000-07:00
* perf: improve membership check performance in column filtering * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
diff --git a/pandas/io/parsers/c_parser_wrapper.py b/pandas/io/parsers/c_parser_wrapper.py
@@ -258,8 +258,9 @@ def read(
                 )
 
                 columns = _filter_usecols(self.usecols, columns)
+                columns_set = set(columns)
 
-                col_dict = {k: v for k, v in col_dict.items() if k in columns}
+                col_dict = {k: v for k, v in col_dict.items() if k in columns_set}
 
                 return index, columns, col_dict
 

Original file line number	Diff line number	Diff line change
`@@ -258,8 +258,9 @@ def read(`
`258`	`258`	`)`
`259`	`259`
`260`	`260`	`columns = _filter_usecols(self.usecols, columns)`
	`261`	`+ columns_set = set(columns)`
`261`	`262`
`262`		`- col_dict = {k: v for k, v in col_dict.items() if k in columns}`
	`263`	`+ col_dict = {k: v for k, v in col_dict.items() if k in columns_set}`
`263`	`264`
`264`	`265`	`return index, columns, col_dict`
`265`	`266`