How to extract the same column from a large dictionary of dataframes efficiently ? (PerformanceWarning)

Below given is the code to compute a dictionary of data frame for the purpose of this example:

import pandas as pd
import numpy as np
import random
from datetime import datetime

# Generate the large dictionary of dataframes
dic = {}

index = pd.date_range(datetime.strptime('01/01/2021', '%d/%m/%Y'), periods=100).tolist()

for i in np.arange(0,1000):

    # Initialise data:
    data = {'col1':random.sample(range(0, 100), 100),
            'col2':random.sample(range(0, 100), 100),
            'col3':random.sample(range(0, 100), 100),
            'desired_col':random.sample(range(0, 100), 100),
            'coln':random.sample(range(0, 100), 100),
           }

    # Create DataFrame
    df = pd.DataFrame(index = index, data = data)
    
    dic['df'+str(i)] = df

–> I have now a dictionary containing df0, df1, df2, dfn… etc, with all unique data but with the same columns’ name and index.

My objective is to filter out / extract the same column from all these dfs and save it into a final data frame.

My temporary solution would be to go for a for loop arround the keys:

# Generate the new df's index
df_filtered = pd.DataFrame(index=dic[list(dic.keys())[0]].index)

for key in list(dic.keys()):
            
            # Extract the df of the dictionary
            df_i = dic[key].copy()
            
            # Save the relevant column into the final df
            df_filtered[key] = df_i[['desired_col']].copy()

But I get a PerformanceWarning: PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling frame.insert many times, which has poor performance. Consider using pd.concat instead.

I am sure there would be a more elegant way to accomplish the same task.

Many thanks in advance.

Answer

You can use .concat + .unstack:

df = pd.concat(dic)
print(df["desired_col"].unstack(level=0))

Prints:

            df0  df1  df2  df3  df4  df5  df6  df7  df8  df9  df10  df11  df12  df13  df14  df15  df16  df17  df18  df19  df20  df21  df22  df23  df24  df25  df26  df27  df28  df29  df30  df31  df32  df33  df34  df35  df36  df37  df38  df39  df40  df41  df42  df43  df44  df45  df46  df47  df48  df49  df50  df51  df52  df53  df54  df55  df56  df57  df58  df59  df60  df61  df62  df63  df64  df65  df66  df67  df68  df69  df70  df71  df72  df73  df74  df75  df76  df77  df78  df79  df80  df81  df82  df83  df84  df85  df86  df87  df88  df89  df90  df91  df92  df93  df94  df95  df96  df97  df98  df99  df100  df101  df102  df103  df104  df105  df106  df107  df108  df109  df110  df111  df112  df113  df114  df115  df116  df117  df118  df119  df120  df121  df122  df123  df124  df125  df126  df127  df128  df129  df130  df131  df132  df133  df134  df135  df136  df137  df138  df139  df140  df141  df142  df143  df144  df145  df146  df147  df148  df149  df150  df151  df152  df153  df154  df155  df156  df157  df158  df159  df160  df161  df162  df163  df164  df165  df166  df167  df168  df169  df170  df171  df172  df173  df174  df175  df176  df177  df178  df179  df180  df181  df182  df183  df184  df185  df186  df187  df188  df189  df190  df191  df192  df193  df194  df195  df196  df197  df198  df199  df200  df201  df202  df203  df204  df205  df206  df207  df208  df209  df210  df211  df212  df213  df214  df215  df216  df217  df218  df219  df220  df221  df222  df223  df224  df225  df226  df227  df228  df229  df230  df231  df232  df233  df234  df235  df236  df237  df238  df239  df240  df241  df242  df243  df244  df245  df246  df247  df248  df249  df250  df251  df252  df253  df254  df255  df256  df257  df258  df259  df260  df261  df262  df263  df264  df265  df266  df267  df268  df269  df270  df271  df272  df273  df274  df275  df276  df277  df278  df279  df280  df281  df282  df283  df284  df285  df286  df287  df288  df289  df290  df291  df292  df293  df294  df295  df296  df297  df298  df299  df300  df301  df302  df303  df304  df305  df306  df307  df308  df309  df310  df311  df312  df313  df314  df315  df316  df317  df318  df319  df320  df321  df322  df323  df324  df325  df326  df327  df328  df329  df330  df331  df332  df333  df334  df335  df336  df337  df338  df339  df340  df341  df342  df343  df344  df345  df346  df347  df348  df349  df350  df351  df352  df353  df354  df355  df356  df357  df358  df359  df360  df361  df362  df363  df364  df365  df366  df367  df368  df369  df370  df371  df372  df373  df374  df375  df376  df377  df378  df379  df380  df381  df382  df383  df384  df385  df386  df387  df388  df389  df390  df391  df392  df393  df394  df395  df396  df397  df398  df399  df400  df401  df402  df403  df404  df405  df406  df407  df408  df409  df410  df411  df412  df413  df414  df415  df416  df417  df418  df419  df420  df421  df422  df423  df424  df425  df426  df427  df428  df429  df430  df431  df432  df433  df434  df435  df436  df437  df438  df439  df440  df441  df442  df443  df444  df445  df446  df447  df448  df449  df450  df451  df452  df453  df454  df455  df456  df457  df458  df459  df460  df461  df462  df463  df464  df465  df466  df467  df468  df469  df470  df471  df472  df473  df474  df475  df476  df477  df478  df479  df480  df481  df482  df483  df484  df485  df486  df487  df488  df489  df490  df491  df492  df493  df494  df495  df496  df497  df498  df499  df500  df501  df502  df503  df504  df505  df506  df507  df508  df509  df510  df511  df512  df513  df514  df515  df516  df517  df518  df519  df520  df521  df522  df523  df524  df525  df526  df527  df528  df529  df530  df531  df532  df533  df534  df535  df536  df537  df538  df539  df540  df541  df542  df543  df544  df545  df546  df547  df548  df549  df550  df551  df552  df553  df554  df555  df556  df557  df558  df559  df560  df561  df562  df563  df564  df565  df566  df567  df568  df569  df570  df571  df572  df573  df574  df575  df576  df577  df578  df579  df580  df581  df582  df583  df584  df585  df586  df587  df588  df589  df590  df591  df592  df593  df594  df595  df596  df597  df598  df599  df600  df601  df602  df603  df604  df605  df606  df607  df608  df609  df610  df611  df612  df613  df614  df615  df616  df617  df618  df619  df620  df621  df622  df623  df624  df625  df626  df627  df628  df629  df630  df631  df632  df633  df634  df635  df636  df637  df638  df639  df640  df641  df642  df643  df644  df645  df646  df647  df648  df649  df650  df651  df652  df653  df654  df655  df656  df657  df658  df659  df660  df661  df662  df663  df664  df665  df666  df667  df668  df669  df670  df671  df672  df673  df674  df675  df676  df677  df678  df679  df680  df681  df682  df683  df684  df685  df686  df687  df688  df689  df690  df691  df692  df693  df694  df695  df696  df697  df698  df699  df700  df701  df702  df703  df704  df705  df706  df707  df708  df709  df710  df711  df712  df713  df714  df715  df716  df717  df718  df719  df720  df721  df722  df723  df724  df725  df726  df727  df728  df729  df730  df731  df732  df733  df734  df735  df736  df737  df738  df739  df740  df741  df742  df743  df744  df745  df746  df747  df748  df749  df750  df751  df752  df753  df754  df755  df756  df757  df758  df759  df760  df761  df762  df763  df764  df765  df766  df767  df768  df769  df770  df771  df772  df773  df774  df775  df776  df777  df778  df779  df780  df781  df782  df783  df784  df785  df786  df787  df788  df789  df790  df791  df792  df793  df794  df795  df796  df797  df798  df799  df800  df801  df802  df803  df804  df805  df806  df807  df808  df809  df810  df811  df812  df813  df814  df815  df816  df817  df818  df819  df820  df821  df822  df823  df824  df825  df826  df827  df828  df829  df830  df831  df832  df833  df834  df835  df836  df837  df838  df839  df840  df841  df842  df843  df844  df845  df846  df847  df848  df849  df850  df851  df852  df853  df854  df855  df856  df857  df858  df859  df860  df861  df862  df863  df864  df865  df866  df867  df868  df869  df870  df871  df872  df873  df874  df875  df876  df877  df878  df879  df880  df881  df882  df883  df884  df885  df886  df887  df888  df889  df890  df891  df892  df893  df894  df895  df896  df897  df898  df899  df900  df901  df902  df903  df904  df905  df906  df907  df908  df909  df910  df911  df912  df913  df914  df915  df916  df917  df918  df919  df920  df921  df922  df923  df924  df925  df926  df927  df928  df929  df930  df931  df932  df933  df934  df935  df936  df937  df938  df939  df940  df941  df942  df943  df944  df945  df946  df947  df948  df949  df950  df951  df952  df953  df954  df955  df956  df957  df958  df959  df960  df961  df962  df963  df964  df965  df966  df967  df968  df969  df970  df971  df972  df973  df974  df975  df976  df977  df978  df979  df980  df981  df982  df983  df984  df985  df986  df987  df988  df989  df990  df991  df992  df993  df994  df995  df996  df997  df998  df999
2021-01-01   82   26   60    7   21   85   78   98   67   45    24    53    29    61    47    86    89    55    95    95    26    47    57     6    44    81    98    48    48    65    91    11    54    58    74    88    62    55    73    36    67    81    46    36    18    21    65    32    33    61     0    57    81    71    42    98    82     9    16    54    91    41    80    83    24    52    65    86     5    67    17    86    56    36    44    88    34    14     3    63    11    36    54    44    88    98    85    15    95    35    31     1     9    94    35    63     0    86    13    22     73     80     79     90     26     79     79     46      3     97     19     35     11     20     84     16     83     11     93     26     43     49     18     38     12     41     35     12     73     11     17     78     45     29     13     15     29     52     70     58     30     29      2     81     32      5     84     91     28     81     85     77     59     11     61     45     11     97     36     10     77     84     26     93     56     60     12     36     47     42     73     65     47     79     86     51     32     77     99     58     99     19     38     88     40     49     66     82     13     97     23     65      4     75     63     18     35     48      1      6     37     82     55     96     93     93      5     67     42     51     57     75     72     59     55     40      5     93     35     62     67     29     41     39     54     96     98     75     96      1     55     27     51     16     88     28     78     31     24     54     88     27     67     94     34     83      4     30     92     85      4     30     24     76     60      9     86     38     87     31     40     15     13     99     51      3     94     98     24      6     11     85     57     32     41      3     56     31     24     17     33     99     32     39     22     42     40     30     35     61      3     96     25     35     83     77     79     64     79     60     36     57     97      6     42     11     55     97     25      2     28     86     30     84     24     72     54     86     61     35     65     65     38     67     58     39     39     81     43     63     91     68      7     35     27     21     60     95     79     16     71     84     99     25     97     42     39     64      8     40     19     31      6     84     69     49     48     15     78     46     28     66     97     81     97     26     29     60     96     52     90      9     48     39     74     52     76     97     78      8     96      6     60     34     11     88     66      8     57     66     11     35     64     59     25     53     56     90     60     35     29     81     74     18     39     23     82     75     28     23     34     18     78     75     30     25     41     62     66      9      6     63     79     21     67     22     78     90     59     11     76     85     50     45     25     64     84     98     11     12     62     41     90     89     11     18     71      4     31     56     96     48     31     13     12     54     70     52     91      3      3      5     95     24     16     81     88     10     97     71     80     58     19     62     79     31     42     44     73     21     63     56     15     15     79     31      8     43      5     55     28     23     23     32     74     85     44     88     96     39     77     41     44     64     26     60     56     53     31     30     35     24     32     50     49     35     89     79     87     38     42     64     21     62     56     91     47     78     23     75     89     54     66     38     60     88     25     28     83     74     11     87     11     78     25     41     46     58     59     39     39     67     42     52     27      2     20     21     17     72     59     92     66     32      0     39     81     93     17     82     12     20     64     67     93     78     72     80     53     14     74     78      9     76      5     29     57     68     67     40     20     74      0     13     70      2     41     41      5     29     83     10     43     19     55     10     66     68     92     64     16     59     36     70     14     62     97     28     66     84     69     70     76     43     32     33      0     52     62     88     38     58      5     10     56     16     19     70     45      7     50     15     44     29     37      5     50     84      1     90     48     25     41      4     50     47     77     43     42     31     32     55     24     13     52     67     44     87     22     53     11     80     89     45     86     38     90      9     88     96     48     59     37     22     59     76     99     54     98     30     99     96     77     94     46     31     25     25     99     74     72     37     95     14     58     74     82     34     52     28     19     53     55     28     62     60     98     53     38      1     24     87     84     12     56     83     87     20      7     85     11     46     37     47     99     61     49      0     17     90     47     50     13     32     57     87     27     51     42      6     82     75     78     61     49     55     36     47     59     77     20     62     10     45     58     50     70     24     80     67     43     65     32     94     45     27     17     27     29     21     97      5     15     80     95     58      0      5     90     28     96      3     38     48     93     51     44     70     85     85     72     35     25     69     25     54     58     38     58     51      7     30     65     13     57     35      9     73     80      7     66     78     46     46      0     41     41     29     72     34     61     52     84     25     86     92     13     56     67     44     59     18     29     52     54     54     76     67     98     53     83     63     14     93     20     54     35     18     69     52     31     54     83     93     92     41     75     34     45     95     27     94     22      1     56     90     80     58     39     16     73     64     15     53     96     91     37     65     13     86      0     89     42     93      7     11     22     16     14     65     61     39     20     96     84     73     57     64      2     69     51     30     93     70     70     11     13     23     12     93     73     82     42     37     79     45     36     11     21     23     81     46     95     55     51     32     37      4      8     92     13     39     56     94     57     81     55     90     54     94     25      5     61      7     91     47     23     80     42     33     26     18     63     70     87     99     27     46     65     74     10     72      8     32     23     86     88     86      6     97     29     33     50     41     40      1     13     53     53      3     15     45     29     83     13     42     55     34     50     39
2021-01-02   46   77   94   40   15   39   79   69   94   11    60    39     1    57    50    81    23    52    72     6    37    10    32    76    92    53     9    20    46    87    36    64    94    25    32    35    17    71    89    28    47    21    85    71    72    69    49    46    80    20    95    61    68    89    93    41    86    54    81    29    79    77    22    52    97     9    63    60    66    47    60    49     1    48    76    32    52    51    30    69    16    54    15    89    10    29    96    16    52    66    49    87    97    82    60    56    93    80    69    52     40     33      1     63     82     88     91     55     61     59     68     77     12     93     47     70     24     76      3      7     32      8     75     81     58     83     49     75     58     22     32     53     30     71     35     28     17      3     95     17     32     38     80     39     24     76     99     46     82     42     68      7     19     80     52     10      8     82     34     13     39     66      8      9     28     78     98     12     58     69     52     96     54     10     55     30     96     74     16     74     29     61     10      8     99     86     64      1     26     38     38     38     67     33     15      0     39     99     84      8     17     78     99      4     27     27     97     77      0     98     72     76     32      2     98     93      3     61     43     41     37      1     77     12     49     47     46     29     83     14     54     35     95     39     23     41     64     27     62     89     54     54     76      8     21     13     24     62     48     76     18     64     22     36     87     25     65     34     89     97     86     35     78     44     95     90     43     93     51     19     29     10     17     97     80     33     97     20     76     62     52     76     46     45     99      6     86     11     59     77     10     26     90     71      9     28     52     57      4     81     81     76     51     75     52     33     86     17     10     55     44     20     97     28     11      6     15     35     35     97     79      4     95     36     71      5      0     79     39     32     37     73     89      5     16     28     22     61     36     44     85     91     84     10     85     66     34      2     80     68     52     43     17     40     20     45     85     75     27     92     60     87     76     74     19     61     97     50     11     90     54     25     21     32     48     38     29     16     16     31     31     61     78     35     12     78     84     39     86     40     55     93     40     42     65     65     41     80     90     99     51     52     11     34      0     50     49     69     19     62     51     37     42     88     64     24     74     63     60     75     63     57     80     60     26     84     33     96      8     46      1     35     83     60     45      9     69     67     94     75      1     82     15     64      2      5      2     59     68     86     89     73     74     38     85     56     43      6     95     42     51     73      4     97     91     23     13     92     60     63     56     56      8     70     31     63     87      3     55     76     46     69     32     79     72     80     66     93     26     82     22     71     76     14      8     43     96     37     90     28     15     48     41      7      8     17     25     78     90      9      3      6     16     10     26     91     76     58     61      5     78     82     48     89     38     62     58     94     25     11     28     16     13     80     86      8     35     85     90     27     84     95     77     35     63     56     86     50      1     14     86      5     53     60     77     69     74      6     31     94     43     74     99     75     43     75     40     55      1     78     57     27     56     36     31     41     25     17     51     90     82     76     25     79     63      8     10      1     29     46     81     89      2      9     37     85     59     40     13     55     74     78     30     93     29     64     31     36     87     51     98     42     18     25     26     22     76     94     65     56     68      6     43     16     63      2     93     77     74      0     17     37     32     70     33     40     13     44     41     28      3     56     11     57     65     53     69     24     86     31     37     80     50     74     91      6     37     18     79     62     65      8     80     29     38     86     26     59     65     84     23     90     16     63     98     61     65     51     39     26      9     94     41     89     71     78     23     95     90     91     88     63     19      2     24     28     89     26     64     80     73     51     84     93     61     62     98     10     28     81      6     52     79     52     13     25     83     50     68     68     61      1     83     92     18     19     60     22     33     98     17     56      3     80     70     97     24     81      3     69      1     47     92     52     46     10     33     13     85     80     89     16     67     49     61     95     69     25     63     56     88     15      6     57     13     55     95     25     32     53     97     34     76     50     99     81     42     86     38     43     12     87     76     84     29     29     93     43     96     92      6     14     69     40     91     46     51     55     28     14     36     17      0     24      4     39     61     18     51     86     94     33     71     64     52     68     12     46     97     17     97     46     18     55     50     82     97      5     97     58     95     97     73     19     23     11     36     14     55     35      0     42     51     14     15      5      6     34     82     34     27     27     88     64     61     35     51      3     72     27     11     27     77     19     67     27     87     25     50     87     66     60     63     77     31     36     77      0     15     84     71     79     93     81     33     25     90     72     77      8     98     50     71     52     36     10     67     72     18     94     89      0     33     26     79     12     36     95     22     26     27     97     41     16      3     49      2     76     53     47     76     33     92     63     99     19     10     96      7     35     92     99     87      6     52      3      8     80     77     48     66     60      1      6     13     85      2      4     79     33     37     79     65     55     72     95      9     18     44     86     14     23     44     72     73     97     85     97     97     75     86     32     26      9     42     34     79     83     55     10     31     69     29      9      7     70     14     46      2     73     86     29     74      4     49     95     91     73     43     27
...