pyspark.pandas.DataFrame.reindex

DataFrame.reindex(labels=None, index=None, columns=None, axis=None, copy=True, fill_value=None)

Conform DataFrame to new index with optional filling logic, placing NA/NaN in locations having no value in the previous index. A new object is produced unless the new index is equivalent to the current one and copy=False.

Parameters

labels: array-like, optional

New labels / index to conform the axis specified by ‘axis’ to.

index, columns: array-like, optional

New labels / index to conform to, should be specified using keywords. Preferably an Index object to avoid duplicating data

axis: int or str, optional

Axis to target. Can be either the axis name (‘index’, ‘columns’) or number (0, 1).

copybool, default True

Return a new object, even if the passed indexes are the same.

fill_valuescalar, default np.NaN

Value to use for missing values. Defaults to NaN, but can be any “compatible” value.

Returns

DataFrame with changed index.

See also

DataFrame.set_index

Set row labels.

DataFrame.reset_index

Remove row labels or move them to new columns.

Examples

DataFrame.reindex supports two calling conventions

• (index=index_labels, columns=column_labels, ...)

• (labels, axis={'index', 'columns'}, ...)

We highly recommend using keyword arguments to clarify your intent.

Create a dataframe with some fictional data.

>>> index = ['Firefox', 'Chrome', 'Safari', 'IE10', 'Konqueror']
>>> df = ps.DataFrame({
...      'http_status': [200, 200, 404, 404, 301],
...      'response_time': [0.04, 0.02, 0.07, 0.08, 1.0]},
...       index=index,
...       columns=['http_status', 'response_time'])
>>> df
           http_status  response_time
Firefox            200           0.04
Chrome             200           0.02
Safari             404           0.07
IE10               404           0.08
Konqueror          301           1.00

Create a new index and reindex the dataframe. By default values in the new index that do not have corresponding records in the dataframe are assigned NaN.

>>> new_index= ['Safari', 'Iceweasel', 'Comodo Dragon', 'IE10',
...             'Chrome']
>>> df.reindex(new_index).sort_index()
               http_status  response_time
Chrome               200.0           0.02
Comodo Dragon          NaN            NaN
IE10                 404.0           0.08
Iceweasel              NaN            NaN
Safari               404.0           0.07

We can fill in the missing values by passing a value to the keyword fill_value.

>>> df.reindex(new_index, fill_value=0, copy=False).sort_index()
               http_status  response_time
Chrome                 200           0.02
Comodo Dragon            0           0.00
IE10                   404           0.08
Iceweasel                0           0.00
Safari                 404           0.07

We can also reindex the columns.

>>> df.reindex(columns=['http_status', 'user_agent']).sort_index()
           http_status  user_agent
Chrome             200         NaN
Firefox            200         NaN
IE10               404         NaN
Konqueror          301         NaN
Safari             404         NaN

Or we can use “axis-style” keyword arguments

>>> df.reindex(['http_status', 'user_agent'], axis="columns").sort_index()
           http_status  user_agent
Chrome             200         NaN
Firefox            200         NaN
IE10               404         NaN
Konqueror          301         NaN
Safari             404         NaN

To further illustrate the filling functionality in reindex, we will create a dataframe with a monotonically increasing index (for example, a sequence of dates).

>>> date_index = pd.date_range('1/1/2010', periods=6, freq='D')
>>> df2 = ps.DataFrame({"prices": [100, 101, np.nan, 100, 89, 88]},
...                    index=date_index)
>>> df2.sort_index()
            prices
2010-01-01   100.0
2010-01-02   101.0
2010-01-03     NaN
2010-01-04   100.0
2010-01-05    89.0
2010-01-06    88.0

Suppose we decide to expand the dataframe to cover a wider date range.

>>> date_index2 = pd.date_range('12/29/2009', periods=10, freq='D')
>>> df2.reindex(date_index2).sort_index()
            prices
2009-12-29     NaN
2009-12-30     NaN
2009-12-31     NaN
2010-01-01   100.0
2010-01-02   101.0
2010-01-03     NaN
2010-01-04   100.0
2010-01-05    89.0
2010-01-06    88.0
2010-01-07     NaN