pyspark.pandas.read_spark_io#

pyspark.pandas.read_spark_io(path=None, format=None, schema=None, index_col=None, **options)[source]#

Load a DataFrame from a Spark data source.

Parameters
pathstring, optional

Path to the data source.

formatstring, optional

Specifies the output data source format. Some common ones are:

  • ‘delta’

  • ‘parquet’

  • ‘orc’

  • ‘json’

  • ‘csv’

schemastring or StructType, optional

Input schema. If none, Spark tries to infer the schema automatically. The schema can either be a Spark StructType, or a DDL-formatted string like col0 INT, col1 DOUBLE.

index_colstr or list of str, optional, default: None

Index column of table in Spark.

optionsdict

All other options passed directly into Spark’s data source.

See also

DataFrame.read_table
DataFrame.read_delta
DataFrame.read_parquet

Examples

>>> ps.range(1).spark.to_spark_io('%s/read_spark_io/data.parquet' % path)
>>> ps.read_spark_io(
...     '%s/read_spark_io/data.parquet' % path, format='parquet', schema='id long')
   id
0   0

>>> ps.range(10, 15, num_partitions=1).spark.to_spark_io('%s/read_spark_io/data.json' % path,
...                                                format='json', lineSep='__')
>>> ps.read_spark_io(
...     '%s/read_spark_io/data.json' % path, format='json', schema='id long', lineSep='__')
   id
0  10
1  11
2  12
3  13
4  14

You can preserve the index in the roundtrip as below.

>>> ps.range(10, 15, num_partitions=1).spark.to_spark_io('%s/read_spark_io/data.orc' % path,
...                                                format='orc', index_col="index")
>>> ps.read_spark_io(
...     path=r'%s/read_spark_io/data.orc' % path, format="orc", index_col="index")
... 
       id
index
0      10
1      11
2      12
3      13
4      14