How to use Spark and Pandas to prepare big data

If you want to train machine learning models, you may need to prepare your data ahead of time. Data preparation can include cleaning your data, adding new columns, removing columns, combining columns, grouping rows, sorting rows, etc.

Once you write your data preparation code, there are a few ways to execute it:

1. Download the data onto your local computer and run a script to transform it
2. Download the data onto a server, upload a script, and run the script on the remote server
3. Run some complex transformation on the data from a data warehouse using SQL-like language
4. Use a Spark job with some logic from your script to transform the data

We’ll be sharing how Mage uses option 4 to prepare data for machine learning models.

Read more here