I have about 100MB of CSV data that is cleaned and used for training in Keras stored as Panda DataFrame. What is a good (simple) way of saving it for fast reads? I don't need to query or load part of it.
Some options appear to be:
Topic serialisation keras csv dataset python
Category Data Science
You can find a nice benchmark for every approach in here.
Your data size is not that much huge, but there are some debates whenever you deal with big data What is the best way to store data in Python and Optimized I/O operations in Python. They all depend on the way the serialisation occurs and the policies which are taken in different layers. For instance, security, valid transactions and such things. I guess the latter link can help you dealing with large data.
With 100MB data, you can store it in any filesystem as CSV since read is going to take less than a second.
Most of the time is going to be spent by dataframe runtime in parsing data and creation of in-memory data structures.
Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.