Which storage level to choose

Cache judiciously

Just because you can cache a RDD in memory doesn’t mean you should blindly do so. Depending on how many times the dataset is accessed and the amount of work involved in doing so, recomputation can be faster than the price paid by the increased memory pressure.

It should go without saying that if you only read a dataset once there is no point in caching it, i it will actually make your job slower. The size of cached datasets can be seen from the Spark Shell.

Which storage level to choose

By dafault Spark will cache() data using MEMORY_ONLY level,
MEMORY_AND_DISK_SER can help cut down on GC and avoid expensive recomputations.


  • MEMORY_ONLY

    • Store RDD as deserialized Java objects in the JVM. If the RDD does not fit in memory, some partitions will not be cached and will be recomputed on the fly each time they're needed. This is the default level.
  • MEMORY_AND_DISK

    • Store RDD as deserialized Java objects in the JVM. If the RDD does not fit in memory, store the partitions that don't fit on disk, and read them from there when they're needed.
  • MEMORY_ONLY_SER

    • Store RDD as serialized Java objects (one byte array per partition). This is generally more space-efficient than deserialized objects, especially when using a fast serializer, but more CPU-intensive to read.
  • MEMORY_AND_DISK_SER

    • Similar to MEMORY_ONLY_SER, but spill partitions that don't fit in memory to disk instead of recomputing them on the fly each time they're needed.
  • DISK_ONLY

    • Store the RDD partitions only on disk.
  • MEMORY_ONLY_2, MEMORY_AND_DISK_2, etc.

    • Same as the levels above, but replicate each partition on two cluster nodes.
  • OFF_HEAP (experimental)

    • Store RDD in serialized format in Tachyon. Compared to MEMORY_ONLY_SER, OFF_HEAP reduces garbage collection overhead and allows executors to be smaller and to share a pool of memory, making it attractive in environments with large heaps or multiple concurrent applications. Furthermore, as the RDDs reside in Tachyon, the crash of an executor does not lead to losing the in-memory cache. In this mode, the memory in Tachyon is discardable. Thus, Tachyon does not attempt to reconstruct a block that it evicts from memory.

results matching ""

    No results matching ""