What is the downside to prefetching

Accuracy of the algorithms used in prefetching, as they can only provide an informed prediction Larger caches still take a long time to search and so cache size limits how much data can be stored