Intermittent Sample Glitching

Hello. New M4 Pro owner. I am having an issue with my sample libraries (Embertone Walker 1955 piano in Kontakt, Keyscape) occasionally glitching in a way that when I play my Nord keyboard, for a split second the sample is delayed, like the sample is trying to catch up. This only happens sometimes. No CPU spikes and I have tried increasing my DAW buffer. This only seems to happen with sample-based instruments, as I have not had issues with any of my modeled instruments. I have these sample libraries installed on the internal SSD on my MacBook Pro. On my previous Intel computer, I had my sample libraries installed on an external SSD and did not have any issues with performance. I had emailed recently with Embertone and Spectrasonics to see if I needed an external drive with my new silicon chip setup, and both companies thought I should be able to run everything fine with samples just on my internal hard drive. I'm wondering if this is a sample streaming issue, perhaps something to do with preload memory buffer size or something along those lines. I'm not really sure. Any help would be appreciated! I have probably left out some helpful info so please let me know what else I can provide. Thanks!