There is no set rule but your multiple border runs are definitely a red flag for immigration. You might pass through without issue once, but the next time, you could be questioned—that depends on the immigration officer. Your history doesn’t disappear; it just improves slightly since you have a visa. Make sure you are fulfilling your visa requirements in case they decide to question you (an ongoing job/activity is highly recommended in your case).