Are we really sure St. Paul persecuted Christians before his conversion?

Christ said to St. Paul, "Why do you persecute me?" St. Paul did not "literally" persecute Christ. Likewise, are we sure when we say St. Paul persecuted Christians before his conversion that we mean persecuted "literally" or does figuratively fit better since there are no descriptions of St. Paul physically persecuting Christians?