In terms of geographical distance, Koh Lipe is closer to Krabi than Phuket. However, when considering travel times, it often depends on the available transportation options.
From both Krabi and Phuket, you would typically have to travel to Trang or to Pak Bara Pier in Satun province to take a boat to Koh Lipe. The travel duration to get to Koh Lipe from either Krabi or Phuket can take a similar amount of time when considering all the modes of transportation.
In the end, it would be best to look at specific transportation options available during your travel dates, as the availability of direct ferries or combined bus + ferry tickets can vary depending on the time of the year.
Keep in mind that as of my knowledge cutoff in September 2021, travel circumstances may have changed. Always check the latest information.