Are we insane? 4 Day Sri Lanka Itinerary

My friend and I (21F, 22M) are planning on taking advantage of a public holiday weekend next month and are looking into travelling from Singapore to Sri Lanka. We have come up with a rough itinerary, but are really struggling with deciding if this is a worthwhile trip in such a short time-frame.

Day 1 (Thur): Arrive in Colombo at 11:15 AM

Spend the day in Colombo

Take a 12:30AM - 6 AM night bus from Colombo to Ella

Day 2 (Fri): Day in Ella

See the Nine Arch Bridge

Little Adams Peak

Explore Ella and the Area

Sleep in Ella

Day 3 (Sat): Transit to Dambulla

Seems like we can take a 6hr journey from 6 am - 12 pm with a transfer in Kandy by public transport

Sigiriya and Pidurangala Rock

Explore Dambulla in evening

Day 4 (Sun)

Early morning Dambulla Cave Temple

Minneriya National Park Safari

Private Transfer to Colombo Airport for 1:50 AM flight (4hr30m Journey)

Is this too packed? Is it even worth it for us to go to Ella since it adds that huge transfer on public transport? Should we just go Colombo --> Kandy --> Dambulla --> Airport? Adams Peak instead of Ella? Our flights are refundable for 24hrs and we are happy to look into other destinations if this isn't feasible.