Chasing 243, India made a rather slow start and lost both their openers inside 10 overs before one-down Virat Kohli (82) and Rohit (101 not out) resurrected the innings to help India romp home with 6.3 overs to spare at the Queens Sports Club.
Earlier, the tidy Indian bowlers produced a disciplined effort to bundle out Sri Lanka for 242 after captain Suresh Raina had opted to field.
Today's win also marked Raina's first victory as captain.
Coming together in the 10th over, Kohli and Rohit added 154 runs from 168 balls for the third wicket to take the game away from Sri Lanka.
Rohit, who hit 114 against Zimbabwe in India's first match though for a lost cause, hit six fours and two sixes in his unbeaten 100-ball, his second ODI century in his 44th match.
He got a superb support from Kohli whose 82 came from 92 balls with the help of four boundaries.
After opening bowler Nuwan Kulasekara and Ajantha Mendis achieved early success by removing Murali Vijay (14) and Dinesh Karthik (18) respectively, the Sri Lankan bowlers were left high and dry with Kohli and Rohit stamping authority.
The duo initially dealt mostly in singles with occasional boundaries before opening up later on. They also did not allow the asking rate to jump over five.
Rohit took 60 balls to reach his fifty which came of a single off Ajantha Mendis in the 28th over and Kohli reached his seventh half-century an over later with a single off Dilhara Fernando.
Kohli, however, perished in the 38th over while trying to finish the game early. He came down the track to play a lofted shot off a Suraj Randiv delivery only to hole out to long-on fielder Dilhara Fernando.
When Kohli fell, India needed just 42 runs from more than 12 overs and Rohit ensured that they wrapped up the match without any hiccup.
Rohit took a four off Mendis in the 43rd over to reach his second successive ODI century while Raina hit the winning runs in the next over.
India, who lost their opening match against Zimbabwe on Friday, play the hosts again in their next match on Thursday at Harare Sports Club.
Earlier, save for Angelo Mathews (75) and captain Tillakaratne Dilshan (61), the Lankan batsmen struggled for runs under tight Indian bowling after they were put into bat.
The Lankan innings eventually folded for 242 in 49.5 overs with three of their batsmen running themselves out.
Except for rookie Umesh Yadav (1/61), all the Indian bowlers bowled tight spells with Ashok Dinda (2/44), Pragyan Ojha (2/44) and Ravindra Jadeja (2/49) taking two wickets apiece.
Leg-spinner Amit Mishra was also economical, conceding 40 runs from his 10 overs though without taking a wicket.
The Lankans were dealt an early blow when Upul Tharanga (1) became the first to get run out.
After hitting an Ashok Dinda delivery to the covers, Tharanga was left stranded when Virat Kohli, who initially fumbled in collecting the ball, threw it accurately at the Lankan's end while he was attempting a third run.
Thilan Samaraweera (19) didn't last long either and was stumped by Dinesh Karthik while trying to charge at a Pragyan Ojha delivery.
Dilshan and Mathews then steadied the ship and put on 58 runs for the third wicket before the Sri Lankan skipper became the second Lankan batsmen to be run out.
Dilshan, whose 81-ball knock comprised four boundaries and a six, took off for a single after hitting Mishra to the square leg but rushed back after a couple of strides only to find himself short of ground as the bowler had dislodged the bails by then.
But Mathews held one end together with a gritty knock that included a couple of sixes and an equal number of fours. His prized wicket ultimately went to Umesh Yadav, who bowled him.
All-rounder Ravindra Jadeja bagged two late wickets in Chamara Kapugedera (20) and Chamara Silva (5) to make sure that the Lankan innings never took off.
Thissara Perera's late 32-run cameo down the order did, however, help the Lankan to go past the 200 mark.