The game started slowly as both teams took time to get accustomed to the changes in the line ups from their usual starting sides in league games.
Grays had the first chance of the game in the 18th minute as Joe Sweeney won the ball on the edge of the area but his shot was straight at Aaron Omand in the Forest goal.
On the half hour mark James Bunn played Dan Gilchrist, making his first team debut, through on goal and his fierce drive was well saved by a diving Omand to keep the score level.
George Kamurasi was called into action shortly after as he pushed a shot from the edge of the area over the bar. From the resulting corner the ball was headed in by Arian Retkoceri to give the away side the lead.
Waltham Forest looked to have doubled their lead on 40 minutes a free kick was swung to the back post where Ben Head rose to nod the ball into the net, but the assistant had flagged for offside, cancelling out the goal.
Grays came close to drawing level in the 53rd minute as a corner was played into the area which was not dealt with by the Forest defence. The ball bounced around the area before eventually being claimed by Omand.
Grays made a couple of substitutions as they replaced Dan Gilchrist and Luke Marshall with Leon Lalite and Dan Fry, also making his first team debut.
The home side were almost level again in the 71st minute as a cross from the right wing by Dan Fry found James Bunn in the area. His shot was into the ground before bouncing over Omand and onto the bar before being cleared to safety.
As Grays pushed on late to try and draw level, they were caught on the counter attack in the 80th minute as Corey Whitely was played in down the right before driving into the area and firing into the top right hand corner of the net.
Grays made their last substitution in the 83rd minute replacing James Bunn with Tyrone Scarlett.
He could only look on, along with everyone else as Matt Waldon stepped up to fire a 35 yard free kick into the top right hand corner of the net with just seven minutes left.
Grays could not create any further chances and will be disappointed to exit the League Cup at this early stage, although this is not the priority for them this season.
Grays Athletic: George Kamurasi, Craig Pope, Junior Baker, Grant Cooper, Danny Bunce, Jake Hall, Harry Agombar, Dan Gilchrist (Leon Lalite 59), Luke Marshall (Dan Fry 70), James Bunn (Tyrone Scarlett 83), Joe Sweeney
Unused Subs: Danny Swallow, John Nash