Happened on Halloween, 1924. Sez so in the Times. At an exhibition game organized by the Elks Club, on land owned by Union Oil. The Brea Museum and Heritage Center has signed pictures and a baseball to prove it.
Chris Epting, author of Roadside Baseball, has a video showing the pictures of Babe Ruth, Walter Johnson, and others at that game--Go to http://www.chrisepting.com/theBabeInBrea.htm. (look around the website--he's also got pictures of the Coliseum).
Apparently Johnson lived near Brea and attended Fullerton Union High School. Just before that exhibition game, he'd led his Washington Senators to victory in the World Series, at age 36. That game, btw, drew 15,000 spectators. Brea's population at the time was about 1500.