The Oldfields estate was built by businessman Hugh McKennan Landon as part of a real estate development that came to be called the Town of Woodstock. Landon began construction of his new residence in the early 1910s, when he was an executive of the Indianapolis Water Company. With the French chateau-style house as its centerpiece, the Oldfields landscape underwent further development in the 1920s according to designs by Percival Gallagher of Olmsted Brothers, a successor to the firm founded by Frederick Law Olmsted. Gallagher combined grand, formal elements such as the allée at the front of the house, with intimate and informal areas to create a variety of visual and emotional experiences.

J. K. Lilly Jr. and his wife Ruth Brinkmeyer Lilly purchased Oldfields from Landon in 1932 and hired Indianapolis architect Frederick Wallick to oversee changes to the house. Wallick enhanced the symmetry of the façade by reducing the number of doors and by moving the entrance from its original off-center position. Wallick also designed a library at the south end of the house to contain Mr. Lilly's growing collection of rare books.

After their initial remodeling of Oldfields, the Lillys continued to expand and make improvements to the property. In 1939, they chose a site to the southeast of their residence to construct a house called Newfield for their son J. K. Lilly III and his new wife Jean Heller. The next year, they constructed a recreation building to the north of Newfield, which featured indoor and outdoor swimming pools and a garden with a four seasons theme.

The late 1950s brought extensive redecoration to Oldfields. The Lillys undertook decorative work throughout the house, including scenic and trompe l'oeil wall paintings by Canadian muralist Douglas Riseborough. Highlighting the new rooms was a collection of portraits by painters such as Nattier, Fragonard, Reynolds and Goya. Many of these were later purchased from the Lilly estate by Herman C. Krannert and donated to the Indianapolis Museum of Art; some are now on view in the galleries.

Mrs. Lilly died in 1965 and Mr. Lilly died the following year. Oldfields was given to the Art Association of Indianapolis by their children, Ruth Lilly and J. K. Lilly III, to serve as the new site for the museum. In 1970, the newly renamed Indianapolis Museum of Art opened to the public at its new location.