Businessman, philanthropist, collector and Oldfields owner Josiah K. Lilly Jr. was a grandson of Colonel Eli Lilly, founder of the pharmaceutical firm Eli Lilly and Company. A member of the third generation of the Lilly family to lead the company, J. K. Lilly Jr. joined the firm in 1914, the same year he married Ruth Brinkmeyer. The couple had two children: Ruth Lilly, born in 1915, and J. K. Lilly III, born in 1916. J. K. Lilly Jr. and his family purchased the Oldfields estate in 1932. Although they owned other houses in Indianapolis and in Massachusetts, they kept Oldfields as their primary residence throughout their lives.

In 1937, J. K. Lilly Jr. joined his father J. K. Lilly Sr. and brother Eli Lilly in forming Lilly Endowment, a charitable trust intended to continue the family's philanthropic activities. Now one of America's largest charitable foundations,Lilly Endowment supports humanitarian and cultural endeavors across the country, with a special focus on Indiana.

J. K. Lilly Jr. was an ardent collector. In the mid-1920s, his lifelong love of literature developed into a passion for rare books. For 30 years, he devoted himself to gathering rare books on many subjects, including the works of Hoosier authors; American, British, and European literature; children's literature; and scientific and medical works. He donated his collection, including some 20,000 volumes and 17,000 manuscripts, to Indiana University in 1954, where it forms the core of the Lilly Library.

During the late 1940s, Mr. Lilly turned his collecting interest to stamps, and in the early 1950s, to gold coins. Although begun slightly later than the stamp collection, the coin collection developed faster, becoming one of the world's largest. The collection eventually comprised over 6,000 items and was acquired by the Smithsonian Institution. Mr. Lilly's stamp collection began to grow in earnest after about 1960, also developing into one of the world's largest. Numbering some 77,000 items, the collection was sold at auction after Mr. Lilly's death.

Ruth Brinkmeyer Lilly died in 1965 and J. K. Lilly Jr. died the following year. Oldfields was given to the Art Association of Indianapolis by their children, Ruth Lilly and J. K. Lilly III, to serve as a new site for the museum. In 1970, the renamed Indianapolis Museum of Art opened to the public at its new location.