The Fiery Furnaces are brother and sister Eleanor  and Matt  Friedberger (both on vocals and guitar, Matt on synth and organ). Siblings Eleanor and Matt were born and raised in Oak Park, IL, but relocated to New York to form the Fiery Furnaces. As children, the two weren't particularly close, a dynamic that shaped their shaky relationship over the future years.

After college and separate trips abroad Matt and Eleanor returned to their home in Oak Park and began working on music together. They mixed simple, poppy melodies with a dizzying array of wordplay, sounds, and influences, including the Who, Captain Beefheart, Os Mutantes; dashes of folk, blues, and garage rock; and Eleanor's adventures in Europe.

A series of early appearances throughout NYC resulted in them being discovered by British underground label Rough Trade, which they signed up with in 2002 on the basis of their demo. By the time they signed to Rough Trade, their debut, Gallowsbird's Bark was completed and the Fiery Furnaces were already at work on the follow-up.  Released in the fall of 2003, Gallowsbird's Bark  garnered critical praise for its clever wordplay and original songwriting.

The band gained more momentum the following year, when praise for the debut album dovetailed with the release of the group's even more diverse and challenging sophomore album, Blueberry Boat, that summer. The Fiery Furnaces spent much of 2004 touring with Ted Leo & the Pharmacists, Franz Ferdinand, and the Shins, but already had plans for their third and fourth albums, including an album of duets with their grandmother and a set of beatnik-inspired songs. Early in 2005, the group released the simply-titled EP, a mini-album gathering most of their B-sides along with a few new songs.

April 2006 saw the release of the Fiery Furnaces' fourth studio album proper, a collection of "love songs to dance to" according to Matt, titled Bitter Tea, which Matt described as "sissy psychedelic satanism". The album was supposed to be released at the same time as Rehearsing My Choir, however, financial issues prevented Rough Trade, the band's label, from doing so. The label's US parent, Sanctuary, was forced to licence the US release of Bitter Tea to Fat Possum.