Born as Aleck Ford on the Sara Jones Plantation in Tallahatchie County, Mississippi, his date and year of birth are a matter of uncertainty. He claimed to have been born on December 5, 1899, but one researcher, David Evans, claims to have found census record evidence that he was born around 1912. His gravestone lists his date of birth as March 11, 1908.
He lived and worked with his sharecropper stepfather, Jim Miller, whose last name he soon adopted, and mother, Millie Ford, until the early 1930s. Beginning in the 1930s, he traveled around Mississippi and Arkansas and encountered Big Joe Williams, Elmore James and Robert Lockwood, Jr., also known as Robert Junior Lockwood, who would play guitar on his later Checker Records sides. He was also associated with Robert Johnson during this period. Miller developed his style and raffish stage persona during these years. Willie Dixon recalled seeing Lockwood and Miller playing for tips in Greenville, Mississippi in the 1930s. He entertained audiences with novelties such inserting one end of the harmonica into his mouth and playing with no hands.