James Trotter of Scotland emigrated to Brunswick County Virginia, where he married Ann about 1743. They had five sons and one daughter. The two oldest sons, George and Isham were Revolutionary War heroes and are chronicled in Virginia archives. William, the third son, born in 1753, moved to Person County, North Carolina in 1784, one year after the death of his father James. William and his wife Nancy had ten children. The oldest son James was grandfather of V.J. Trotter, who built the Trotter House in Monticello. James was originally a schoolteacher and then became a farmer. It is notable that this wife, Mary Draper, was the granddaughter of Solomon Draper, an Englishman, who at age eighteen was deported to Virginia as a felon for stealing a horse. After 14 years of servitude, Solomon immediately bought 300 acres of farmland, suggesting that he may have married the daughter of his indenturer. The James Trotter family moved to Summerville, Tennessee in 1836 where they had a farm about twenty five miles east of Memphis. James himself died in 1840, and it was rumored that he "died the death of a drunkard", however, this is not confirmed. James's son, Henry J. Trotter, then assisted his mother, Mary (Draper), in running the farm. Besides Henry J., there were nine other children, William B., Maritia, Ann, Solomon, Mary M., Green, Cornelia, Elizabeth-Susan, and Thomas. In 1852 Mary and all the family except William B. left the farm in Tennessee with the intention of settling in Daingerfield, Texas with her daughter-in-law's family, the Orange Connors. (Orange was so named because of his Protestant Irish origins.) Unfortunately, the Saline and Ouachita Rivers were continuously flooded for a two year period. The Trotter family stopped north of Monticello, but when the flood waters did not recede they decided to homestead in Arkansas. Their farm about 6 miles north of Monticello raised mostly cotton. Henry J. was an energetic entrepreneur and soon bought a cotton gin which serviced many of the local farms. The cost of this gin was $500 as attested by a bill from the Memphis dealer! Henry J.'s wife, Malvina Connor, was a strong believer in education and well read in the classics. There were six children, John Rufus, William Cicero, Henry Juan, Virgil Josephus, and Mary Caledonia. Although the nearest rail or steamboat access was fifty miles away, three of the sons graduated from college, a very unusual occurrence at that time. Cicero received an M.D. from the University of Louisville, Henry Juan a doctorate of religion from Vanderbilt, and John a degree in Latin from Vanderbilt. It is ironic that Virgil went only through the fifth grade, but later was to be president of Monticello School Board. Henry J. (the elder) was a practical man, but like his father and grandfather was heavily involved with the Methodist Church. Each generation had given land to the church in their respective states of North Carolina, Tennessee, and Arkansas for a "meeting house". When the Civil War broke out, he forbade any of his sons entering the military, stating that "the South should listen to Lincoln, and free the slaves". He himself did have several slaves, and the fact that they kept the Trotter name indicates they were at least decently treated. One of the best known slave descendants was Wiley Trotter, a black veterinarian who lived in Drew County for over 100 years!

The Trotter House was built by Virgil Josephus Trotter shortly before his marriage to Willie Ada Ford in 1896. He had left the farm and joined in partnership with J.D. McCloy in Monticello. The mercantile business which he co-founded in 1887 remained in continuous business until sold to the Van Atkins chain in 1962. The original house was twice enlarged as the family of eight children required more space. The house was said to be the first in Monticello with an indoor sanitary sewage system, and at one time was lighted by illuminating gas. The original heating was by individual fireplaces in the various rooms, and these were still used until about 1950 when gas heaters were installed. The large attic was once used as a roller skating rink by Juan and Henry, and it was not unusual to find the boys running about on the roof of the adjacent carriage house. A fig tree in the back yard sank its roots beneath the original three-holer toilet in the carriage house, and still bears fruit today. Virgil Josephus Trotter lived to age 100, dying at the home in 1956. None of his direct descendents are now living in Drew County, although there are grandchildren of his Uncle Green Trotter still in the area, and a grandson, Henry Trotter, runs the Ford dealership in Pine Bluff. The oldest son of V.J., Virgil Juan lived in the Trotter House from birth in 1897 until his death September 25, 1994. A grandson, Reginald Trotter, who now lives in France, is a frequent guest at The Trotter House.