Long before Jesus of Nazareth appeared on the scene it was already customary among middle eastern people, including the Jews, to display mourning, sadness, and repentance using ashes. Among many examples Mordecai, a Persian official, put dust and ashes on his head during a desperate hour (Esther, chapter 4 verse 1 ERV), Daniel in Bablylon did the same (Daniel, chapter 9 verse 3 ), and the ancient patriarch Job famously exclaimed (upon realizing he'd gotten God all wrong), "I am ashamed of myself. I am so sorry. As I sit in the dust and ashes, I promise to change my heart and my life." (Job, chapter 42 verse 6 ERV).
Significance of Ashes
It's a natural connection to make; ashes are what is left when everything has been destroyed. Job and the rest would actually sit on ashpiles to show how abject they'd become (Job, chapter 2 verse 8 ERV). One of the poets in the Book of Psalms (described in the title as "an oppressed person") cried out,"Great sadness (Literally, "ashes') is my only food. My tears fall into my drink," (Book of Psalms, chapter 102 verse 9). Ashes and sorrow, ashes and regret, ashes and repentance just go together.
Jesus picked up on this image, applying it to two recalcitrant cities: "It will be bad for you, Chorazin! It will be bad for you, Bethsaida! I did many miracles in you. If those same miracles had happened in Tyre and Sidon, then the people in those cities would have changed their lives and stopped sinning a long time ago. They would have worn sackcloth and sat in ashes to show that they were sorry for their sins." (Gospel of Luke, chapter 10 verse 13 ERV).
For Jesus, sitting in ashes is an appropriate symbol of sorrow for your sins.
Years later, this was still true for his people. For example, about 150 years after Jesus' time a Christian lawyer named Tertullian wrote an entire book about repenting. Among other things he described a repentant person as, "unwashen, sordidly attired, estranged from gladness, they must spend their time in the roughness of sackcloth, and the horridness of ashes, and the sunkenness of face caused by fasting," (On Repentance, Chapter 11).
Connection to the Season of Lent
Lent, which begins on Ash Wednesday, is itself a time of reflection on your life and "repenting" to prepare for the great feast of Easter. Jesus, Moses, and the prophet Elijah each prepared for important phases of their work by fasting for forty days. Followers of Jesus imitate him by fasting for 40 days -- although we don't do it with nearly the intensity (no food!) he did. In the same way we don't sit on a pile of ashes dressed in goat hair, like Job or the person Tertullian was describing. But we do have ashes smeared on our foreheads in the shape of a cross, reminded that we come and ultimately return to dust, and are told to remember the meaning of our baptism.
Incidentally, the first person we know of that definitely mentions having ashes "imposed" on one's forehead at the beginning of Lent was a English monk named Ælfric (Ælfric’s Lives of Saints, pp. 262-266) around AD 1000. But it was already an old custom by then. In fact, as far back as the AD 700's the day was called "dies cinerum" -- Day of Ashes.
Since Ælfric’s time a little daub of ashes on the forehead has become the near-universal symbol of what is going on inside Jesus' followers during this time: examining one's life, bringing it into union with his teachings, and rooting out what contradicts them. And this so as to be ready for the High Feast of the Christian year, the celebration of Christ going through death and coming out the other side!