Transliteration, Spell-Checking, and Translation: Navigating the Challenges and Epiphanies

Introduction:

In my previous article, I shared the journey behind the publication of Echoes of Sorrow - Unveiling the Urdu Heart in English. Today, I want to delve deeper into some of the unexpected insights I encountered during this journey—particularly in transliterating Urdu into Roman Urdu, automating spell-checks, and translating poetry into English. What initially seemed like straightforward tasks quickly revealed themselves as intricate puzzles, each requiring a different approach to solve.

?Transliteration Epiphany: When Character Mapping Fell Short

At first, transliterating my Urdu poetry into Roman Urdu seemed like a simple character mapping exercise. I therefore, built a dictionary to map each Urdu Unicode character to its Roman Urdu equivalent, expecting smooth results. However, this approach led to some comically inaccurate outcomes. Without understanding the context of words and sentences, the dictionary-based transliteration often produced awkward and incorrect results that made for a funny read but required more time to correct than if I had written them manually.

Even in the age of AI, context matters.?

Ultimately, I turned to prompt engineering with AI to handle the task. With carefully crafted prompts, the model began to generate more accurate Roman Urdu transliterations, understanding the nuances of language much better than my initial approach. This pivot was one of my first major lessons: even in the age of AI, context matters just as much as the data itself

Spell-Checking Challenges: The Diacritics Dilemma

When I moved on to spell-checking the Urdu text, I assumed that existing libraries would be able to handle the task. However, I was surprised to find that most tools struggled with Urdu diacritics. Words containing diacritic marks would often be flagged as incorrect, making the initial spell-checking process nearly useless. I had to write custom code to filter out diacritics before running spell-checks.

Adapt tools to fit the unique aspects of a language

This experience taught me the importance of adapting tools to fit the unique aspects of a language rather than expecting a one-size-fits-all solution. In the end, I relied on a mix of traditional dictionary searches, AI-based spell-checks, and manual review to ensure accuracy across over 70 poems in the manuscript. It was a labor-intensive but deeply rewarding process that highlighted the complexity of working with less widely supported languages like Urdu.

Translation Trials: From Automation to Editorial Review

For the English poetic translation of my poetry, I hoped that using AI- OpenAI api with finely crafted/engineered prompt, would provide quick, accurate results. While the translations were often technically sound, they lacked the emotional depth and subtlety of the original poems. There were also issues with Rhyme and Rhythm, that were just annoying Thus each poem’s essence, rhythm, and cultural context were at risk of being lost in mechanical translations. I found myself reviewing every line, making adjustments- or downright rewriting, to preserve the tone and meaning I intended.

Some elements of creativity and language are inherently human

This realization reminded me that while technology can accelerate the process, some elements of creativity and language are inherently human. It’s in the subtleties—how a word carries the weight of nostalgia, or how a metaphor might evoke a specific image—that the true spirit of a poem lives. Remember

Poetry is when an emotion has found its thought and the thought has found its word -Robert Frost

Reflections and What Lies Ahead

These experiences reinforced my belief that technology is a powerful ally, but at least for now, it cannot replace the human touch when it comes to capturing the essence of creative work. Moving forward, I plan to explore more advanced NLP tools that might help reduce manual intervention in translations; the ones that are specifically trained for poetry and poetic expressions. As mentioned in the previous article, I am also considering collaborative translation platforms, where multiple reviewers can contribute to refining the translations and bring diverse perspectives to the text. Perhaps even contributing to the training of the models...?

Conclusion:

Each challenge along this journey has offered a new perspective—one that made me appreciate both the richness of language and the potential of technology. As I continue to reflect and refine the process, I hope to make these tools even more effective, not only for my work but for others seeking to bridge the gap between tradition and innovation. I welcome you all, to do your own thought experiments, and help push the boundaries of innovation and unique ways fo using technology, far beyond the traditional use cases.