It started with a post about stereotypes in Google Translate:
In short, when the translators (Google translate, Microsoft translator, etc) translate a sentence from gender neutral language(e.g. Turkish) to a non-gender neutral language(e.g. English), it make a guess on the gender(not really random guess, but fact/trained guess).
This behaviour has gotten my attention. So I did few rounds of tests with Malay and Chinese language, because I am a Malaysian Chinese. 🇲🇾 Malay is a gender neutral language. Chinese is mixed — it could be gender neutral, or not gender neutral.
It’s a simple 3 level tests. Tested in both Google Translate and Microsoft translators.
Both Google Translate and Microsoft Translator return
Both Google and Microsoft return same result.
Same result as level 1.
Both Google and Microsoft return same result.
Google Translate is slightly smarter. It translates both sentences correctly, while Microsoft Translator got the first sentence right, but insist that Jecelyn is a “he”!
Google translates both sentence correctly
Microsoft translates the first sentence correctly, but it insists Jecelyn is a “he”…
In Chinese，the word “他” is gender neutral. It can refer to “he” or “she”. We do have another word — “她” that is referring to female “she”, but that’s not mandatory. If you can read Chinese, you can read the long discussion and history about these two words here.
I tested with a simple sentence with context in the same sentence, this is the result:
In this case, the translators(both Google and Microsoft) return male, even context is given in same sentence.
If using “他”, the result will consistently return as male, no matter with or without context in the same sentence. Only if I change to use the female specific “她”, the result is correct:
Did the above tests with occupations — doctor, teacher, scientist, graphic designer as well. You can sorta guess the result.
definition of problem
So the question is: Is this a problem? Is Artifical Intelligence(AI) gender bias?
AI isn’t gender bias. It learns from data, trained and designed to return the result. It returned a logical result based on the model - we, human design that.
I would say the translations are statistically correct. Based on the popular surveys:
Only 9.1 registered nurses and 7.6% of licensed practical nurses are men.
We can say that the result make sense. At the end of the day, how often will you meet a female programmer, even myself doesn’t meet many female programmers. You probably won’t see a male nurse too frequent too. So, can we conclude that translators are smart? They know the probabilities of their guesses are certainly right!
NO. For me, it is a problem. It’s an area that need to be improved.
Statistically correct doesn’t mean it’s correct.
I believe, if we don’t see this a problem, then this issue wouldn’t be fixed nor there will be any improvement.
I am reading this book from Charles Wheelan (highly recommended). It’s a book about statistics.
That’s one chapter, he talks about statistical discrimination and rational discrimination.
He wrote, “Is it okay to discriminate if the data tell us that we’ll be right far more often than wrong?
It would be naive to think that gender, age, race, ethnicity, religion, and country of origin collectively tell us nothing about anything related to law enforcement. But what we can or should do with that kind of information is a philosophical and legal question, not a statical one.
If we can build a model that identifies drug smugglers correctly 80 out of 100 times, what happens to the poor souls in the 20 percent — because our model is going to harass them over and over and over again.
For the elegance and prediction of probability, there is no substitute for thinking about the calculations we are doing and why we are doing that.
We can sometimes do the calculations correctly and end up blundering in a dangerous direction.”
Of course, we are just talking about translation now, not criminal. It might not be as serious as the drug smugglers example. However, think about it, if AI in translations behave this way, how about AI in other areas?
Seriously, I don’t know. I am not a AI nor language expert. I was thinking a few solutions, but none of them good enough,
Translation is hard. It’s not only have to take grammar into account–they have to take into account context, subtext, implied meanings, cultural quirks, and a million other subjective factors and then turn them into code.
I came across this article(published in year 2013):
Google Translate's Gender Problem (And Bing Translate's, And Systran's...)_Google Translate is the world's most popular web translation platform, but one Stanford University researcher says it…_www.fastcompany.com
During that time,
Translating this English sentence
Men are men, and men should clean the kitchen
to German, return
Männer sind Männer, und Frauen sollten die Küche sauber
which means “Men are men and women should clean the kitchen.”
The translation is fixed now. That’s an improvement!
As the translation process getting better, probably we will solve this “he”, “she” soon?
Google Translate AI invents its own language to translate with_Google Translate is getting brainier. The online translation tool recently started using a neural network to translate…_www.newscientist.com
I have a few discussions with my friends. It almost turn into gender / language war.
I share the level 2 test result (Context provided in the following sentence) with my friends.
One of my friend replied me with this message:
and send me the level 3 Google translate test result (thank you, I didn’t thought of using comma!), and mention that I should use comma, instead of full stop because:
My points are,
I am pretty sure that this sentence is grammatically correct in Malay, but AI still got it wrong.
After sharing that in my Facebook. Some people starts to debate about sexist. Here is a few interesting messages.
A friend sent me this with caption “Sexist ai, man cannot be bra model?”
“If AI identify all males but actually 1 is a transgender. Does that make the AI sexist?”
“Most of the teacher are female, so translators are using “she”, make sense right?”
I were saying I see it as a problem. My friend replies “that’s the problem, you are seeing it as a problem!”
“Almost” means no wars started, heh. Starting any wars aren’t something that I’m interested in and definitely not something I wanted to.
What I would like to stress again is the righteous of translation, the consideration we put in during design the AI, systems, training models, whatever.
Logical and statically correct doesn’t mean it’s right. We need diversity. Especially in the age of AI.