It all started with the fruit itself. The original ancient Indian Sanskrit name of the tree was nāraṅgaḥ. Indians and Arabs traded with each other and so the word passed into Arabic as naranjah. For a time, Spain was ruled by Arabic-speaking north African Moors who passed the fruit and its name into Spain as naranja (pronounced nar-AHN-ha).
Eventually, this fruit came to be known by the English who called it a naranj. Words ending with the letter 'j' are rare in the English language, so the spelling changed to narange.
Over time, "a narange" became "an arange." Then, in the 14th century, ARANGE, not to be confused with ARRANGE, was transformed into ORANGE (several years later Aluminum became Aloo-MINI-yum).
Wait... Nice try, but surnames don't count in my book. How about I change my last name to "Zorange?" Does that count as a rhyme too?! As they might say in Wales atop Blorenge hill, "That's bollocks!" And since the Welsh don't really speak English, I'm calling bullshit on Blorenge too.