In reality, tables are used in specific fashions and not fully generally. (Partly because Lua itself even recognizes some of these fashions and provides relevant operations accordingly.) Lua tables are not really a single data structure; it is a single type that acts as multiple data structures at once.
Yeah Lua optimizes array tables for performance as long as they're arrays. That's the only optimization I'm aware of. I get why they didn't just add arrays, to keep the syntax and semantics clean, simple, and unambiguous. I just don't like it. If you take that to its extreme, you get Lisp. Natural human languages are messy and full of warts, but they work despite that, or perhaps because of that, because human life is messy and warty. "Pure" languages never can or do catch on. My favorite language right now is unironically TypeScript despite all its baggage.
More than just keeping it simple, it's also about reducing the API surface of the language. Lua's main design constraint is that it's meant to be embedded inside other applications, and the versatile table type helps a lot in that regard.