Matrix is pretty dang solid overall; there's still some room for improvement if you've got the drive and the time.
- An alternate homeserver implementation. There's only one production-ready homeserver right now: Synapse. The main downsides with Synapse have historically been performance-related, but that's constantly improving - they've significantly reduced memory and compute usage in the past few months of releases. There are two front-runners to fill this slot (probably due to the complexity of the Matrix standard). Dendrite is the official project, spearheaded by the Matrix project leads. Second, I'd put Ruma - but it's a far off second, it hasn't seen a commit in about a year.
- More bridges. Personally I notice the absence of a functional GroupMe bridge (there is a bridge available but it won't run on my system, and I haven't taken the time to troubleshoot it yet). There are many other chat services out there. Ideally, bridges should be double-puppet style, requiring only my own personal authentication to both services, or appservice style, providing connectivity transparently between a Matrix room and another chat service's room. These are the most useful bridge styles.
- Solid webhooks integration. There's matrix-appservice-webhooks which fulfills most of the requirements I have, but the majority of the services I use that offer a webhook integration expect Slack to be on the other end. Appservice-webhooks doesn't quite conform to the Slack webhook style, and as a result incoming content from some services gets garbled (resulting in projects like ruby-grafana-matrix to provide a webhook endpoint that properly displays Grafana alerts).
- Variety in languages used. Almost all of the Matrix ecosystem services I run are Node.js applications. Matrix itself is not, and neither is the Hangouts bridge; they're both Python. But that's the extent of it. Ruma, mentioned above, is written in Rust; it's far from production-ready (and appears dead as a project). I'd like to see more bridges written in different languages to provide more resilience to the community as a whole. As of now we're pretty tied to Node.js's lifecycle and popularity.