Besides using Group Tables, what approaches can be used to program forwarding trees into network switches? Assume a scenario similar to SDN where a controller installs forwarding rules on switches.
Assuming your concern is about OpenFlow, what about adding multiple outputs to the same flow rule? https://www.sdntesting.com/can-an-openflow-switch-replace-a-tap-aggregator/
The IETF's mboned working group has written an internet draft that suggests that any source multicast, especially in the case of interdomain multicast, should be deprecated. The mboned working group suggests that things would be greatly simplified if only source-specific multicast were used. For example, rendezvous points would no longer be required. I've attached a link to the internet draft, in case you want to read all the details.
With that in mind, I am also assuming that you'd still be using IGMPv3 or MLDv2 for hosts to indicate their desire to receive certain multicast packets. I suppose that in principle, with SDN, the controller could make up its own rules for who should be provided multicast packets from which multicast group. But I'm assuming we retain the leaf-initiated multicast model described in RFC 1112 and its successors.
Well, there should be various strategies available, and my thinking is that the academic community will be trying to devise which strategies are most "efficient," under different scenarios. One trivial strategy, which requires very little state information and little processing, is simply to flood any multicast packets to all switches. This won't scale well, clearly, so it's viable only if very few multicast sources exist.
If we assume that only source-specific multicast (SSM) is to be used, then another strategy could be for the SDN controller to set up individual unicast paths, from each multicast source to each host indicating an interest in receiving multicast packets from that source. You don't necessarily have to create "group tables" for this. You can literally use the IP destination address of each destination device, which is a field in the IGMP or MLD join packet, so it is available to the SDN controller. I'm only suggesting this because you asked for "no group tables."
There are many possible strategies, I would expect. One thing, though, since IPTV is so often mentioned in these discussions, is that IPTV usually involves primarily video on demand. That's one of the big advantages of IPTV, and the majority of traffic volume will be for video on demand, most of the time. IP multicast is only useful for the minority of cases, in which many people really want to watch something "live."
So, in order to accommodate video on demand in large scale, IPTV networks have to determine how many mirrored servers to deploy, and how close to the edges to deploy these servers. All of course based on the popularity of the different programs. The distributed servers are provided with their video content using some sort of out of band scheme, which can even be satellite broadcast. Now my question becomes, why can't these same servers, close to the network edges, which can be fed their video programms out of band, also be used for live programming? The live streams are fed to the servers, and then individual unicast sessions from servers to each host. Point being, the system must scale for video on demand, which is the prevalent method used by far. This being the case, why not also use these distributed servers for the live streams? Depending how the distributed servers are fed with the program material, no multicast may be involved at all!