When Moses was kicked out of Egypt, he went to Midian, where he encountered Jethro, the priest of the Midianites, and married one of Jethro’s daughters. As the priest there, he definitely appears to have been polytheistic, as were all the Caananites at the time. This would also appear to include the Israelites, who seem to have primarily been worshiping El up til this point. While he was staying with Jethro, he encountered Yhwh. (Exodus 3&4)
He then goes back to lead Israel (the People of El) out of Egypt, and upon successfully leaving Egypt with them, he is met by his father in law, Jethro the Priest of Midian, who appears to be quite impressed with what Moses has done with YHWH, and tells Moses “Now I know YHWH is greater than all gods” in Exodus 18:11. Jethro tells Moses (since he is a priest) that Moses should start acting like a priest, and go between the People of El (Israelites) and his new god YHWH. Jethro then returns to Midian.
When Moses takes him up on this, and climbs the mountain to get the 10 commandments, the Israelites go and make a golden calf (a bull being the usual depiction of El). There is also one mention of YHWH in ancient texts as a Son of El. Of course Moses doesn’t like this, since he wants the people to follow him.
Probably at this point Moses’ uncle, in his job as priest back in Midian, begins placing an emphasis on and perhaps primarily worshipping Yhwh, instead of the other gods.
In Number 10:29-36, we see that a little later Moses talks his brother in law (from Midian) into coming with them.
It seems that as the Israelites go through the land, people are scared of them, including the current king of Midian, who appears to be a follower of Baal (Numbers 22), not Yahweh as Jethro the priest of Midian, and Moses his son-in-law now do.
It appears that at some point Jethro the priest leaves the Midianites, and comes along with Moses and the Israelites, along with his extended family, so apparently the Midianites were divided into a group that followed Jethro the priest, and a group that stayed and followed the king. We see this in Judges 1:16. “And the children of the Kenite, Moses’ father in law, went up out of the city of palm trees with the children of Judah into the wilderness of Judah, which lieth in the south of Arad; and they went and dwelt among the people”
So when Moses and Israel attack the remaining Midianites, it’s only because Moses’ extended family from Midian, and their friends have already left and are now following him with the Israelites.
There’s also a lot of external material from way back before Moses’ time (think around Abraham) that talks about El who was the God of Abraham and the God of Nahor (another ancient city in Canaan, perhaps named after Abraham’s grandfather Genesis 31:53). We also know from the Torah that Laban, who was Nahor’s son, and Abraham’s uncle was a polytheist (Genesis 31:30 – “…wherefore hast thou stolen my gods.”)
Eventually, and perhaps to make it easier for the Israelites to worship El, the word el becomes a more generic word for any god, and the proper name El, as well as the plural Elohim come to all be used interchangeably with YHWH.
In fact the entire old testament shows that Israelites themselves were never really monotheistic, and there are all sorts of accounts of them worshipping this god and that god. There was only a small group of them that really focused on Yhwh and as a group, they didn’t get the monotheism act together until quite a bit later…