From Halopedia, the Halo wiki
In Halo multiplayer terminology, a Betrayal occurs when a player kills one of their own teammates during a team match. The act of deliberately betraying one or more teammates is referred to as teamkilling.
In order for a betrayal to occur, the betraying player must be completely responsible for the death of their teammate. Enemy players must play little or no role in the act. For example, if two players on opposing teams are battling each other, and a third player accidentally kills his already-damaged teammate with a grenade, it will not be counted as a betrayal, and the enemy will be credited with the kill. On the other hand, gunning down a teammate in your own base while no enemies are around absolutely counts as a betrayal.
Most betrayals occur with the use of explosives, as a result of unanticipated splash damage. Almost all betrayals occurring without splash damage are deliberate, though there are exceptions -- for example, a player may betray a teammate if, while taking a Warthog over a Man Cannon on Halo 3's map Valhalla, they land on their teammate. Also, in the reverse sense, a player may be driving around in a Warthog and have a friendly player fall from the man cannon on top of them, giving them a betrayal by splatter.
In Halo 2, deliberate teamkillers are a great hassle to deal with, as such an offense was not ban-worthy. In Halo 3, a new option called "Betrayal Booting" was introduced; players that commit too many betrayals can be booted by their most recent victim even during matchmade games, with the teamkiller also receiving an EXP penalty. In Custom Games, "Betrayal Booting" can be set manually, to decide how many betrayals must occur before the player is booted. In most Matchmaking games, the betrayal count is set to two.
There are instances where a player is betrayed, is unable to boot their betrayer, and then gets booted when they retaliate against the griefer. It's unknown why this occurs.
Forced team kill occurs when a player, usually a griefer, does something to get him/herself killed by their own team so that they can boot teammates. Example; a griefer deliberately steps into a sniper's line of fire, getting killed in the process and then booting the sniper, who had not intended on killing their teammate. Other examples include shooting but not killing teammates to aggrivate them, manuvering themselves under falling equipment or vehicles, and walking over a friendly grenade that is about to explode. All these scenarios allow the griefer to stay in the game to cause more trouble, while the innocent player is kicked out of the match.
There are problems with the booting system; for example, a player gets betrayed, but for some reason the boot option does not appear. The betrayed player attacks the betrayer and is booted immediately. Occasionally, one player may be able to betray many teammates, or concentrate on one, without the boot option becoming available, while attempts at retaliation result in the betrayed players being booted.
Campaign Scoring will not penalize the victim of a betrayal if they are killed by another player, but will still penalize that player.