I don't think this has ever been tried before. At least not in the modern era.

I would try this once in a while (maybe 3 times a season). On an obvious passing down (3rd and 14, for instance), I'd line up my defense in a punt formation. The punter 12 yards back, three protectors (two side by side, and one standing behind, ready to step over once the ball is hiked). Then I'd have 7 guys "in the box".
I think is likely to cause some confusion on the offense, kind of like the first time someone ran the Duck. Maybe it would even trick the QB into doing a quick kick. If they did quick kick it, the punter could quickly run back to the field the kick. And the 3 protectors are ready to form a wedge. If you don't fool them, you have the standard 4 guys (3 protectors and the punter) ready to defend against the pass. Meanwhile, you have 7 guys "in the box" ready to put some pressure on the passer, maybe even scoring a "sack".