I had a similar thought a month ago as part of a larger initiative and ended up asking a question on Stack Overflow about it: http://stackoverflow.com/questions/553264/can-stack-overflow-help-you-explore-the-magic-and-beauty-of-computer-science

It got a mostly negative response (and was closed initially; I was able to get it re-opened). The most interesting response was from Jeff Atwood (creator of Stack Overflow) via Twitter regarding my question:

“well said, but Stack Overflow is more of a sausage factory where we grind up meat and bones” – http://twitter.com/codinghorror/status/1215401481

That said, I’m not giving up hope and am trying to work with others to come up with something. It’d be great to work together if you’re interested in making something happen.

Feel free to respond to my question on Stack Overflow. I really think something like what you’re talking about is greatly needed.

I’ve noticed, for example, that I get much better feedback on my CodeProject articles than I do on my academic articles. Sure, some people who comment on CodeProject don’t know what they’re talking about, but the same is true for academic journal reviewers.

One down side of CodeProject and StackOverflow is that early voters are very influential: there’s a strong tendency to echo the status quo. It might be better to not publish vote tallies until there have been a minimum number of votes. Still, there’s a benefit to having many reviewers, even if they’re not independent, rather than just one or two.

I completely agree with your comment that reviewers hardly ever try to implement an algorithm. The reproducible research blog has several stories about completely irreproducible papers that sailed through the review process.