One staffing buyer comes to mind when I think about harnessing the competitive spirit of the supplier community. This buyer discloses the ranking and number of placements for each of the top 15 vendors in the program each month on an all-supplier call, and those stats are also provided via email following the call to the vendors and internal stakeholders. This not only brings transparency, builds credibility and creates trust in the program, but it also generates a level of focus and priority to that client because of the open competition it creates.
I’ve just scratched the surface of this, but I hope you got the idea that scalability can mean quite different things. In Big Data (meaning the infrastructure side of it) what you want to compute is pretty well defined, for example some kind of aggregate over your data set, so you’re left with the question of how to parallelize that computation well. In machine learning, you have much more freedom because data is noisy and there’s always some freedom in how you model your data, so you can often get away with computing some variation of what you originally wanted to do and still perform well. Often, this allows you to speed up your computations significantly by decoupling computations. Parallelization is important, too, but alone it won’t get you very far.
Sex is nature’s way of bringing different data sets together, that is our genome, and creating something new that has a chance to survive.