Kafka and Spotify

https://labs.spotify.com/2016/02/25/spotifys-event-delivery-the-road-to-the-cloud-part-i/

Apache NiFi aka DataFlow

http://www.infoworld.com/article/2975833/hadoop/hortonworks-buys-better-hadoop-data-flow-management.html

Hive Streaming

https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest This leverages Hive transaction capability but is limited to tables with ORC format. Only supports Storm and Flume.

Spark Streaming vs Flink

Hortonworks HDP

http://hortonworks.com/hdf/use-cases/

Workflow Design and Execution Engines - Luigi Airflow Pinball

http://bytepawn.com/luigi-airflow-pinball.html

Streamsets - Open Source flume interface

https://streamsets.com

azkaban open source workflow

this has become better now... check this out https://azkaban.github.io