Here is what some data engineering influencers said about the State of Data survey: To view the full results of the survey, go to. For Data Catalogs, there were three companies leading the way in terms of popularity, DataHub, Alation, and Amundsen.For Reverse ETL, it was essentially a tie between Hightouch and Census as leaders, and pretty much open after that.For Data Quality, leaders are Great Expectations and Monte Carlo and a lack of awareness among other alternatives.For Business Intelligence, the leaders are Looker and Tableau, but newer technologies are close behind and show lots of interest.It should be noted that Dagster is definitely coming up the ranks with the highest number of ‘want to try’. Most people are still using self-hosted Airflow, which may again (like in Data Ingestion) indicate a preference for self-hosted deployments for larger organizations. For Data Orchestration, most people are still using self-hosted Airflow, especially in the enterprise segment, but Dagster and Prefect show lots of interest.In the larger organization segments, Databricks popularity is near on par with Snowflake and BigQuery. The most used data warehouses are Snowflake and Google BigQuery, then AWS Redshift and Databricks with Azure Synapse lagging behind.However, dbt shows the most “want to try” among those users. This is even more noticeable in the larger organization segments where both Spark and Pandas are more used than dbt. For Data Transformation, most used is Pandas while dbt shows the most “want to try” among respondents.(Airbyte Open Source being the dominant solution there.) However, the enterprise segment (1,000+ employees) shows a propensity for enterprises to adopt an open-source self-hosted platform. In terms of company size, Airbyte is strong in the small/medium-sized segment with less adoption in the mid-size market (500-1,000 employees). Airbyte shows double the number of people that want to try it. For the Data Ingestion category of the modern data stack, clear leaders are Airbyte and Fivetran.In terms of compensation, larger companies correlate with more pay, and North America has the highest salaries.Nearly half the respondents were looking to hire for their data teams with consistent results across different worldwide geographic regions.Noteworthy findings include the following. “New options are introduced every month, so this research is a way for us to take a step back and understand what the community is using and feeling excited about.” “In the past year, the data ecosystem has been evolving rapidly, so this research of the user community is a way to see the signal through the noise in the modern data stack,” said John Lafleur, co-founder and chief operating officer, Airbyte. Analytics engineer, data analyst, and data scientist were around 5% apiece. The primary job title was data engineer at 38%, another 20% in management positions, and 11% software engineers. There were 886 respondents in the survey – the largest related to data engineering – that was fairly evenly distributed by geography (North America, Europe, and Asia), as well as company size, and years of experience working. Its first worldwide State of Data survey displays results in an interactive format so that anyone can drill further into the information using filters to see, for example, adoption patterns by organization size. Airbyte, creators of a fast-growing open-source data integration platform, made available results of the biggest data engineering survey in the market which provides insights into the latest trends, tools, and practices in data engineering – especially adoption of tools in the modern data stack.
0 Comments
Leave a Reply. |