Entity Resolution

EntityResolution.dev

At EntityResolution.dev, our mission is to provide a comprehensive resource for individuals and organizations seeking to improve their data management practices. We specialize in entity resolution, master data management, centralizing identity, record linkage, and data mastering. Our goal is to help our users join data from many sources into unified records, incrementally improving the accuracy and completeness of their data. We strive to provide high-quality content, tools, and resources that empower our users to make informed decisions and achieve their data management objectives.

/r/dataengineering Yearly

📄 "We have great datasets"

📄 i just want sleep

📄 PSA: Learn Vendor Agnostic Technologies!

📄 Exporting to excel is always a people pleaser...

📄 Data driven organisations

📄 Who owns data quality?

📄 Your Snowflake credits at work.

📄 It's amazing how many organizations workflows still revolve around Excel. I've seen CFOs and COOs folders filled with 20 different versions of the same Excel file.

📄 I’ve had the definition wrong this entire time…

📄 The current data landscape

📄 Happy (or not so happy) Wednesday! What part of your technical work do you dread the most? What are you doing about it?

📄 Data engineering projects with template: Airflow, dbt, Docker, Terraform (IAC), Github actions (CI/CD) & more

📄 If data engineering did Spotify Wrapped

📄 Follow up on that Google Drive question...

📄 Anyone read this book? It came out in 2022 so it's very modern and up to date.

📄 State of Data Engineering 2022

📄 What’s your process for deploying a data pipeline from a notebook, running it, and managing it in production?

📄 Finally got a job

📄 What are your favourite GitHub repos that shows how data engineering should be done?

📄 Data engineering with ChatGPT

📄 It's not always Old Man Jenkins...

📄 It is a recession after all, isn't it?

📄 Free Data engineering bootcamp - Data Engineering Zoomcamp - starts in 10 days

📄 How are you exporting your prod DB tables to your data warehouse?

📄 The "Big Three's" Data Storage Offerings

📄 How are you monitoring your data pipelines and what are you using to debug production issues?

📄 I didn’t know you guys were paid THIS well

📄 I like caravans more.

📄 ETL using pandas

📄 You SHALL pass...?

📄 just got laid off (FAANG)

📄 DBT lays off 15% of their staff

📄 Data engineers processing data access requests

📄 If you know, you know

📄 So I watched a few videos about Fabric, and started to cry a little...

📄 I got the job!

📄 The only insightful venn diagram I've ever made

📄 Don't Fall for the Hype: A Data Professional's Perspective on Familiar Concepts Rebranded as Innovations

📄 Job search for Data Engineering in Stockholm (2yoe)

📄 can't wait for an end to end python stack with no JVM

📄 The problem with data industry is hiring roles instead of people

📄 One day we’ll get the respect we deserve 🥲

📄 Snowflake pushing snowpark really hard

📄 PSA: we learned the hard way DBT Cloud support doesn’t work weekends…

📄 What's your favorite data quality horror story?

📄 Getting tired of “How do I break into DE posts”

📄 It's cron all the way down

📄 Just turned down a 150k job offer when I was unemployed just 2 years ago.

📄 Data pipeline design patterns

Introduction

Entity resolution, also known as record linkage or data mastering, is the process of identifying and linking records that refer to the same entity across different data sources. It is a critical component of master data management, which involves centralizing identity and joining data from many sources into unified records. This cheat sheet provides an overview of the key concepts, topics, and categories related to entity resolution and master data management.

Key Concepts

Entity: An entity is a person, place, thing, or concept that is represented in a data source. Examples of entities include customers, products, and locations.
Record: A record is a collection of data that represents an entity in a data source. A record may contain one or more attributes, such as name, address, and phone number.
Data Source: A data source is a collection of records that are stored in a specific format, such as a database or a file.
Entity Resolution: Entity resolution is the process of identifying and linking records that refer to the same entity across different data sources.
Record Linkage: Record linkage is another term for entity resolution. It refers to the process of linking records that refer to the same entity across different data sources.
Data Mastering: Data mastering is the process of creating a single, unified view of an entity by combining data from multiple sources.
Master Data Management: Master data management is the process of centralizing identity and joining data from many sources into unified records.
Data Quality: Data quality refers to the accuracy, completeness, and consistency of data.
Data Governance: Data governance is the process of managing the availability, usability, integrity, and security of data used in an organization.
Data Integration: Data integration is the process of combining data from multiple sources into a single, unified view.

Topics

Entity Resolution Techniques: There are several techniques for entity resolution, including deterministic matching, probabilistic matching, and machine learning.
Data Matching: Data matching is the process of comparing records from different data sources to identify matches and non-matches.
Data Cleansing: Data cleansing is the process of identifying and correcting errors and inconsistencies in data.
Data Standardization: Data standardization is the process of converting data from different sources into a common format.
Data Enrichment: Data enrichment is the process of adding additional data to existing records to enhance their value.
Data Profiling: Data profiling is the process of analyzing data to gain insights into its quality, completeness, and consistency.
Data Privacy: Data privacy refers to the protection of personal information from unauthorized access, use, or disclosure.
Data Security: Data security refers to the protection of data from unauthorized access, use, or disclosure.
Data Governance Frameworks: There are several data governance frameworks, including COBIT, ITIL, and ISO 38500.
Data Integration Tools: There are several data integration tools, including Talend, Informatica, and IBM InfoSphere.

Common Terms, Definitions and Jargon

1. Entity Resolution: The process of identifying and linking different data records that refer to the same real-world entity.
2. Master Data Management: A set of processes and tools used to manage an organization's critical data assets, including customer, product, and supplier data.
3. Centralizing Identity: The process of creating a single, authoritative source of identity information for an organization.
4. Record Linkage: The process of identifying and linking records from different data sources that refer to the same real-world entity.
5. Data Mastering: The process of creating a single, authoritative version of a data record by combining and reconciling data from multiple sources.
6. Data Integration: The process of combining data from multiple sources into a unified view.
7. Data Quality: The degree to which data meets the requirements of its intended use.
8. Data Governance: The process of managing the availability, usability, integrity, and security of an organization's data assets.
9. Data Stewardship: The process of managing and maintaining the quality of an organization's data assets.
10. Data Profiling: The process of analyzing data to understand its structure, content, and quality.
11. Data Cleansing: The process of identifying and correcting errors and inconsistencies in data.
12. Data Matching: The process of comparing data records to identify similarities and differences.
13. Data Enrichment: The process of enhancing data with additional information from external sources.
14. Data Standardization: The process of converting data into a common format or structure.
15. Data Normalization: The process of organizing data into a consistent and logical format.
16. Data Deduplication: The process of identifying and removing duplicate data records.
17. Data Fusion: The process of combining data from multiple sources to create a more complete and accurate view of a real-world entity.
18. Data Silos: Isolated data repositories that are not integrated with other data sources.
19. Data Warehouse: A centralized repository of data that is used for reporting and analysis.
20. Data Mart: A subset of a data warehouse that is focused on a specific business area or department.

Editor Recommended Sites

AI and Tech News
Best Online AI Courses
Classic Writing Analysis
Tears of the Kingdom Roleplay
Learn webgpu: Learn webgpu programming for 3d graphics on the browser
Code Talks - Large language model talks and conferences & Generative AI videos: Latest conference talks from industry experts around Machine Learning, Generative language models, LLAMA, AI
Coding Interview Tips - LLM and AI & Language Model interview questions: Learn the latest interview tips for the new LLM / GPT AI generative world
Cloud Code Lab - AWS and GCP Code Labs archive: Find the best cloud training for security, machine learning, LLM Ops, and data engineering
AI ML Startup Valuation: AI / ML Startup valuation information. How to value your company