数据库中数据重复如何删除_数据科学中的重复数据删除

忘是亡心i 2023-03-05 11:56 136阅读 0赞

数据库中数据重复如何删除

## 重复数据删除 **(**Data Deduplication**)** ##

**Data deduplication** is a technique accustomed to cut back the quantity of space in which a corporation save its data. In most organizations, the storage systems contain duplicate copies of the many items of information. As an example, a similar file could also be saved in many completely different places by different users, or 2 or additional files that aren't identical should embrace a lot of similar data.

**重复数据删除**是一种习惯于减少公司保存数据的空间量的技术。 在大多数组织中，存储系统包含许多信息项的重复副本。 例如，相似的文件也可以由不同的用户保存在许多完全不同的位置，或者2个或不同的其他文件应包含许多相似的数据。

**Deduplication** eliminates these additional copies by saving only one copy of the info and substitution the opposite copies with pointers that lead back to the first copy. Firms oftentimes use deduplication in backup and disaster recovery applications, however, it is an accustomed liberate area in primary storage still.

**重复数据删除**通过仅保存信息的一个副本并用指向第一副本的指针替换相反的副本来消除这些其他副本。 公司通常在备份和灾难恢复应用程序中使用重复数据删除，但是，它仍然是主存储中的常规自由区域。

**Deduplication at the File or Lager Bock Level**

**在文件或更大容量的Bock级别进行重复数据删除**

In its simplest kind, deduplication takes place on the file level, it eliminates duplicate copies of a similar file. This sort of deduplication is typically known as **file-level deduplication or Single Instance Storage (SIS)**.

以最简单的方式，重复数据删除发生在文件级别，它消除了相似文件的重复副本。 这种重复数据删除通常称为**文件级重复数据删除或单实例存储(SIS)** 。

**Deduplication** also can surface on the block level, eliminating duplicate blocks of information that occur in non-identical files.

**重复数据删除**也可以出现在块级别，从而消除了在不相同文件中出现的重复信息块。

Block-level deduplication frees up extra space than SIS, and a specific kind called a variable block or variable-length deduplication has become very famous. Usually, the phrase data deduplication is employed as a word for block-level or variable-length deduplication

块级重复数据删除比SIS释放了更多的空间，一种称为可变块重复数据删除或可变长度重复数据删除的特殊类型已经非常著名。 通常，短语重复数据删除被用作块级或可变长度重复数据删除的单词

### 重复数据删除的好处 **(**Benefits of Data Deduplication**)** ###

The primary advantage of data deduplication is that it reduces the quantity of disk or tape that organizations got to purchase, which successively reduces prices. NetApp reports that in some cases, deduplication will cut back storage needs up to 95%, however, the kind of data you're attempting to deduplicate and therefore the quantity of file sharing your organization will influence your deduplication magnitude relation. Whereas deduplication is applied to data held on tape, the comparatively high prices of disk storage create deduplication a fashionable possibility for disk-based systems. Eliminating additional copies of information saves cash not solely on direct disk hardware prices, however additionally on connected prices, like electricity, cooling, maintenance, floor area, etc.

重复数据删除的主要优点在于，它减少了组织购买的磁盘或磁带的数量，从而降低了价格。 NetApp报告说，在某些情况下，重复数据删除将最多减少95％的存储需求，但是，您尝试进行重复数据删除的数据类型以及因此组织共享的文件数量将影响您的重复数据删除规模关系。 尽管将重复数据删除技术应用于磁带上保存的数据，但是磁盘存储的相对较高的价格却为基于磁盘的系统提供了一种流行的重复数据删除技术。 消除额外的信息副本不仅可以节省直接磁盘硬件价格上的现金，而且还可以节省电价，冷却，维护，占地面积等连接价格。

Deduplication also can cut back the quantity of network information measures needed for backup processes, and in some cases, it will speed up the backup and recovery method.

重复数据删除还可以减少备份过程所需的网络信息量，在某些情况下，它将加快备份和恢复方法的速度。

### 重复数据删除与压缩 **(**Deduplication vs. Compression**)** ###

**Deduplication** is typically confused with compression, another technique for reducing storage needs. Whereas deduplication eliminates redundant data, compression uses algorithms to save lots of data additional in brief. Some compression is lossless, which means that no data is lost within the method, However, "lossy" compression, which is often used with audio and video files, truly deletes a number of the less-important data enclosed in an exceedingly come to save lots of areas. Against this, deduplication solely eliminates additional copies of data; none of the first data is lost. Also, compression doesn't get obviate duplicated data -- the storage system might still contain multiple copies of compressed files.

**重复数据删除**通常与压缩相混淆，压缩是减少存储需求的另一种技术。 重复数据消除消除了冗余数据，而压缩则使用算法来简单地保存大量数据。 某些压缩是无损的，这意味着该方法不会丢失任何数据。但是，经常与音频和视频文件一起使用的“有损”压缩会真正删除一些非常重要的数据，以节省大量时间很多地区。 与此相对，重复数据删除仅消除了额外的数据副本。 没有丢失第一批数据。 而且，压缩不会消除重复的数据-存储系统可能仍包含压缩文件的多个副本。

**Deduplication** usually includes a larger impact on computer file size than compression. In an exceedingly typical enterprise backup state of affairs, compression might cut back backup size by a magnitude relation of 2:1 or 3:1, whereas deduplication will cut back backup size by up to 25:1, betting on what quantity duplicate data is within the systems. Usually, enterprises utilize deduplication and compression along to maximize their savings.

**重复数据删除**通常比压缩对计算机文件大小的影响更大。 在极其典型的企业备份状态下，压缩可能会将备份大小缩减2：1或3：1的大小关系，而重复数据删除会将备份大小缩减至25：1，这取决于重复数据的数量在系统内。 通常，企业利用重复数据删除和压缩来最大程度地节省成本。

### 实施重复数据删除 **(**Implementing Data Deduplication**)** ###

The process for implementing data deduplication technology varies wide betting on the kind of product and therefore the merchant. As an example, if deduplication technology is enclosed in an exceedingly backup appliance or storage answer, the implementation method is going to be a lot of completely different than for standalone deduplication code.

实施重复数据删除技术的过程对产品的种类以及商家的选择大相径庭。 例如，如果重复数据删除技术包含在一个极其备份的设备或存储解决方案中，则实现方法将与独立重复数据删除代码完全不同。

In general, deduplication technology is deployed in one amongst 2 basic ways: at the supply or the target. In supply deduplication, data copies are eliminated in primary storage before the info is shipped to the backup system. The advantage of supply deduplication is that reduces the information measure needs and time necessary for backing up data. On the drawback, supply deduplication consumes additional processor resources, and it is tough to integrate with existing systems and applications.

通常，重复数据删除技术是通过以下两种基本方式之一部署的：在供应方或目标方。 在重复数据删除中，在将信息传送到备份系统之前，已在主存储中消除了数据副本。 供应重复数据删除的优点是减少了信息度量需求和备份数据所需的时间。 在缺点上，重复数据删除会消耗额外的处理器资源，并且很难与现有系统和应用程序集成。

By distinction, **target deduplication** takes place at intervals the backup system and is commonly a lot of easier to deploy.

通过区别， **目标重复数据删除**是在备份系统的间隔时间进行的，并且通常更易于部署。

Target deduplication comes in 2 types: **in-line** or **post-process**.

目标重复数据删除有两种类型： **在线**或**后处理** 。

1.  **In-line deduplication** takes place before the backup copy is written to disk or tape. The advantage of in-line deduplication is that it needs less cupboard space than post-process deduplication, however, it will weigh down the backup method.
    
    **在线重复数据删除**发生在将备份副本写入磁盘或磁带之前。 在线重复数据删除的优点是，与后处理重复数据删除相比，它需要的机柜空间更少，但是，它将降低备份方法的负担。
2.  **Post-process deduplication** takes place when the backup has been written, thus it needs that organizations have a good deal of cupboard space out there for the first backup. However, post-process deduplication is typically quicker than in-line deduplication.
    
    写入备份**后，**将发生**后处理重复数据删除** ，因此需要组织为第一次备份留出足够的存储空间。 但是，处理后重复数据删除通常比在线重复数据删除更快。

### 重复数据删除技术 **(**Deduplication Technology**)** ###

**Data deduplication** may be extremely proprietary technology. Deduplication ways vary widely from the merchant to vendor, and lots of these ways are proprietary. As an example, Microsoft includes a patent on single instance storage. Additionally, Quantum owns a patent on variable-length deduplication. Several alternative vendors additionally own patents associated with deduplication technology.

**重复数据删除**可能是非常专有的技术。 重复数据删除方法因商家而异，并且其中许多都是专有的。 例如，Microsoft包含有关单实例存储的专利。 此外，昆腾还拥有可变长度重复数据删除的专利。 一些替代供应商还拥有与重复数据删除技术相关的专利。

> 翻译自: [https://www.includehelp.com/data-science/data-deduplication.aspx][https_www.includehelp.com_data-science_data-deduplication.aspx]

数据库中数据重复如何删除

[https_www.includehelp.com_data-science_data-deduplication.aspx]: https://www.includehelp.com/data-science/data-deduplication.aspx