22-----使用可组合的工作流程转换表格数据

Dear 丶 2023-09-27 19:38 1阅读 0赞

txtai 支持处理非结构化和结构化数据。结构化或表格数据被分组为行和列。这可以是电子表格、返回 JSON 或 XML 的 API 调用，甚至是键值对列表。

本文将通过示例介绍如何使用带有表格管道的工作流来转换和索引结构化数据。

安装依赖  
安装txtai和所有依赖项。我们将安装 api、pipeline 和 workflow 可选的附加包。  
pip install txtai\[api,pipeline,similarity\] sacremoses  
CSV 工作流程  
第一个示例将转换和索引 CSV 文件。COVID-19 开放研究数据集( CORD-19) 是涵盖 COVID-19 的医学文章库。此工作流读取输入 CSV 并构建语义搜索索引。

第一步是在本地下载数据集。

## Get CORD-19 metadata file ##

!wget https://ai2-semanticscholar-cord-19.s3-us-west-2.amazonaws.com/2021-11-01/metadata.csv  
!head -1 metadata.csv > input.csv  
!tail -10000 metadata.csv >> input.csv  
下一节将创建一个由表格管道组成的简单工作流。表格管道构建一个 (id, text, tag) 元组列表，可以轻松加载到嵌入索引中。对于此示例，我们将使用该url列作为 id 并将该title列用作文本列。textcolumns 参数采用列列表来支持从多个列中索引文本内容。

处理文件 input.csv 并显示前 5 行。  
from txtai.pipeline import Tabular  
from txtai.workflow import Task, Workflow

## Create tabular instance mapping input.csv fields ##

tabular = Tabular(“url”, \[“title”\])

## Create workflow ##

workflow = Workflow(\[Task(tabular)\])

## Print 5 rows of input.csv via workflow ##

list(workflow(\[“input.csv”\]))\[:5\]  
\[(‘https://doi.org/10.1016/j.cmpb.2021.106469; https://www.ncbi.nlm.nih.gov/pubmed/34715516/’,  
‘Computer simulation of the dynamics of a spatial susceptible-infected-recovered epidemic model with time delays in transmission and treatment.’,  
None),  
(‘https://www.ncbi.nlm.nih.gov/pubmed/34232002/; https://doi.org/10.36849/jdd.5544’,  
‘Understanding the Potential Role of Abrocitinib in the Time of SARS-CoV-2’,  
None),  
(‘https://doi.org/10.1186/1471-2458-8-42; https://www.ncbi.nlm.nih.gov/pubmed/18234083/’,  
“Can the concept of Health Promoting Schools help to improve students’ health knowledge and practices to combat the challenge of communicable diseases: Case study in Hong Kong?”,  
None),  
(‘https://www.ncbi.nlm.nih.gov/pubmed/32983582/; https://www.sciencedirect.com/science/article/pii/S2095809920302514?v=s5; https://api.elsevier.com/content/article/pii/S2095809920302514; https://doi.org/10.1016/j.eng.2020.07.018’,  
‘Buying time for an effective epidemic response: The impact of a public holiday for outbreak control on COVID-19 epidemic spread’,  
None),  
(‘https://doi.org/10.1093/pcmedi/pbab016’,  
‘The SARS-CoV-2 spike L452R-E484Q variant in the Indian B.1.617 strain showed significant reduction in the neutralization activity of immune sera’,  
None)\]  
接下来，我们获取工作流输出，构建嵌入索引并运行搜索查询。  
from txtai.embeddings import Embeddings

## Embeddings with sentence-transformers backend ##

embeddings = Embeddings(\{“method”: “transformers”, “path”: “sentence-transformers/paraphrase-mpnet-base-v2”\})

## Index subset of CORD-19 data ##

data = list(workflow(\[“input.csv”\]))  
embeddings.index(data)

for uid, \_ in embeddings.search(“insulin”):  
title = \[text for url, text, \_ in data if url == uid\]\[0\]  
print(title, uid)  
Importance of diabetes management during the COVID-19 pandemic. https://doi.org/10.1080/00325481.2021.1978704; https://www.ncbi.nlm.nih.gov/pubmed/34602003/  
Position Statement on How to Manage Patients with Diabetes and COVID-19 https://www.ncbi.nlm.nih.gov/pubmed/33442169/; https://doi.org/10.15605/jafes.035.01.03  
Successful blood glucose management of a severe COVID-19 patient with diabetes: A case report https://www.ncbi.nlm.nih.gov/pubmed/32590779/; https://doi.org/10.1097/md.0000000000020844  
该示例搜索了术语insulin。最上面的结果提到了糖尿病和血糖，它们是与糖尿病密切相关的术语。

包含存储内容的工作流  
接下来，我们将重新运行添加完整内容存储的相同示例。全内容存储支持 SQL 查询。  
import json

## Create tabular instance mapping input.csv fields ##

tabular = Tabular(“url”, \[“title”\], True)

## Create workflow ##

workflow = Workflow(\[Task(tabular)\])

## Embeddings with sentence-transformers backend ##

embeddings = Embeddings(\{“method”: “transformers”, “path”: “sentence-transformers/paraphrase-mpnet-base-v2”, “content”: True\})

## Index subset of CORD-19 data ##

data = list(workflow(\[“input.csv”\]))  
embeddings.index(data)

for result in embeddings.search(“select title, abstract, authors, doi from txtai where similar(‘insulin’)”):  
print(json.dumps(result, default=str, indent=2))  
\{  
“title”: “Importance of diabetes management during the COVID-19 pandemic.”,  
“abstra