Python решение (протестировано на Python 3.5):
del_duplicates.py скрипт:
import sys
with open(sys.argv[1], 'r') as f: # reading the file (passed in via command line)
lines = f.read().splitlines() # split the file into list of lines
for l in lines: # for each line
items = l.split('\t') # split line by tab `\t`
for k,i in enumerate(items): # processing fields
if k > 0: items[k] = ','.join(set(i.split(','))) # getting unique values via set object
print('\t'.join(items)) # join separate fields back into straight line
Использование:
python del_duplicates.py yourfile
Вывод:
1 B C D
2 A B E