首页 科技正文

日照文明网:HBase Filter 过滤器之 Comparator 原理及源码学习

admin 科技 2020-04-26 33 0

前言:上篇文章HBase Filter 过滤器概述对HBase过滤器的组成及其家谱举行简朴先容,本篇文章主要对HBase过滤器之对照器作一个弥补先容,也算是HBase Filter学习的必备低阶魂技吧。本篇文中源码基于HBase 1.1.2.2.6.5.0-292 HDP版本。

HBase所有的对照器实现类都继续于父类ByteArrayComparable,而ByteArrayComparable又实现了Comparable接口;差异功效的对照器差异在于对父类compareTo()方式的重写逻辑差异。

下面分别对HBase Filter默认实现的七大对照器逐一举行先容。

1. BinaryComparator

先容:二进制对照器,用于按字典顺序对照指定字节数组。

先看一个小例子:

public class BinaryComparatorDemo {

    public static void main(String[] args) {

        BinaryComparator bc = new BinaryComparator(Bytes.toBytes("bbb"));

        int code1 = bc.compareTo(Bytes.toBytes("bbb"), 0, 3);
        System.out.println(code1); // 0
        int code2 = bc.compareTo(Bytes.toBytes("aaa"), 0, 3);
        System.out.println(code2); // 1
        int code3 = bc.compareTo(Bytes.toBytes("ccc"), 0, 3);
        System.out.println(code3); // -1
        int code4 = bc.compareTo(Bytes.toBytes("bbf"), 0, 3);
        System.out.println(code4); // -4
        int code5 = bc.compareTo(Bytes.toBytes("bbbedf"), 0, 6);
        System.out.println(code5); // -3
    }
}

不难看出,该对照器的对照规则如下:

  • 两个字符串首字母差异,则该方式返回首字母的asc码的差值
  • 介入对照的两个字符串若是首字符相同,则对照下一个字符,直到有差异的为止,返回该差异的字符的asc码差值
  • 两个字符串不一样长,可以介入对照的字符又完全一样,则返回两个字符串的长度差值

看一下以上规则对应其compareTo()方式的源码实现:
实现一:

static enum UnsafeComparer implements Bytes.Comparer<byte[]> {
INSTANCE;
....
public int compareTo(byte[] buffer1, int offset1, int length1, byte[] buffer2, int offset2, int length2) {
	if (buffer1 == buffer2 && offset1 == offset2 && length1 == length2) {
		return 0;
	} else {
		int minLength = Math.min(length1, length2);
		int minWords = minLength / 8;
		long offset1Adj = (long)(offset1 + BYTE_ARRAY_BASE_OFFSET);
		long offset2Adj = (long)(offset2 + BYTE_ARRAY_BASE_OFFSET);
		int j = minWords << 3;

		int offset;
		for(offset = 0; offset < j; offset += 8) {
			long lw = theUnsafe.getLong(buffer1, offset1Adj + (long)offset);
			long rw = theUnsafe.getLong(buffer2, offset2Adj + (long)offset);
			long diff = lw ^ rw;
			if (diff != 0L) {
				return lessThanUnsignedLong(lw, rw) ? -1 : 1;
			}
		}

		offset = j;
		int b;
		int a;
		if (minLength - j >= 4) {
			a = theUnsafe.getInt(buffer1, offset1Adj + (long)j);
			b = theUnsafe.getInt(buffer2, offset2Adj + (long)j);
			if (a != b) {
				return lessThanUnsignedInt(a, b) ? -1 : 1;
			}

			offset = j + 4;
		}

		if (minLength - offset >= 2) {
			short sl = theUnsafe.getShort(buffer1, offset1Adj + (long)offset);
			short sr = theUnsafe.getShort(buffer2, offset2Adj + (long)offset);
			if (sl != sr) {
				return lessThanUnsignedShort(sl, sr) ? -1 : 1;
			}

			offset += 2;
		}

		if (minLength - offset == 1) {
			a = buffer1[offset1 + offset] & 255;
			b = buffer2[offset2 + offset] & 255;
			if (a != b) {
				return a - b;
			}
		}

		return length1 - length2;
	}
}

实现二:

static enum PureJavaComparer implements Bytes.Comparer<byte[]> {
	INSTANCE;

	private PureJavaComparer() {
	}

	public int compareTo(byte[] buffer1, int offset1, int length1, byte[] buffer2, int offset2, int length2) {
		if (buffer1 == buffer2 && offset1 == offset2 && length1 == length2) {
			return 0;
		} else {
			int end1 = offset1 + length1;
			int end2 = offset2 + length2;
			int i = offset1;

			for(int j = offset2; i < end1 && j < end2; ++j) {
				int a = buffer1[i] & 255;
				int b = buffer2[j] & 255;
				if (a != b) {
					return a - b;
				}

				++i;
			}

			return length1 - length2;
		}
	}
}

实现一是对实现二的一个优化,都引自Bytes类,HBase优先执行实现一方案,若是有异常再执行实现二方案。如下:

public static int compareTo(byte[] buffer1, int offset1, int length1, byte[] buffer2, int offset2, int length2) {
	return Bytes.LexicographicalComparerHolder.BEST_COMPARER.compareTo(buffer1, offset1, length1, buffer2, offset2, length2);
}
...
...

static final String UNSAFE_COMPARER_NAME = Bytes.LexicographicalComparerHolder.class.getName() + "$UnsafeComparer";
static final Bytes.Comparer<byte[]> BEST_COMPARER = getBestComparer();
static Bytes.Comparer<byte[]> getBestComparer() {
	try {
		Class<?> theClass = Class.forName(UNSAFE_COMPARER_NAME);
		Bytes.Comparer<byte[]> comparer = (Bytes.Comparer)theClass.getEnumConstants()[0];
		return comparer;
	} catch (Throwable var2) {
		return Bytes.lexicographicalComparerJavaImpl();
	}
}

2. BinaryPrefixComparator

先容:二进制对照器,只对照前缀是否与指定字节数组相同。

先看一个小例子:

public class BinaryPrefixComparatorDemo {

    public static void main(String[] args) {

        BinaryPrefixComparator bc = new BinaryPrefixComparator(Bytes.toBytes("b"));

        int code1 = bc.compareTo(Bytes.toBytes("bbb"), 0, 3);
        System.out.println(code1); // 0
        int code2 = bc.compareTo(Bytes.toBytes("aaa"), 0, 3);
        System.out.println(code2); // 1
        int code3 = bc.compareTo(Bytes.toBytes("ccc"), 0, 3);
        System.out.println(code3); // -1
        int code4 = bc.compareTo(Bytes.toBytes("bbf"), 0, 3);
        System.out.println(code4); // 0
        int code5 = bc.compareTo(Bytes.toBytes("bbbedf"), 0, 6);
        System.out.println(code5); // 0
        int code6 = bc.compareTo(Bytes.toBytes("ebbedf"), 0, 6);
        System.out.println(code6); // -3
    }
}

该对照器只是基于BinaryComparator对照器稍作更改而已,以下代码一目了然:

public int compareTo(byte[] value, int offset, int length) {
	return Bytes.compareTo(this.value, 0, this.value.length, value, offset, this.value.length <= length ? this.value.length : length);
}

看一下同BinaryComparator方式的异同:

public int compareTo(byte[] value, int offset, int length) {
	return Bytes.compareTo(this.value, 0, this.value.length, value, offset, length);
}

区别只在于最后一个传参,即length=min(this.value.length,value.length),取小。这样在后面的字节逐位对照时,即只需对照min length次。

3. BitComparator

先容:位比价器,通过BitwiseOp提供的AND(与)、OR(或)、NOT(非)举行对照。返回效果要么为1要么为0,仅支持 EQUAL 和非 EQUAL。

先看一个小例子:

public class BitComparatorDemo {

    public static void main(String[] args) {

        // 长度相同按位或对照:由低位起逐位对照,每一位按位或对照都为0,则返回1,否则返回0。
        BitComparator bc1 = new BitComparator(new byte[]{0,0,0,0}, BitComparator.BitwiseOp.OR);
        int i = bc1.compareTo(new byte[]{0,0,0,0}, 0, 4);
        System.out.println(i); // 1
        // 长度相同按位与对照:由低位起逐位对照,每一位按位与对照都为0,则返回1,否则返回0。
        BitComparator bc2 = new BitComparator(new byte[]{1,0,1,0}, BitComparator.BitwiseOp.AND);
        int j = bc2.compareTo(new byte[]{0,1,0,1}, 0, 4);
        System.out.println(j); // 1
        // 长度相同按位异或对照:由低位起逐位对照,每一位按位异或对照都为0,则返回1,否则返回0。
        BitComparator bc3 = new BitComparator(new byte[]{1,0,1,0}, BitComparator.BitwiseOp.XOR);
        int x = bc3.compareTo(new byte[]{1,0,1,0}, 0, 4);
        System.out.println(x); // 1
        // 长度差异,返回1,否则按位对照
        BitComparator bc4 = new BitComparator(new byte[]{1,0,1,0}, BitComparator.BitwiseOp.XOR);
        int y = bc4.compareTo(new byte[]{1,0,1}, 0, 3);
        System.out.println(y); // 1
    }
}

上述注释论述的规则,对应以下代码:
···
public int compareTo(byte[] value, int offset, int length) {
if (length != this.value.length) {
return 1;
} else {
int b = 0;

	for(int i = length - 1; i >= 0 && b == 0; --i) {
		switch(this.bitOperator) {
		case AND:
			b = this.value[i] & value[i + offset] & 255;
			break;
		case OR:
			b = (this.value[i] | value[i + offset]) & 255;
			break;
		case XOR:
			b = (this.value[i] ^ value[i + offset]) & 255;
		}
	}

	return b == 0 ? 1 : 0;
}

}
···
焦点头脑就是:由低位起逐位对照,直到b!=0退出循环。

4. LongComparator

先容:Long 型专用对照器,返回值:0 -1 1。上篇概述没有提到,这里补上。

先看一个小例子:

public class LongComparatorDemo {

    public static void main(String[] args) {
        LongComparator longComparator = new LongComparator(1000L);
        int i = longComparator.compareTo(Bytes.toBytes(1000L), 0, 8);
        System.out.println(i); // 0
        int i2 = longComparator.compareTo(Bytes.toBytes(1001L), 0, 8);
        System.out.println(i2); // -1
        int i3 = longComparator.compareTo(Bytes.toBytes(998L), 0, 8);
        System.out.println(i3); // 1
    }
}

这个对照器实现相当简朴,不多说了,如下:

public int compareTo(byte[] value, int offset, int length) {
	Long that = Bytes.toLong(value, offset, length);
	return this.longValue.compareTo(that);
}

5. NullComparatorDemo

先容:控制对照式,判断当前值是不是为null。是null返回0,不是null返回1,仅支持 EQUAL 和非 EQUAL。

先看一个小例子:

public class NullComparatorDemo {

    public static void main(String[] args) {
        NullComparator nc = new NullComparator();
        int i1 = nc.compareTo(Bytes.toBytes("abc"));
        int i2 = nc.compareTo(Bytes.toBytes(""));
        int i3 = nc.compareTo(null);
        System.out.println(i1); // 1
        System.out.println(i2); // 1
        System.out.println(i3); // 0
    }
}

这个对照器实现相当简朴,不多说了,如下:

public int compareTo(byte[] value) {
	return value != null ? 1 : 0;
}

6. RegexStringComparator

先容:提供一个正则的对照器,支持正则表达式的值对照,仅支持 EQUAL 和非 EQUAL。匹配乐成返回0,匹配失败返回1。

先看一个小例子:

public class RegexStringComparatorDemo {

    public static void main(String[] args) {
        RegexStringComparator rsc = new RegexStringComparator("abc");
        int abc = rsc.compareTo(Bytes.toBytes("abcd"), 0, 3);
        System.out.println(abc); // 0
        int bcd = rsc.compareTo(Bytes.toBytes("bcd"), 0, 3);
        System.out.println(bcd); // 1

        String check = "^([a-z0-9A-Z]+[-|\\.]?)+[a-z0-9A-Z]@([a-z0-9A-Z]+(-[a-z0-9A-Z]+)?\\.)+[a-zA-Z]{2,}$";
        RegexStringComparator rsc2 = new RegexStringComparator(check);
        int code = rsc2.compareTo(Bytes.toBytes("zpb@163.com"), 0, "zpb@163.com".length());
        System.out.println(code); // 0
        int code2 = rsc2.compareTo(Bytes.toBytes("zpb#163.com"), 0, "zpb#163.com".length());
        System.out.println(code2); // 1
    }
}

其compareTo()方式有两种引擎实现,对应两套正则匹配规则,分别是JAVA版和JONI版(面向JRuby),默以为RegexStringComparator.EngineType.JAVA。如下:

public int compareTo(byte[] value, int offset, int length) {
	return this.engine.compareTo(value, offset, length);
}

public static enum EngineType {
	JAVA,
	JONI;

	private EngineType() {
	}
}

详细实现都很简朴,都是挪用正则语法匹配。以下是JAVA EngineType 实现:

public int compareTo(byte[] value, int offset, int length) {
	String tmp;
	if (length < value.length / 2) {
		tmp = new String(Arrays.copyOfRange(value, offset, offset + length), this.charset);
	} else {
		tmp = new String(value, offset, length, this.charset);
	}

	return this.pattern.matcher(tmp).find() ? 0 : 1;
}

JONI EngineType 实现:

public int compareTo(byte[] value, int offset, int length) {
	Matcher m = this.pattern.matcher(value);
	return m.search(offset, length, this.pattern.getOptions()) < 0 ? 1 : 0;
}

都很容易明白,不多说了。

7. SubstringComparator

先容:判断提供的子串是否出现在value中,并且不区分大小写。包罗字串返回0,不包罗返回1,仅支持 EQUAL 和非 EQUAL。

先看一个小例子:

public class SubstringComparatorDemo {

    public static void main(String[] args) {
        String value = "aslfjllkabcxxljsl";
        SubstringComparator sc = new SubstringComparator("abc");
        int i = sc.compareTo(Bytes.toBytes(value), 0, value.length());
        System.out.println(i); // 0

        SubstringComparator sc2 = new SubstringComparator("abd");
        int i2 = sc2.compareTo(Bytes.toBytes(value), 0, value.length());
        System.out.println(i2); // 1

        SubstringComparator sc3 = new SubstringComparator("ABC");
        int i3 = sc3.compareTo(Bytes.toBytes(value), 0, value.length());
        System.out.println(i3); // 0
    }
}

这个对照器实现也相当简朴,不多说了,如下:

public int compareTo(byte[] value, int offset, int length) {
	return Bytes.toString(value, offset, length).toLowerCase().contains(this.substr) ? 0 : 1;
}

到此,七种对照器就先容完了。若是对源码不敢兴趣,也建议一定要看看文中的小例子,熟悉下每种对照器的组织函数及效果输出。后续在使用HBase过滤器的过程中,会经常用到。固然除了这七种对照器,人人也可以自定义对照器。

转载请注明出处!迎接关注本人微信民众号【HBase事情条记】

,

申博Sunbet

申博Sunbet-有你喜欢的sunbet真人、sunbet电子、sunbet棋牌、sunbet代理合作。

版权声明

本文仅代表作者观点,
不代表本站Allbet的立场。
本文系作者授权发表,未经许可,不得转载。

评论